| Inhalt |
|---|
HLRN NHR provides tailored WORK file systems for improved IO throughput of IO intense job workloads.
...
WORK is the default shared file system for all jobs and can be accessed using the $WORK variable. WORK is accessible for all users and consists of 8 Metadata Targets (MDT's) with NVMe SSDs and 28 Object Storage Targets (OST's) on Lise and 100 OST's on Emmy handling the data. Both using classical hard drives.
...
Especially large shared file IO patterns will benefit from striping. Up to 28 OSTs on Lise and up to 100 OST's on Emmy can be used, recommended are up to 8 OSTs for Lise and 32 OSTs on Emmy. We have preconfigured a progressive file layout (PFL), which sets an automatic striping based on the file size.
...
For unshared local IO this is the best performing file system to use.
| Lise: SSD | Lise: CAS | Emmy: SSD | |
|---|---|---|---|
| Access | via partition: using | via partition: using using | |
| Type and size | Intel NVMe SSD DC P4511 (2 TB) | Intel NVMe SSD DC P4511 (2 TB) using Intel Optane SSD DC P4801X (200 GB) as write-trough cacheIntel S-ATA SSD DC S4500 (480 GB) Intel NVMe SSD DC P4511 (1TB) |
FastIO
WORK is extended with 4 (Lise)/8 (Emmy) additional OST's using NVMe SSDs to accelerate heavy (random) IO-demands. To accelerate specific IO-demands further striping for up to these 4 /8 OSTs is available.
Access:
Lise: create a new directory in $WORK and set lfs setstripe -p flash <dir>Emmy: Using either $SHARED_SSD_TMPDIR for a job specific folder (liek $LOCAL_TMPDIR) oder set the storage pool ddn_ssd for a new directory like for Lise (there the SSD pool is called flash).
Size:
Lise: 55 TiB - quotedEmmy: 120 TiB - quoted
IME - Emmy only
DDN Infinite Memory Engine (IME) based Burst Buffers is a fast NVMe Solid State Disk (SSD) based data tier. Like Lise's FastIO this is especially helpful for random IO (especially random read) on large files.
Using the Burst Buffer for random IO helps to avoid overloading the global filesystem which results in slow job runtimes for all users. Beside the POSIX interface a native API and a MPI-IO module for further acceleration is available.
...
Access: add the option --constraint=ime to your jobscript and use $IME_TMPDIR for a job specific folder or /ime/usr/$user for more permanent usage.
Size: 300 TiB
Finding the right File System
If your jobs have a significant IO part we recommend asking your consultant via support@hlrnsupport@nhr.zib.de to recommend the right file system for you.
...
If you have a significant amount of node-local IO which is not needed to be accessed after job end and will be smaller than 2 TB on Lise and 400 GB/1 TB (depending on the node) on Emmy we recommend using $LOCAL_TMPDIR. Depending on your IO pattern this may accelerate IO to up to 100%.
...
Especially random IO will be accelerated up to 200% using FastIO on Lise or IME on Emmy.
Performance Comparison of the different File Systems and SSDs for Emmy
Please remember that we are comparing here a single SSD for the node local SSDs, 360 SSDs for IME, 43 SSDs for Lustre SSD and 1000 HDDs for Lustre HDD using 32 IO processes per node.
For the global filesystems 64 nodes were used to achieve near maximum performance for IME and the Lustre HDD pool. A typical user job will see lower performance values. The numbers for the global filesystems indicate the aggregate performance that is distributed across all users.