Zum Ende der Metadaten springen
Zum Anfang der Metadaten

Sie zeigen eine alte Version dieser Seite an. Zeigen Sie die aktuelle Version an.

Unterschiede anzeigen Seitenhistorie anzeigen

« Vorherige Version anzeigen Version 13 Nächste Version anzeigen »

HLRN provides tailored WORK file systems for improved IO throughput of IO intense job workloads.

Default Lustre (WORK)

WORK is the default shared file system for all jobs and can be accessed using the $WORK variable. WORK is accessible for all users and consists of 8 Metadata Targets (MDT's) with NVMe SSDs and 28 Object Storage Targets (OST's) on Lise and 100 OST's on Emmy handling the data. Both using classical hard drives.

Access: $WORK

Size: 8 PiB quoted


Special File System Types

Lustre with striping (WORK)

Some workloads will benefit of striping. Files will be split between a number of OSTs.

Especially large shared file IO patterns will benefit from striping. Up to 28 OSTs on Lise and up to 100 OST's on Emmy can be used, recommeded are up to 8 OSTs for Lise and 32 OSTs on Emmy.

Access: create a new directory in $WORK and set lfs setstripe -c <stripsize> <dir>

Size: 8 PiB like WORK


Local SSDs

Some Compute Nodes are installed with local SSD storage up to 2 TB on Lise and 400 GB on Emmy.

Data on local SSDs can not be shared across nodes and will be deleted after the job is finished.

For unshared local IO this is the best performing file system to use.



Lise: SSDLise: CASEmmy: SSD
Access

via queue: standard96:ssd

using $LOCAL_TMPDIR

via queue: large96 and huge96

using $LOCAL_TMPDIR

via queue: medium40 and large40

using $LOCAL_TMPDIR

Type and sizeIntel NVMe SSD DC P4511 (2 TB)

Intel NVMe SSD DC P4511 (2 TB) using

Intel Optane SSD DC P4801X (200 GB)

as write-trough cache

Intel DC S4500 (400 GB)


FastIO - Lise only

WORK is extended with 4 additional OST's using NVMe SSDs to accelerate heavy IO-demands. To accelerate specific IO-demands further striping for up to 4 OSTs is available.

Access: ask support@hlrn.de for access

Size: 55 TiB - quoted


IME - Emmy only

DDN Infinite Memory Engine (IME) based Burst Buffers is a fast NVMe Solid State Disk (SSD) based data tier. Like Lise's FastIO this is especially helpful for random IO (especially random read) on large files.

Using the Burst Buffer for random IO helps to avoid overloading the global filesystem which results in slow job runtimes for all users. Beside the POSIX interface a native API and a MPI-IO module for further acceleration is available.

IME servers are currently available for use in EMMY.

Access: ask support@hlrn.de for access

Size: 48 TiB


Finding the right File System

If your jobs have a significant IO part we recommend asking your consultant via support@hlrn.de to recommend the right file system for you.

Local IO

If you have a significant amount of node-local IO which is not needed to be accessed after job end and will be smaller than 2 TB on Lise and 400 GB on Emmy we recommend using $LOCAL_TMPDIR. Depending on your IO pattern this may accelerate IO to up to 100%.

Global IO

Global IO is defined as shared IO which will be able to be accessed from multiple nodes at the same time and will be persistent after job end.

Especially random IO on small files will be accelerated up to 200% using FastIO on Lise or IME on Emmy







INTERNAL - not public - will be deleted

Recommendation Matrix:

Max performance gain on IO versus default $WORK in brackets.


small random IOlots of large IO per processfew large IO accessed from many nodesunknown IO

local IOglobal IOlocal IOglobal IOlocal IOglobal IOlocal IOglobal IO
Code e.g.
OpenFOAM?


FESOM?

write IOLocal SSDs (+100%)FastIO stripe=4 (+30%)Local SSDs (+15%)$WORK$WORK

FastIO stripe=4       (+200%) or

$WORK stripe=4-8 (+200%)

Local SSDs (+40%)

FastIO stripe=4       (+80%)

$WORK stripe=4-8  (+70%)

read IOLocal SSDs (+30%)FastIO             (+140%)Local SSDs (+30%)FastIO stripe=4 (+20%)Local SSDs (+35%)FastIO stripe=4       (+200%)Local SSDs (+30%)

FastIO stripe=4         (+120%)

$WORK stripe=MAX (+90%)

balanced IOLocal SSDs (+60%)FastIO stripe=4 (+90%)Local SSDs (+25%)FastIO               (+15%)Local SSDs (+20%)

FastIO stripe=4         (+200%) or

$WORK stripe=MAX (+150%)

Local SSDs (+35%)

FastIO stripe=4         (+100%)

FastIO                        (+50%)

Commands

Striping einrichten

mkdir <dirname>

lfs setstripe -c <count> <dirname>

DoM einrichten

mkdir <dirname>

lfs setstripe -E 64K -L mdt -E -1 -p work.rotational <dirname>

FastIO einrichten

mkdir <dirname>

lfs setstripe -p flash <dirname>

Fastio mit Striping 4 einrichten:


mkdir <dirname>


lfs setstripe -p flash -c 4 <dirname>

Prüfen

lfs getstripe <dirname>






Vorschlag Ankündigung

Subj: additional WORK file systems available

Dear HLRN Users.

To achieve improved IO performance HLRN has installed additional file systems tailored for dedicated IO demands.

LISE has now node-local SSDs using NVMe and Optane to accelerate local IO. Furthermore we have installed a SSD based Lustre storage pool called 'FastIO'.

On EMMY the IME burst buffer can be used for fast random IO demands and all phase 1 nodes are equipped with local SSDs.

If your jobs have a significant amount of IO which can be accelerated visit: 13730697 or contact our support.

Kind Regards

HLRN-Team


  • Keine Stichwörter