Finding the right File System

If your jobs have a significant I/O part we recommend asking your consultant via support@nhr.zib.de to recommend the right file system for you.

Local I/O

Typically, depending on your I/O pattern, it is much faster if data is first collected/packed node-locally on $LOCAL_TMPDIR and then copied together over the network to $WORK in a second step. In contrast, if all MPI tasks of a node communicate individually with the remote storage servers it can be a bottleneck.
$LOCAL_TMPDIR exits on all compute nodes. On compute nodes without SSD/NVMe (see Local Disks section below) $LOCAL_TMPDIR points to the /tmpfs filesystem. In this case, its capacity is capped around 50% of the total RAM - see:
df -h $LOCAL_TMPDIR
Please note that after your Slurm job finishes all data on $LOCAL_TMPDIR will be removed (cleaned for the next user), so you need to copy it to another location before.
An example of how to make use of local I/O is given under Ex. moving local data parallel to program execution.

Global IO

Global IO is defined as shared IO which will be able to be accessed from multiple nodes at the same time and will be persistent after job end.

Local Disks

Some Compute Nodes have local SSD or NVMe storage, available in

$LOCAL_TMPDIR

An empty directory is created on job start, and data will be deleted after the job is finished. Local data can not be shared across nodes.

This is the best performing file system to use for data that doesn't need to be shared.

Special File System Types

Lustre with striping (WORK) DEPRECATED

Some workloads will benefit of striping. Files will be split transparently between a number of OSTs.

Especially large shared file IO patterns will benefit from striping. Up to 28 OSTs on Lise can be used, recommended are up to 8 OSTs for Lise. We have preconfigured a progressive file layout (PFL), which sets an automatic striping based on the file size.

Access: create a new directory in $WORK and set lfs setstripe -c <stripsize> <dir>

Size: 8 PiB like WORK

FastIO DEPRECATED

WORK is extended with 4 additional OST's using NVMe SSDs to accelerate heavy (random) IO-demands. To accelerate specific IO-demands further striping for up to these 4 OSTs is available.

Access:

create a new directory in $WORK and set lfs setstripe -p flash <dir>

Size:

55 TiB - quoted