File Systems at HLRN-IV

HLRN is operating at each site 3 central storage systems:

• IBM Spectrum Scale file system with 340 TiByte capacity, which exports via NFS
  • $HOME directories /home/${USER}/
  • Centrally managed software and the module system in /sw
• Lustre parallel file system with 8 PiByte capacity containing
  • $WORK directories /scratch/usr/${USER}/
  • $TMPDIR directories /scratch/tmp/${USER}/
  • project data directories /scratch/projects/<projectID>/ (not yet available)
• Tape archive with multiple petabyte capacity with additional harddisk caches

The system Emmy has addtional storage options for high IO demands:

• Phase 1 nodes (partitions medium40 and large40): Local SSD for temporary data at $LOCAL_TMPDIR (400 GiB shared among all jobs running on the node). The environment variable $LOCAL_TMPDIR is available on all nodes, but on the phase 2 systems it points to a ramdisk.
• DDN IME based burst buffer with 48TiB NVMe storage (general availability together with the phase 2 nodes)

Login and copying data between HLRN sites

Inter-complex login (ssh) as well data copy (rsync/sftp) between both sites (Berlin and Göttingen) should work right out of the box. The same is true for inner-complex ssh and scp between nodes of one site. This is enabled through hostbased authentication.

Always use the short hostname for ssh/rsync, either the generic names blogin, glogin or the specific names like blogin5, glogin2, etc. This allows the usage of the direct intersite connection HLRN Link, which ich much faster then the internet connection, which is used, when you access the nodes of the other site by the hostnames b/glogin.hlrn.de.

Home and Software

The home filesystem and /sw are mounted via NFS, so performance is medium. We take daily snapshots of the filesystem, which can be used to restore a former state of a file or directory. These snapshots can be accessed through the path /home/.snapshots or /sw/.snapshots. There are additional regular backups to restore the filesystem in case of a catastrophic failure.

Work

The Lustre based work filesystem /scratch is the main work filesystem for the HLRN clusters. We provide no backup of this filesystem. The storage system of Emmy provides around 65GiB/s streaming bandwith and Lise around 85GiB/s in normal operation. The storage system is harddisk based (with solidstate disks vor metadata), so the best performance can be reached with sequential IO of large files that is aligned to the fullstripe size of the underlying RAID6 (Emmy 1MiB, Lise 16MiB).

If you are accessing a large file (1GiB+) from multiple nodes in parallel, please consider to activate striping of the file with the Lustre command lfs setstripe (specific to this file or for a whole directory, changes apply only for new files, so applying a new striping to an existing file requires a file copy) with a sensible stripe_count (recommendation: Emmy up to 32, Lise up to 8) and a stripe_size, which is a multiple of the RAID6 fullstripe size and matches the IO sizes of your job.

A general recommendation for network filesystems is to keep the number of metadata operations for open and closing files, as well as checks for file existence or changes as low as possible. These operations often become a bottleneck for the IO of your job and on large clusters, as the ones operated by HLRN, can easily overload the file servers.

Tape archive PERM

The magnetic tape archive provides additional storage for inactive data to free up space on the work filesystem. It is directly accessible via the login nodes at the mountpoint /perm/${USER}/.

Emmy provides the additional option to access the PERM archive via ssh to the archive nodes gperm1 and gperm2, so you can use rsync, scp, sftp for file transfer.


For reasons of efficiency and performance, small files and/or complex directory structures should not be transferred to the archive directly. Please aggregate your data to compressed tarballs or other archive containers with a maximum size of 5,5TiB before copying your data to the archive.

Quota

The following default quota limits apply (additional space allocations up to the requirements estimate in your project application can be granted on request):

You are able to exceed your soft-quota limit for a grace period of 2 weeks until further write access is denied. Exceeding hard-quota limits result in a immediate deny.

On HLRN Quota is UNIX-group based. All files on a specific filesystem are counted based on their GID, nevertheless of their physical path.

The individual quota usage can be checked using 'hlrnquota' on the Login nodes.

If you have questions regarding your quota please contact your consultant.