Systems

(1) Hosts with Intel Optane Memory

Hostnames: apass{1,2}

The two systems apass{1,2}  are equipped with Intel Optane Memory components (first generation Apache Pass): Storage Class Memory modules (SCM/NVRAM) and SSDs. The main difference between the two hosts is the memory capacity.

The Optane Memory, i.e. each of the systems, can be configured in two (three) modes:

• Memory Mode: The Optane Memory is exposed as RAM to the OS. Although the hardware technology is different to DRAM, this mode allows transparent usage of the SCM memory. The main benefits are the gained memory capacity and easy usage (no modification of applications required). The DRAM memory acts as cache for Optane Memory. In this mode, the SCM is effectively not persistent.
• AppDirect Mode: The Optane Memory is exposed as block device(s) to the OS (usually as /dev/pmemX, depending on actual configuration). On such block devices, a file system can be created which should support the direct access (DAX) option. This allows to map data on the persistent Optane Memory into an application's virtual address space while avoiding the OS' page cache. Thus, direct media access to the SCM is possible with load/store operations. Note that data in the Optane Memory is effectively cleared when the mode is changed. Therefore, data on /dev/pmemX should be considered as ephemeral.
• (Mixed/Hybrid Mode: The System can also be configured to provide a portion of the Optane capacity for the Memory Mode and another portion for AppDirect mode.)

By default: apass1 is configured in Memory Mode while apass2 is configured in AppDirect Mode. If you need a different configuration contact S. Christgau Steffen Christgau (Unlicensed) . The mount points for the persistent memory are usually /mnt/pmemX. X often matches the NUMA domain of the socket/processor the memory is attached to. To be sure run lstopo from the hwloc environment module. Not every pmem device might be mounted or accessible if a system is in AppDirect mode because other software (DAOS, e.g.) may exclusively grab a device. Check the output of mount to find mount points of /dev/pmemX.

The login message (message of the day) displays the mode in which the system is currently running in. You can also check the CurrentVolatileMode property in the /var/run/optane/state file. As a further simple check for the given mode, you can run free -h. If the total memory capacity is around or larger than 3 TB the system is in memory mode. Further, if /dev/pmem[01] exists, the AppDirect (or Mixed/Hybrid) mode is in effect.

(2) NEC SX-Aurora TSUBASA A300-8

Hostname: aurora

(3) 3rd Gen Intel Xeon Cooper Lake

Hostname: cpl

Cooper Lake is Intel's codename for the third-generation of their Xeon scalable processors, developed as the successor to Cascade Lake.

Improvements:

• New bfloat16 instruction
• Support for up to 12 DIMMs of DDR4 memory per CPU socket

(4) Intel Xeon Ice Lake

Hostname: icl

(5) Dataflow Engine hardware accelerator

Hostname: maverick{1,2}

One dataflow engine hardware accelerator card per server (accelerator cards provided by ParTec).

Access

To gain access to the Next-Generation Technology Pool, contact support@nhr.zib.de. Please give a short description of your intention and the system you intend to use.

Use Slurm on the NGT login node to access individual NGT systems (new since June 2022).

The NGT login node is "login-ngt", reachable using ssh via our public login nodes "blogin.hlrn.de" (replace USERNAME with your HLRN account name):

$ ssh -J USERNAME@blogin.hlrn.de USERNAME@login-ngt

Make use of the ssh-agent to avoid repeated prompts for the passphrase (ALL keys used to access the HLRN MUST have a passphrase). Run "ssh-agent" to start the agent and load your default key. Or, if your ssh key is in ~/.ssh/id_rsa_ngt, run:

$ ssh-add ~/.ssh/id_rsa_ngt

With a suitable ssh config, you can jump to the NGT login node using one simple command:

$ ssh login-ngt

The ssh config in ~/.ssh/config looks like this (replace USERNAME with your HLRN account name):

Host login-ngt
     ProxyJump %r@blogin.hlrn.de
     Hostname login-ngt
     User USERNAME
     IdentityFile ~/.ssh/id_rsa_hlrn

Unused compute nodes are shut down. Slurm will start nodes when needed. Depending on the node this takes 2..5 minutes.

Use sinfo to query the node status. In the following example, "icl" is up an running, "cpl" is powered down to save energy (indicated by the "~" mark at the end):

login$ sinfo -N
NODELIST   NODES PARTITION STATE 
icl            1       icl idle  
cpl            1       cpl idle~ 

To start an interactive session on a compute node, use srun.

login$ srun --pty -picl bash -ls
icl$ 

IMPORTANT: Start "srun" directly in your Home directory or on the network filesystem in /net/tmp. Home directories are not shared between login node and compute nodes, and Slurm will silently fail if the working directory does not exist on the compute node.

Alternatively, you can use "salloc" to start and allocate a node:

login$ salloc -picl

When a node is up, direct ssh access is still possible, but needs login-ngt as jump host. An example ssh-config (for node "cpl") is:

Host cpl-ngt
     ProxyJump %r@blogin.hlrn.de,%r@login-ngt.usr.hlrn.de
     Hostname cpl.ngt.hlrn.de
     User USERNAME
     IdentityFile ~/.ssh/id_rsa_hlrn