Table of Contents

Help and answers

For questions, please contact the support crew support@nhr.zib.de.

Login

Login authentication is possible via SSH keys only. Please visit our tutorial SSH Login.

...

office $ ssh -i $HOME/.ssh/id_rsa_nhr nhr_username@blogin.nhr.zib.de
Enter passphrase for key '...':
blogin1 $

File systems

Each complex has the following file systems available. More information about Quota, usage, and best pratices are available hereon Fixing Quota Issues. Hints for data transfer are given here.

• Home file system with 340 TiByte capacity containing $HOME directories /home/${USER}/
• Lustre parallel file system with 8.1 PiByte capacity containing
  • $WORK directories /scratch/usr/${USER}/
  • $TMPDIR directories /scratch/tmp/${USER}/
  • project data directories /scratch/projects/<projectID>/ (not yet available)
• Tape archive with 120 TiByte capacity (accessible on the login nodes, only)On Emmy: SSD for temporary data at $LOCAL_TMPDIR (400 GB shared among all jobs running on the node)

...

Software and environment modules

The webpage Software gives you information about available software on the NHR systems.

...

To avoid conflicts between different compilers and compiler versions, builds of most important libraries are provided for all compilers and major release numbers.

Program build

Here only a brief introduction to program building using the intel compiler is given. For more detailed instructions, including important compiler flags and special libraries, refer to our webpage Compilation GuideCPU CLX.

Examples for building a program on the Atos system

To build executables for the Atos system, call the standard compiler executables (icc, ifort, gcc, gfortran) directly.

...

module load intel
icc -qopenmp -o hello.bin hello.c

MPI, Communication Libraries, OpenMP

We provide several communication libraries:

...

OpenMP support is available with the compilers from Intel and GNU.

Using the batch system

To run your applications on the systems, you need to go through our batch system/scheduler: Slurm. The scheduler uses meta information about the job (requested node and core count, wall time, etc.) and then runs your program on the compute nodes, once the resources are available and your job is next in line. For a more in depth introduction, visit our Slurm documentation.

...

• Interactive job execution
• Job script execution

Resource specification

To request resources, there are multiple flags to be used when submitting the job.

Interactive jobs

Anker
interactive_jobs interactive_jobs

For using compute resources interactively, e.g. to follow the execution of MPI programs, the following steps are required. Note that non-interactive batch jobs via job scripts (see below) are the primary way of using the compute resources.

1. A resource allocation for interactive usage has to be requested first with the salloc --interactive command which should also include your resource requirements.
2. When salloc successfully allocated the requested resources and when working at the Göttingen complex (Emmy), you are automatically logged in at one of the allocated compute nodes. For Berlin (Lise), you have to issue an additional srun command to work one of the allocated nodes (see example below) if you want to work on the compute node.
3. Afterwards, srun or MPI launch commands, like mpirun or mpiexec, can be used to start parallel programs (see according user guides)

blogin1 ~ $ salloc -t 00:10:00 -p standard96:test -N2 --tasks-per-node 24
salloc: Granted job allocation [...]
salloc: Waiting for resource configuration
salloc: Nodes bcn[1001,1003] are ready for job
# To get a shell on one of the allocated nodes at
the Berlin complex/Lise (not required for Göttingen/Emmy)
blogin1 ~ $ srun --pty --interactive --preserve-env ${SHELL}
bcn1001 ~ $ srun hostname | sort | uniq -c
     24 bcn1001
     24 bcn1003
bcn1001 ~ $ exit
# Exit a second time for Berlin/Lise 
blogin1:~ > exit
salloc: Relinquishing job allocation [...]

Job scripts

Please go to our webpage CPU partition "Lise" for more details about job scripts. For introduction, standard batch system jobs are executed applying the following steps:

...

#!/bin/bash
#SBATCH -t 00:20:00
#SBATCH -N 1
#SBATCH --cpus-per-task=96
#SBATCH -p large96:test

# This binds each thread to one core
export OMP_PROC_BIND=TRUE
# Number of threads as given by -c / --cpus-per-task
export OMP_NUM_THREADS=$(($SLURM_CPUS_PER_TASK * 2))
export KMP_AFFINITY=verbose,scatter

hello_world > hello.output

Job Accounting

The webpage Accounting and NPL gives  gives you more information about job accounting.

Every batch job on Lise and Emmy is accounted. The account (project) which is debited for a batch job can be specified using the sbatch parameter --account <account>. If a batch job does not state an account (project), a default is taken from the account database. It defaults to the personal project of the user, which has the same name as the user. Users may modify their default project by visiting the Service portal Portal NHR@ZIB.