Zum Ende der Metadaten springen
Zum Anfang der Metadaten

Sie zeigen eine alte Version dieser Seite an. Zeigen Sie die aktuelle Version an.

Unterschiede anzeigen Seitenhistorie anzeigen

« Vorherige Version anzeigen Version 11 Nächste Version anzeigen »

This page contains all important information about the batch system Slurm, that you will need to run software on the HLRN. It does not contain every feature that Slurm has to offer. For that, please consult the official documentation and the man pages.

Submission of jobs mainly happens via the sbatch command using jobscript, but interactive jobs and node allocations are also possible using srun or  salloc. Resource selecttion (e.g. number of nodes or cores) is handled via command parameters, or may be specified in the job script.

Partitions

PartitionAvailabilityMax. walltimeNodesMax. nodes
per job
Max jobs
per user
Remark
standard96Lise12:00:00952256(var)normal nodes with 384 GB memory, default partition
standard96:testLise1:00:0032 dedicated
+128 on demand
161normal test nodes with higher priority but lower walltime
large96Lise12:00:00284(var)fat nodes with 768 GB memory
large96:testLise1:00:002 dedicated
+2 on demand
21fat test nodes with higher priority but lower walltime
large96:sharedLise48:00:002 dedicated1(var)fat nodes for data pre- and postprocessing
huge96Lise24:00:0021(var)

very fat nodes with 1536 GB memory,
for data pre- and postprocessing

medium40Emmy12:00:00368128unlimitednormal nodes with 192 GB memory, default partition
medium40:testEmmy1:00:00

16 dedicated

+48 on demand

8unlimitednormal test nodes with higher priority but lower walltime
large40Emmy12:00:00114unlimitedfat nodes with 768 GB memory
large40:testEmmy1:00:0032unlimitedfat test nodes with higher priority but lower walltime
large40:sharedEmmy24:00:0021unlimitedfor data pre- and postprocessing
gpuEmmy12:00:0011unlimitedequipped with 4 x NVIDIA Tesla V100 32GB

If you do not request a partition, you will be placed on to the default partition, which is standard96 in Berlin and medium40 in Göttingen.

The default partitions are suitable for most calculations. The :test partitions are, as the name suggests, intended for shorter and smaller test runs. These have a higher priotity and a few dedicated nodes, but are limited in time and number of nodes. The :shared nodes are mainly for postprocessing. Nearly all nodes are exclusive to one job, except for the nodes in these :shared partitions.

Parameters


ParameterComment
# nodes-N #
# tasks-n #
# tasks per node--tasks-per-node #Different defaults between mpirun and srun
partition

-p <name>

standard96/medium40

# CPUs per task

-c #Default 1, interesting for OpenMP/Hybrid jobs
Timelimit-t hh:mm:ss
Mail--mail-type=ALLSee sbatch manpage for different types

Job Scripts

A job script can be any script that contains special instruction for Slurm. Most commonly used forms are shell scripts, such as bash or plain sh. But other scripting languages (e.g. Python, Perl, R) are also possible.

Example Batch Script
#!/bin/bash

#SBATCH -p medium40
#SBATCH -N 16
#SBATCH -t 06:00:00

module load impi
srun mybinary

The job scripts have to have a shebang line at the top, followed by the #SBATCH options. These #SBATCH  comments have to be at the top, as Slurm stops scanning for them after the first non-comment non-whitespace line (e.g. an echo or variable declaration).

Getting Information about Jobs

The commands normally used for job control and management are:


  • Job submission:
    $ sbatch jobscript.sh 
  • Job status of a specific job:
    $ squeue -j jobID for queues/running jobs
    $ scontrol show job jobID for full job information (even after the job finished).
  • Job cancellation:
    $ scancel jobID
  • Job overview:
    $ squeue -l -u $USER
  • Job start (estimated):
    $ squeue --start -j jobID
  • Workload overview of the whole system: sinfo (esp. sinfo --format="%25C %A") , squeue -l

Using the Shared Nodes

We provide a varying number of nodes from the large40 and large96 partitions as post processeing nodes in a shared mode, so that multiple jobs can run at once on a single node. You can request CPUs and memory and should take care, that you do not exceed your limits. For each CPU/Hyperthread, there is about 9.6Gb of Memory on large40:shared or 4 on the large96:shared partition.

The maximum walltime on the shared partitions is currently 2 days.

 Example Job for the shared partition

This is an example for a job script using 10 cores. As this is not a MPI job, srun/mpirun is not needed. This jobs memory usage should not exceed

#!/bin/bash
#SBATCH -p large96:shared
#SBATCH -t 1-0 #one day
#SBATCH -n 10
#SBATCH -N 1

python postprocessing.py

Advanced Options

Job Arrays

Job Dependencies

  • Keine Stichwörter