| Inhalt |
|---|
The compute nodes of Lise in Berlin (blogin.hlrn.de) and Emmy in Göttingen (glogin.hlrn.de) are organized the CPU cluster of system Lise are organised via the following SLURM partitions:
Lise (Berlin)
...
Partition (number holds cores per node)
...
Slurm partitions.
Partition name | Node count | CPU | Main memory (GB) | Max. nodes per job | Max. jobs per user (running/ queued)per user | Usable memory MB per node | CPU | Shared | Charged core-hours per node | Remark | standard96 | bcn#Wall time limit (hh:mm:ss) | Remark | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| cpu-clx | 688 | Cascade 9242 | 362 | 512 | 128 / 500 | 12:00:00 | 1204 | 512 | 16 / 500 | 362 000 | Cascade 9242 | ✘ | 96 | default partition | |||||||||||||||||
| standard96cpu-clx:test | bcn# | 1:00:00 | 32 dedicated +128 on demand | 362 | 16 | 1 / 500 | 362 000 | Cascade 9242 | ✘ | 96 | 01:00:00 | test nodes with higher priority but lower walltimeless wall time | |||||||||||||||||||
| large96 | bfn# | 12:00:00 | 2828 | 747 | 8 | 16 128 / 500 | 747 000 | Cascade 9242 | ✘ | 144 | 12:00:00 | fat memory nodes | |||||||||||||||||||
| large96:test | bfn# | 1:00:00 | 2 dedicated +2 on demand | 2 | 1 / 500 | 747 000 | Cascade 9242 | ✘ | 144 | fat memory test nodes with higher priority but lower walltime | large96:shared | bfn# | 48:00:00 | 2 dedicated | 1 16 / 500 | 747 000 | Cascade 9242 | ✓ | 144 | fat memory nodes for data pre- and postprocessing | huge96 | bsn# | 2401:00:00 | 2 | 1 | 16 / 500 | 1522 000 | Cascade 9242 | ✓ | 192 | very fat memory nodes for data pre- and postprocessing |
12 hours are too short? See here how to pass the 12h walltime limit with job dependencies.
Emmy (Göttingen)
Partition (number holds cores per node)
per job
per user
Usable memory MB per node
CPU, GPU type
gcn#
+128 on demand
| test nodes with higher priority but |
| less wall time |
| large96 |
| :shared | 2 |
| dedicated | 747 |
| 1 |
+2 on demand
| 128 / 500 | 48:00:00 |
2 dedicated
+6 on demand
| fat memory nodes for data pre- and |
| post-processing |
| huge96 |
| 2 |
| 1522 |
very fat memory nodes for data pre- and postprocessing
| 1 |
8 dedicated
+64 on demand
181 000
2 dedicated
+2 on demand
764 000
2 dedicated
+6 on demand
| 128 / 500 | 24:00:00 |
764 000
very fat memory nodes for data pre- and |
764 000 MB per node
(32GB HBM per GPU)
see GPU Usage
500 000 MB per node
(40GB HBM per GPU)
(40GB or 80GB HBM per GPU)
and Zen2 EPYC 7662 + 8 NVidia A100 80GB
see GPU Usage
GPUs are split into slices via MIG (3 slices per GPU)
...
post-processing |
See Slurm usage how to pass the 12h wall time limit with job dependencies.
Which partition to choose?
If you do not request a partition, your job will be placed in the default partition, which is standard96.
The default partitions are partition is suitable for most calculations. The :test partitions are, as the name suggests, intended for shorter and smaller test runs. These have a higher priority and a few dedicated nodes, but are limited in time and number of nodesprovide only limited resources. Shared nodes are suitable for pre- and postprocessingpost-processing. A job running on a shared node is only accounted for its core fraction (cores of job / all cores per node). All non-shared nodes are exclusive to one job , which implies that full NPL are paid.Details about the CPU/GPU types can be found belowonly at a time.
The network topology is described here.The available home/local-ssd/work/perm storages file systems are discussed in under File Systems.
An For an overview of all Slurm partitions and node statuses is provided bystatus of nodes: sinfo -r
To see For detailed information about a particular nodes type: scontrol show node <nodename>
Charge rates
Charge rates for the Slurm partitions can be found under Accounting.
Fat-Tree Communication Network of Lise
See OPA Fat Tree network of Lise
List of CPUs
...
| Short name | Link to manufacturer specifications | Where to find | Units per node | Cores per unit | Clock speed |
|---|---|---|---|---|---|
| Cascade 9242 | Intel Cascade Lake Platinum 9242 (CLX-AP) | Lise and Emmy compute partitionsCPU partition "Lise" | 2 | 48 | 2.3 |
| Cascade 4210 | Intel Cascade Lake Silver 4210 (CLX) | blogin[1-8], glogin[3-86] | 2 | 10 | 2.2 |
| Skylake 6148 | Intel Skylake Gold 6148 | Emmy compute partitions | 2 | 20 | 2.4 |
| Skylake 4110 | Intel Skylake Silver 4110 | glogin[1-2] | 2 | 8 | 2.1 | Tesla V100 | NVIDIA Tesla V100 32GB | Emmy gpu partition | 4 | 640/5120* | Tesla A100 | NVIDIA Tesla A100 40GB and 80GB | Emmy grete partitions | 4 or 8 | 432/6912* |
...