...
Slurm offers a lot of options for job allocation, process placement, job dependencies and arrays and much more. We cannot exhaustively cover all topics here. As mentioned at the top of the page, please consult the official documentation and the man pages for an in depth description of all parameters.
Job Arrays
If Job arrays are the preferred way to submit many similar job. Jobs, for instance, if you need to submit run the same program on a large number of similar jobs, please do not use for loops to submet them, but instead use job arrays (this lessens the burden on the scheduler). Arrays can be defined using the -a <number of jobs>
option. To divide your workload on to the different jobs within your jobscript, there are several environment variables that can be used:
...
number of input files, or with different settings or run them with a range of parameters. Arrays are created with the -a start-finish
sbatch parameter. E.g. sbatch -a 0-19
will create 20 jobs indexed from 0 to 19. There are different ways to index the arrays, which are described below.
The behavior of the jobs can then be tied to Slurm Environment variables, which tell the program, which part of the array they are.
Job Array Indexing, Stepsize and more
Slurm supports a number of ways to set up the indexing in job arrays.
Range:
-a 0-5
Multiple values:
-a 1,5,12
Step size:
-a 0-5:2
(same as-a 0,2,4
)Combined:
-a 0-5:2,20
(same as-a 0,2,4,20
)
Additionally, you can limit the number of simultaneously running jobs with the %x
parameter in there:
-a 0-11%4
only four jobs at once-a 0-11%1
run all jobs sequentially-a 0-5:2,20%2
everything combined. Run IDs 0,2,4,20, but only two at a time.
You can read everything on array indexing in the sbatch man page.
Slurm Array Environment Variables
The most used environment variable in Slurm arrays is $SLURM_ARRAY_TASK_ID
. This contains the index of the job in the array and is different in every Job of the array. Other variables are:
Kein Format | ||
---|---|---|
| ||
SLURM_ARRAY_TASK_COUNT
Total number of tasks in a array.
SLURM_ARRAY_TASK_ID
Job array ID (index) number.
SLURM_ARRAY_TASK_MAX
Job array's maximum ID (index) number.
SLURM_ARRAY_TASK_MIN
Job array's minimum ID (index) number.
SLURM_ARRAY_TASK_STEP
Job array's index step size.
SLURM_ARRAY_JOB_ID
Job array's master job ID number. |
Example Job Array
This is an example of a a job array, creates a job for every file ending in “.inp” in the current workding directory
Codeblock | ||||||
---|---|---|---|---|---|---|
| ||||||
#!/bin/bash #SBATCH -p standard96 #SBATCH -t 12:00:00 #one day #SBATCH -N 16 #SBATCH --tasks-per-node 96 # insert X as the number of .inp files you have -1 (since bash arrays start counting from 0) # ls *.inp | wc -l #SBATCH -a 0-3X #SBATCH -o arrayjob-%A_%a #"%A" is replaced by the job ID and "%a" with the array index. [...] #for safety reasons shopt -s nullglob #create a bash array with all files arr=(./*.inp) #put your command here. This just runs the fictional "big_computation" program with one of the files as input ./big_computation ${arr[$SLURM_ARRAY_TASK_ID]} |
In this case, you have to get the number of files beforehand (fill in the X). You can also automatically do that by removing the #SBATCH -a
line and adding that information when submitting the job:
sbatch -a 0-$(($(ls ./*.inp | wc -l)-1)) jobarray.sh
The part in the parenthesis just uses ls
to output all .inp files, counts them with wc
and then subtracts 1, since bash arrays start counting at 0.