SLURM

SLURM Workload Manager (or simply SLURM, which stands for Simple Linux Utility for Resource Management) is an open-source and highly scalable job scheduling system.

SLURM has three key functions. First, it allocates exclusive and/or non-exclusive access to resources (compute nodes) to users for a specified period of time, allowing them to perform their work. Second, it provides a framework for starting, executing, and monitoring jobs (normally parallel jobs) on the set of allocated nodes. Finally, it arbitrates contention for resources by managing the queue of pending jobs.

Currently, SLURM is the scheduling system used on CRESCO7, CRESCO8 and XCRESCO. Comprehensive documentation is available on this portal as well as on the official SchedMD website.

Monitoring the Cluster

To monitor the status of the cluster compute nodes, use the following command:

sinfo

Example output from the CRESCO8 cluster:

PARTITION       AVAIL   TIMELIMIT    NODES  STATE      NODELIST
cresco8_cpu     up      1-00:00:00     12    mix    cresco8x[058,098,108-110,207-213]
cresco8_gpu     up      1-00:00:00     16   idle    cresco8-gpu[02-17]
cresco8_hbm     up      1-00:00:00     14   idle    cresco8-hbm[01-14]

Where:

PARTITION: name of the cluster partition.
AVAIL: partition state. Can be either up, down, drain, or inact.
TIMELIMIT: maximum time limit for any user job.
NODES: count of nodes with this particular configuration.
STATE: node state (see the node state table).
NODELIST: Names of nodes associated with this particular configuration.

Node States
State	Description
ALLOCATED	The node has been allocated to one or more jobs
ALLOCATED+	The node is allocated to one or more active jobs plus one or more jobs are in the process of COMPLETING
COMPLETING	All jobs associated with this node are in the process of COMPLETING
DOWN	The node is unavailable for use
DRAINED	The node is unavailable for use per system administrator request
DRAINING	The node is currently executing a job, but will not be allocated to additional jobs
ERROR	The node is currently in an error state and not capable of running any jobs
FAIL	The node is expected to fail soon and is unavailable for use per system administrator request
FAILING	The node is currently executing a job, but is expected to fail soon and is unavailable for use per system administrator request
FUTURE	The node is currently not fully configured, but expected to be available at some point in the future
IDLE	The node is not allocated to any jobs and is available for use
MAINT	The node is currently in a reservation with a flag value of “maintenance”
REBOOT	The node is currently scheduled to be rebooted
MIXED	The node has some of its CPUs ALLOCATED while others are IDLE
PERFCTRS (NPC)	Network Performance Counters associated with this node are in use, rendering this node not usable for other jobs
POWER_DOWN	The node is currently powered down and not capable of running any jobs
POWER_UP	The node is currently in the process of being powered up
RESERVED	The node is in an advanced reservation and not generally available
UNKNOWN	The Slurm controller has just started and the node’s state has not yet been determined

Running Applications

With SLURM you can specify the tasks that you want to execute; the system takes care of running these tasks and returning the results to the user. If the resources are fully utilized, SLURM holds the jobs in the queue and runs them when the required resources become available.

SLURM provides two different modes to allocate resources:

Interactive

An interactive session can be started using the srun command. For example, to start an interactive session using one node with CPUs/cores fully allocated, execute the following command on a login node in the user home directory (e.g. ~/):

srun --nodes=1 --ntasks-per-node=2 --cpus-per-task=128 -p <partition_name> --pty /bin/bash

Note

In an xterm session, an X11 interactive session can be started with the --x11 option and a maximum duration of 10 hours. A parallel program can also be executed interactively with the flag --mpi.

Batch

With SLURM you create a batch job which is then submitted to the scheduler. A batch job is a file (a shell script under UNIX) containing the set of commands that you want to execute. It also contains directives that specify the characteristics (attributes) of the job and the resource requirements (e.g. number of processors and CPU time) needed by the job.

Once a job script is created, it can be reused or modified for subsequent runs.

For example, the following is a simple SLURM job script that runs a user application by setting a maximum wall clock time of one hour and requesting one full node with 128 cores:

#!/bin/bash
#SBATCH -N1 -n128                     # 128 cores/threads on 1 node
#SBATCH --time=1:00:00               # time limit: 1 hour
#SBATCH --error=myJob.err            # standard error file
#SBATCH --output=myJob.out           # standard output file
#SBATCH --partition=<partition_name> # partition name
#SBATCH --qos=<qos_name>

./my_application

SLURM Commands

The table below provides a short description of the most commonly used SLURM commands.

Common SLURM Commands
Command	Description
sacct	Report job accounting information about active or completed jobs
salloc	Allocate resources for a job in real time (typically used to allocate resources and spawn a shell, in which the `srun` command is used to launch parallel tasks)
sbatch	Submit a job script for later execution (the script typically contains one or more `srun` commands to launch parallel tasks)
scancel	Cancel a pending or running job
sinfo	Report the state of partitions and nodes managed by Slurm (it has a variety of filtering, sorting, and formatting options)
squeue	Report the state of jobs (it has a variety of filtering, sorting, and formatting options); by default it reports running jobs in priority order followed by pending jobs in priority order
srun	Used to submit a job for execution in an interactive session
scontrol	Report more detailed information about nodes, partitions, jobs, job steps, and configuration
sacctmgr	Used to view and modify (admin only) Slurm account information

All SLURM commands provide extensive help through their manual pages. For example:

man sbatch

This command displays the help pages for the sbatch command.

Partitions and Quality of Service

SLURM partitions are essentially different queues that point to collections of nodes.

SLURM Quality of Service (QoS) defines different resource limits applied to the partitions (for example walltime limits).

The maximum number of cores and the maximum walltime depend on both the selected partition and the associated Quality of Service.

In the following table you can find the main features and limits imposed on the SLURM partitions. For up-to-date information, use the following commands directly on the system:

sinfo -d
scontrol show partition <partition_name>
sacctmgr -p show qos
sacctmgr list qos <partition>

Monitoring Jobs

squeue is the main command used to monitor the state of the system, groups of jobs, or individual jobs.

The squeue command prints the list of current jobs. The output looks similar to the following:

squeue -p <partition_name>

JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)

JOBID: gives the job ID. PARTITION: the partition (or queue) where the job was submitted. NAME: the name of the job (specified by the user in the submission script). USER: the owner of the job. ST: indicates the status of the job (see the Job States table). TIME: reports the elapsed time for each job. NODES: the number of requested nodes. NODELIST: shows the job is running (or the reason why it is not running).

Job States
State	Description
R	RUNNING
PD	PENDING
CA	CANCELLED
CF	CONFIGURING
CG	COMPLETING
CD	COMPLETED
F	FAILED

Some additional useful squeue options include:

-u to display the status of all jobs belonging to a specific user, for example:

squeue -u <user>

-l to display more detailed information
--start to report the expected start time of pending jobs

All available options can be found in the Linux manual page:

man squeue

You can also query detailed information about a specific job:

scontrol show job <jobid>

A real-time report of the stdout of a running job can be displayed with:

sattch <jobid>.0

Deleting Jobs

Use the scancel command to delete a job. For example:

scancel <jobid>

A user can delete their own jobs at any time, whether the job is pending (waiting in the queue) or running. Users cannot delete jobs belonging to other users.

Note

Normally there is a small delay between issuing the scancel command and the time when the job is removed from the queue and terminated. Occasionally a job may not be deleted correctly; in this case the ENEA support team can remove it upon request.

Job History

The SLURM job accounting records provide detailed information about jobs, including job steps and memory usage. Memory usage statistics are accurate for jobs that use srun to launch tasks.

To display accounting information for a specific job:

sacct -l -j <jobid>

To show information about currently running jobs:

sacct -l

To show job information starting from a specific date:

sacct -l --starttime YYYY-MM-DD

Environment Variables

When a job starts execution, SLURM defines several environment variables that can be used within the submission script to manage the workflow of the job.

The most useful environment variables are listed below:

SLURM Environment Variables
Variable	Description
SLURM_JOB_ID	Job ID
SLURM_JOB_NAME	Job Name
SLURM_SUBMIT_DIR	Submit directory
SLURM_JOB_NODELIST	Nodes assigned to the job
SLURM_SUBMIT_HOST	Host from which the job was submitted
SLURM_JOB_NUM_NODES	Number of nodes allocated to the job
SLURM_CPUS_ON_NODE	Number of cores per node
SLURM_NTASKS	Total number of cores for the job
SLURM_NODEID	Index of the node relative to the nodes assigned to the job
SLURM_PROCID	Index of the task relative to the job

Sbatch Common Flags

This is a list of the most common flags that any user may include on scripts to request different resources and features for jobs

SBATCH Common Flags
description	JOB Specification
Script Directive	`#SBATCH`
Job Name	`#SBATCH --job-name=My_Job_Name`
Wall time hours (default is the max walltime of the qos)	`#SBATCH --time=24:0:0`
Number of nodes requested	`#SBATCH --nodes=1`
Number of tasks	`#SBATCH --ntasks=48`
Number of core per node requested	`#SBATCH --ntasks-per-node=48`
Number of cpus per task requested	`--cpus-per-task=48`
send mail at the end of teh job	`#SBATCH --mail-type=end` Event(s) that triggers email notification (BEGIN,END,FAIL,ALL)
user’s email address	`#SBATCH --mail-user=userid@myaddress`
Copy user’s environment	`#SBATCH --export=[ALL\|NONE\|Variables]`
Working Directory	`#SBATCH --workdir=dir-name`
Job Restart	`#SBATCH --requeue`
Share Nodes	`#SBATCH --shared`
Dedicated Nodes	`#SBATCH --exclusive`
Memory Size	`#SBATCH --mem=[mem][M\|G\|T]` or `--mem-per-cpu`
Partition Name	`#SBATCH --partitions=xcresco`
Quality of Service	`#SBATCH --qos=<name>`
Job Arrays	`#SBATCH --array=[array_spec]`

Flag Details

–exclusive flag selects dedicated nodes. All CPUs and memory for each node will be allocated. Programs that rely heavily on data transfer between tasks may be suited for exclusive nodes.
–shared flag will designate that a fraction of each node may be used.
–nodes or -N specifies the number of nodes and takes the form of min-max. If a single number is given, the scheduler will only allocate that number of nodes.
-ntasks or -n specifies the resources needed by giving the number of tasks.
–ntasks-per-node specifies the number of tasks for node.
–cpus-per-task specifies the number of processors for task.
–mem specifies the total amount of memory per node.
–mem-per-cpu specifies the amount of memory per allocated CPU.

Note

Parallel processing may be done with either multiple processes, threads, or a combination of both. A single process may have multiple threads sharing memory. Multiple processes require some form to communicate, for example MPI. In SLURM, the number of processes is controlled by setting the number of “tasks”, while threads are controlled by the number of “cpus” (see below for relevant flags).

Example Batch Scripts

Serial job script

For a typical serial job you can take the following script: myscript.sh as a template, modifying it depending on your needs. The input data are in file “data”, the output file is “job.out”; job.out will contain the std-out whilst the std-err is in the file “job.err”.

#!/bin/bash

#SBATCH --job-name myjobname
#SBATCH --partition=<partition_name>
#SBATCH --output=job.out
#SBATCH --error=job.err
#SBATCH --time=0:10:00
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --mail-type=END
#SBATCH --mail-user=usermail@example.it      # Destination email address

/path/to/exe

In order to submit it, digit the command:

sbatch myscript.sh

OpenMP job script

For a typical OpenMP (Multi-threaded) job you can take the following script: myscript.sh as a template, modifying it depending on your needs. The script asks for 10 minutes wallclock time and runs a multi-treaded application myprogr. The output file is “job.out”; job.out will contain the std-out whilst the std-err is in the file “job.err”.

#!/bin/bash

#SBATCH --job-name myjobname
#SBATCH --partition=<partition_name>
#SBATCH --output=job.out
#SBATCH --error=job.err
#SBATCH --time=0:10:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=128
#SBATCH --mail-type=END
#SBATCH --mail-user=usermail@example.it      # Destination email address

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
cd `/
./myprogr

MPI Job Scripts

For a typical MPI job you can take the following script as a template, modifying it depending on your needs. The script asks for 48 tasks on 2 nodes and 1 hour of wallclock time, and runs a MPI application (myprogr). The job runs on 72 cores of 2 nodes. The output file is “job.out”, the working directory is where the job was submitted from.

#!/bin/bash

#SBATCH --job-name myjobname
#SBATCH --partition=<partition_name>
#SBATCH --output=job.out
#SBATCH --error=job.err
#SBATCH --time=1:00:00
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=48
#SBATCH --mail-type=END
#SBATCH --mail-user=usermail@example.it      # Destination email address

mpirun -np 4 --map-by ppr:2:node ./myprogr

Hybrid MPI+OpenMP job script

The script asks for 2 nodes, 2 MPI task per node and 128 OpenMP threads (one per each core), 1 hours of wallclock time. The application (myprogr) was compiled with the intel compiler and the openmpi library. The output file is “job.out”, the working directory is where the job was submitted from.

#!/bin/bash

#SBATCH --job-name myjobname
#SBATCH --partition=<partition_name>
#SBATCH --output=job.out
#SBATCH --error=job.err
#SBATCH --time=1:00:00
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=128
#SBATCH --mail-type=END
#SBATCH --mail-user=usermail@example.it      # Destination email address

cd ~/
export KMP_AFFINITY=scatter
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
mpirun -np 4 --map-by ppr:2:node --use-hwthread-cpus ./myprogr

Job Array script

It is sometimes necessary to submit large numbers of jobs, especially serial (single core) jobs. When you invoke sbatch with a Job Array, you will have access to a Job index number stored in the $SLURM_ARRAY_TASK_ID environment variable. This allows a single job script to be used for multiple jobs. Here is a sample script to run 100 jobs on 1 node requiring 1 cores.

#!/bin/bash

#SBATCH --job-name myjobname
#SBATCH --partition=<partition_name>
#SBATCH --output=job.out
#SBATCH --error=job.err
#SBATCH --time=0:10:00
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --array=0-99
#SBATCH --mail-type=END
#SBATCH --mail-user=usermail@example.it      # Destination email address

cd ~/
module load R
R < data.$SLURM_ARRAY_TASK_ID

It is possible to use Job Arrays with an exotic numbering scheme. For example:

#SBATCH --array=0-10,50

This job submission would result in 12 jobs (indexed 0 through 10 and a job with an index of 50).

AFS Token Management

To access the AFS filesystem during batch job execution, the token is generated on the compute nodes by SLURM daemons through integration with AUKS and the use of an AUKS helper script. AUKS generates the Kerberos ticket on the remote nodes by retrieving the user credentials from the login node cache. The helper script included with AUKS then invokes aklog to generate the corresponding AFS token from the Kerberos ticket.

To protect the token generated on the compute nodes, it is recommended to execute submission scripts inside a pagsh.

A token generated within a pagsh is visible and usable only by the child processes of that pagsh session. This mechanism allows the tokens generated on a compute node by multiple jobs of the same user to remain separate and protected.

For example, if the command unlog is executed inside a job, it will destroy only the token generated for that specific job. Without the protection provided by pagsh, executing unlog would invalidate the token for all jobs of the same user running on the same compute node.

To enable this mechanism in the previous examples:

Replace the line
```
#!/bin/bash
```
with
```
#!/usr/bin/pagsh
```
Add the command aklog after the #SBATCH directives.

An example of a hybrid job submission script using pagsh is shown below.

#!/usr/bin/pagsh
#SBATCH --nodes=20
#SBATCH --ntasks-per-node=2
#SBATCH --cpus-per-task=24
#SBATCH --ntasks-per-socket=1
#SBATCH --partition=<partition_name>
#SBATCH --job-name=jobName
#SBATCH --auks=yes
#SBATCH --err=%j.err
#SBATCH --out=%j.out
#SBATCH --account=your_account

aklog
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
mpirun --bind-to none /path/to/exe

SLURM

Monitoring the Cluster

Running Applications

Interactive

Batch

SLURM Commands

Partitions and Quality of Service

Monitoring Jobs

Deleting Jobs

Job History

Environment Variables

Sbatch Common Flags

Flag Details

Example Batch Scripts

Serial job script

OpenMP job script

MPI Job Scripts

Hybrid MPI+OpenMP job script

Job Array script

AFS Token Management

USEFUL DOCS