SLURM ====== SLURM Workload Manager (or simply **SLURM**, which stands for *Simple Linux Utility for Resource Management*) is an open-source and highly scalable job scheduling system. SLURM has three key functions. First, it allocates exclusive and/or non-exclusive access to resources (compute nodes) to users for a specified period of time, allowing them to perform their work. Second, it provides a framework for starting, executing, and monitoring jobs (normally parallel jobs) on the set of allocated nodes. Finally, it arbitrates contention for resources by managing the queue of pending jobs. Currently, SLURM is the scheduling system used on **CRESCO7**, **CRESCO8** and **XCRESCO**. Comprehensive documentation is available on this portal as well as on the official `SchedMD website `_. Monitoring the Cluster ---------------------- To monitor the status of the cluster compute nodes, use the following command: .. code-block:: bash sinfo Example output from the CRESCO8 cluster: .. code-block:: bash PARTITION AVAIL TIMELIMIT NODES STATE NODELIST cresco8_cpu up 1-00:00:00 12 mix cresco8x[058,098,108-110,207-213] cresco8_gpu up 1-00:00:00 16 idle cresco8-gpu[02-17] cresco8_hbm up 1-00:00:00 14 idle cresco8-hbm[01-14] Where: - **PARTITION**: name of the cluster partition. - **AVAIL**: partition state. Can be either up, down, drain, or inact. - **TIMELIMIT**: maximum time limit for any user job. - **NODES**: count of nodes with this particular configuration. - **STATE**: node state (see the node state table). - **NODELIST**: Names of nodes associated with this particular configuration. .. list-table:: Node States :header-rows: 1 :widths: 20 80 * - State - Description * - ALLOCATED - The node has been allocated to one or more jobs * - ALLOCATED+ - The node is allocated to one or more active jobs plus one or more jobs are in the process of COMPLETING * - COMPLETING - All jobs associated with this node are in the process of COMPLETING * - DOWN - The node is unavailable for use * - DRAINED - The node is unavailable for use per system administrator request * - DRAINING - The node is currently executing a job, but will not be allocated to additional jobs * - ERROR - The node is currently in an error state and not capable of running any jobs * - FAIL - The node is expected to fail soon and is unavailable for use per system administrator request * - FAILING - The node is currently executing a job, but is expected to fail soon and is unavailable for use per system administrator request * - FUTURE - The node is currently not fully configured, but expected to be available at some point in the future * - IDLE - The node is not allocated to any jobs and is available for use * - MAINT - The node is currently in a reservation with a flag value of "maintenance" * - REBOOT - The node is currently scheduled to be rebooted * - MIXED - The node has some of its CPUs ALLOCATED while others are IDLE * - PERFCTRS (NPC) - Network Performance Counters associated with this node are in use, rendering this node not usable for other jobs * - POWER_DOWN - The node is currently powered down and not capable of running any jobs * - POWER_UP - The node is currently in the process of being powered up * - RESERVED - The node is in an advanced reservation and not generally available * - UNKNOWN - The Slurm controller has just started and the node's state has not yet been determined Running Applications -------------------- With SLURM you can specify the tasks that you want to execute; the system takes care of running these tasks and returning the results to the user. If the resources are fully utilized, SLURM holds the jobs in the queue and runs them when the required resources become available. SLURM provides two different modes to allocate resources: Interactive ~~~~~~~~~~~~ An interactive session can be started using the ``srun`` command. For example, to start an interactive session using one node with CPUs/cores fully allocated, execute the following command on a login node in the user home directory (e.g. ``~/``): .. code-block:: bash srun --nodes=1 --ntasks-per-node=2 --cpus-per-task=128 -p --pty /bin/bash .. note:: In an ``xterm`` session, an X11 interactive session can be started with the ``--x11`` option and a maximum duration of 10 hours. A parallel program can also be executed interactively with the flag ``--mpi``. Batch ~~~~~~ With SLURM you create a batch job which is then submitted to the scheduler. A batch job is a file (a shell script under UNIX) containing the set of commands that you want to execute. It also contains directives that specify the characteristics (attributes) of the job and the resource requirements (e.g. number of processors and CPU time) needed by the job. Once a job script is created, it can be reused or modified for subsequent runs. For example, the following is a simple SLURM job script that runs a user application by setting a maximum wall clock time of one hour and requesting one full node with 128 cores: :: #!/bin/bash #SBATCH -N1 -n128 # 128 cores/threads on 1 node #SBATCH --time=1:00:00 # time limit: 1 hour #SBATCH --error=myJob.err # standard error file #SBATCH --output=myJob.out # standard output file #SBATCH --partition= # partition name #SBATCH --qos= ./my_application SLURM Commands -------------- The table below provides a short description of the most commonly used SLURM commands. .. list-table:: Common SLURM Commands :header-rows: 1 :widths: 20 80 * - Command - Description * - sacct - Report job accounting information about active or completed jobs * - salloc - Allocate resources for a job in real time (typically used to allocate resources and spawn a shell, in which the ``srun`` command is used to launch parallel tasks) * - sbatch - Submit a job script for later execution (the script typically contains one or more ``srun`` commands to launch parallel tasks) * - scancel - Cancel a pending or running job * - sinfo - Report the state of partitions and nodes managed by Slurm (it has a variety of filtering, sorting, and formatting options) * - squeue - Report the state of jobs (it has a variety of filtering, sorting, and formatting options); by default it reports running jobs in priority order followed by pending jobs in priority order * - srun - Used to submit a job for execution in an interactive session * - scontrol - Report more detailed information about nodes, partitions, jobs, job steps, and configuration * - sacctmgr - Used to view and modify (admin only) Slurm account information All SLURM commands provide extensive help through their manual pages. For example: .. code-block:: bash man sbatch This command displays the help pages for the ``sbatch`` command. Partitions and Quality of Service ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ SLURM partitions are essentially different queues that point to collections of nodes. SLURM Quality of Service (QoS) defines different resource limits applied to the partitions (for example walltime limits). The maximum number of cores and the maximum walltime depend on both the selected partition and the associated Quality of Service. In the following table you can find the main features and limits imposed on the SLURM partitions. For up-to-date information, use the following commands directly on the system: .. code-block:: bash sinfo -d scontrol show partition sacctmgr -p show qos sacctmgr list qos Monitoring Jobs ~~~~~~~~~~~~~~~~~~ ``squeue`` is the main command used to monitor the state of the system, groups of jobs, or individual jobs. The ``squeue`` command prints the list of current jobs. The output looks similar to the following: .. code-block:: bash squeue -p JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) JOBID: gives the **job ID**. PARTITION: the **partition** (or queue) where the job was submitted. NAME: the **name of the job** (specified by the user in the submission script). USER: the **owner of the job**. ST: indicates the **status of the job** (see the Job States table). TIME: reports the **elapsed time** for each job. NODES: the number of requested nodes. NODELIST: shows the job is running (or the reason why it is not running). .. list-table:: Job States :header-rows: 1 :widths: 20 80 * - State - Description * - R - RUNNING * - PD - PENDING * - CA - CANCELLED * - CF - CONFIGURING * - CG - COMPLETING * - CD - COMPLETED * - F - FAILED Some additional useful ``squeue`` options include: - ``-u`` to display the status of all jobs belonging to a specific user, for example: .. code-block:: bash squeue -u - ``-l`` to display more detailed information - ``--start`` to report the expected start time of pending jobs All available options can be found in the Linux manual page: .. code-block:: bash man squeue You can also query detailed information about a specific job: .. code-block:: bash scontrol show job A real-time report of the stdout of a running job can be displayed with: .. code-block:: bash sattch .0 Deleting Jobs ~~~~~~~~~~~~~~ Use the ``scancel`` command to delete a job. For example: .. code-block:: bash scancel A user can delete their own jobs at any time, whether the job is **pending** (waiting in the queue) or **running**. Users cannot delete jobs belonging to other users. .. note:: Normally there is a small delay between issuing the ``scancel`` command and the time when the job is removed from the queue and terminated. Occasionally a job may not be deleted correctly; in this case the ENEA support team can remove it upon request. Job History ~~~~~~~~~~~~ The SLURM job accounting records provide detailed information about jobs, including job steps and memory usage. Memory usage statistics are accurate for jobs that use ``srun`` to launch tasks. To display accounting information for a specific job: .. code-block:: bash sacct -l -j To show information about currently running jobs: .. code-block:: bash sacct -l To show job information starting from a specific date: .. code-block:: bash sacct -l --starttime YYYY-MM-DD Environment Variables --------------------- When a job starts execution, SLURM defines several environment variables that can be used within the submission script to manage the workflow of the job. The most useful environment variables are listed below: .. list-table:: SLURM Environment Variables :header-rows: 1 :widths: 40 60 * - Variable - Description * - SLURM_JOB_ID - Job ID * - SLURM_JOB_NAME - Job Name * - SLURM_SUBMIT_DIR - Submit directory * - SLURM_JOB_NODELIST - Nodes assigned to the job * - SLURM_SUBMIT_HOST - Host from which the job was submitted * - SLURM_JOB_NUM_NODES - Number of nodes allocated to the job * - SLURM_CPUS_ON_NODE - Number of cores per node * - SLURM_NTASKS - Total number of cores for the job * - SLURM_NODEID - Index of the node relative to the nodes assigned to the job * - SLURM_PROCID - Index of the task relative to the job Sbatch Common Flags -------------------- This is a list of the most common flags that any user may include on scripts to request different resources and features for jobs .. list-table:: SBATCH Common Flags :header-rows: 1 :widths: 40 60 * - description - JOB Specification * - Script Directive - ``#SBATCH`` * - Job Name - ``#SBATCH --job-name=My_Job_Name`` * - Wall time hours (default is the max walltime of the qos) - ``#SBATCH --time=24:0:0`` * - Number of nodes requested - ``#SBATCH --nodes=1`` * - Number of tasks - ``#SBATCH --ntasks=48`` * - Number of core per node requested - ``#SBATCH --ntasks-per-node=48`` * - Number of cpus per task requested - ``--cpus-per-task=48`` * - send mail at the end of teh job - ``#SBATCH --mail-type=end`` Event(s) that triggers email notification (BEGIN,END,FAIL,ALL) * - user's email address - ``#SBATCH --mail-user=userid@myaddress`` * - Copy user's environment - ``#SBATCH --export=[ALL|NONE|Variables]`` * - Working Directory - ``#SBATCH --workdir=dir-name`` * - Job Restart - ``#SBATCH --requeue`` * - Share Nodes - ``#SBATCH --shared`` * - Dedicated Nodes - ``#SBATCH --exclusive`` * - Memory Size - ``#SBATCH --mem=[mem][M|G|T]`` or ``--mem-per-cpu`` * - Partition Name - ``#SBATCH --partitions=xcresco`` * - Quality of Service - ``#SBATCH --qos=`` * - Job Arrays - ``#SBATCH --array=[array_spec]`` Flag Details ~~~~~~~~~~~~~~~~~~~~~~~ - ``–exclusive`` flag selects dedicated nodes. All CPUs and memory for each node will be allocated. Programs that rely heavily on data transfer between tasks may be suited for exclusive nodes. - ``–shared`` flag will designate that a fraction of each node may be used. - ``–nodes`` or ``-N`` specifies the number of nodes and takes the form of min-max. If a single number is given, the scheduler will only allocate that number of nodes. - ``-ntasks`` or ``-n`` specifies the resources needed by giving the number of tasks. - ``–ntasks-per-node`` specifies the number of tasks for node. - ``–cpus-per-task`` specifies the number of processors for task. - ``–mem`` specifies the total amount of memory per node. - ``–mem-per-cpu`` specifies the amount of memory per allocated CPU. .. note:: Parallel processing may be done with either multiple processes, threads, or a combination of both. A single process may have multiple threads sharing memory. Multiple processes require some form to communicate, for example MPI. In SLURM, the number of processes is controlled by setting the number of “tasks”, while threads are controlled by the number of “cpus” (see below for relevant flags). Example Batch Scripts ---------------------- Serial job script ~~~~~~~~~~~~~~~~~~~~ For a typical serial job you can take the following script: myscript.sh as a template, modifying it depending on your needs. The input data are in file “data”, the output file is “job.out”; job.out will contain the std-out whilst the std-err is in the file “job.err”. .. code-block:: bash #!/bin/bash #SBATCH --job-name myjobname #SBATCH --partition= #SBATCH --output=job.out #SBATCH --error=job.err #SBATCH --time=0:10:00 #SBATCH --nodes=1 #SBATCH --ntasks=1 #SBATCH --mail-type=END #SBATCH --mail-user=usermail@example.it # Destination email address /path/to/exe In order to submit it, digit the command: .. code-block:: bash sbatch myscript.sh OpenMP job script ~~~~~~~~~~~~~~~~~~~~ For a typical OpenMP (Multi-threaded) job you can take the following script: myscript.sh as a template, modifying it depending on your needs. The script asks for 10 minutes wallclock time and runs a multi-treaded application myprogr. The output file is “job.out”; job.out will contain the std-out whilst the std-err is in the file “job.err”. .. code-block:: bash #!/bin/bash #SBATCH --job-name myjobname #SBATCH --partition= #SBATCH --output=job.out #SBATCH --error=job.err #SBATCH --time=0:10:00 #SBATCH --ntasks=1 #SBATCH --cpus-per-task=128 #SBATCH --mail-type=END #SBATCH --mail-user=usermail@example.it # Destination email address export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK cd `/ ./myprogr MPI Job Scripts ~~~~~~~~~~~~~~~~~~ For a typical MPI job you can take the following script as a template, modifying it depending on your needs. The script asks for 48 tasks on 2 nodes and 1 hour of wallclock time, and runs a MPI application (myprogr). The job runs on 72 cores of 2 nodes. The output file is “job.out”, the working directory is where the job was submitted from. .. code-block:: bash #!/bin/bash #SBATCH --job-name myjobname #SBATCH --partition= #SBATCH --output=job.out #SBATCH --error=job.err #SBATCH --time=1:00:00 #SBATCH --nodes=2 #SBATCH --ntasks-per-node=48 #SBATCH --mail-type=END #SBATCH --mail-user=usermail@example.it # Destination email address mpirun -np 4 --map-by ppr:2:node ./myprogr Hybrid MPI+OpenMP job script ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The script asks for 2 nodes, 2 MPI task per node and 128 OpenMP threads (one per each core), 1 hours of wallclock time. The application (myprogr) was compiled with the intel compiler and the openmpi library. The output file is “job.out”, the working directory is where the job was submitted from. .. code-block:: bash #!/bin/bash #SBATCH --job-name myjobname #SBATCH --partition= #SBATCH --output=job.out #SBATCH --error=job.err #SBATCH --time=1:00:00 #SBATCH --nodes=2 #SBATCH --ntasks-per-node=1 #SBATCH --cpus-per-task=128 #SBATCH --mail-type=END #SBATCH --mail-user=usermail@example.it # Destination email address cd ~/ export KMP_AFFINITY=scatter export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK mpirun -np 4 --map-by ppr:2:node --use-hwthread-cpus ./myprogr Job Array script ~~~~~~~~~~~~~~~~~~~ It is sometimes necessary to submit large numbers of jobs, especially serial (single core) jobs. When you invoke sbatch with a Job Array, you will have access to a Job index number stored in the $SLURM_ARRAY_TASK_ID environment variable. This allows a single job script to be used for multiple jobs. Here is a sample script to run 100 jobs on 1 node requiring 1 cores. .. code-block:: bash #!/bin/bash #SBATCH --job-name myjobname #SBATCH --partition= #SBATCH --output=job.out #SBATCH --error=job.err #SBATCH --time=0:10:00 #SBATCH --nodes=1 #SBATCH --ntasks=1 #SBATCH --array=0-99 #SBATCH --mail-type=END #SBATCH --mail-user=usermail@example.it # Destination email address cd ~/ module load R R < data.$SLURM_ARRAY_TASK_ID It is possible to use Job Arrays with an exotic numbering scheme. For example: .. code-block:: bash #SBATCH --array=0-10,50 This job submission would result in 12 jobs (indexed 0 through 10 and a job with an index of 50). AFS Token Management ~~~~~~~~~~~~~~~~~~~~~~~ To access the **AFS filesystem** during batch job execution, the token is generated on the compute nodes by **SLURM daemons** through integration with **AUKS** and the use of an **AUKS helper script**. AUKS generates the **Kerberos ticket** on the remote nodes by retrieving the user credentials from the login node cache. The helper script included with AUKS then invokes ``aklog`` to generate the corresponding **AFS token** from the Kerberos ticket. To protect the token generated on the compute nodes, it is recommended to execute submission scripts inside a ``pagsh``. A token generated within a ``pagsh`` is visible and usable only by the **child processes** of that pagsh session. This mechanism allows the tokens generated on a compute node by multiple jobs of the same user to remain **separate and protected**. For example, if the command ``unlog`` is executed inside a job, it will destroy only the token generated for that specific job. Without the protection provided by ``pagsh``, executing ``unlog`` would invalidate the token for **all jobs of the same user running on the same compute node**. To enable this mechanism in the previous examples: - Replace the line .. code-block:: bash #!/bin/bash with .. code-block:: bash #!/usr/bin/pagsh - Add the command ``aklog`` after the ``#SBATCH`` directives. An example of a **hybrid job submission script** using ``pagsh`` is shown below. .. code-block:: bash #!/usr/bin/pagsh #SBATCH --nodes=20 #SBATCH --ntasks-per-node=2 #SBATCH --cpus-per-task=24 #SBATCH --ntasks-per-socket=1 #SBATCH --partition= #SBATCH --job-name=jobName #SBATCH --auks=yes #SBATCH --err=%j.err #SBATCH --out=%j.out #SBATCH --account=your_account aklog export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK} mpirun --bind-to none /path/to/exe USEFUL DOCS ------------