SLURM: Main Commands

Slurm Commands

Submit jobs - [sbatch]

Batch job submission can be accomplished with the command sbatch. Like in Torque qsub, we create a bash script to describe our job requirement: what resources we need, what softwares and processing we want to run, memory and CPU requested, and where to send job standard output and error etc. After a job is submitted, Slurm will find the suitable resources, schedule and drive the job execution, and report outcome back to the user. The user can then return to look at the output files.

Example-1:

In the first example, we create a small bash script, run it locally, then submit it as a job to Slurm using sbatch, and compare the results.  

Example-2:

Follow the recipe below to submit a job. The job can be used later as an example for practicing how to check job status. In my test its running duration is about 7 minutes.

Below is the content of the bash script "run-matlab.s" just used in the job submission:

The job has been submitted successfully. And as the example box showing, its job ID is 11615. Usually we should let the scheduler to decide on what nodes to run jobs. In cases there is a need to request a specific set of nodes, use the directive nodelist, e.g. '#SBATCH --nodelist=c09-01,c09-02'.

Check cluster status - [sinfo]

The sinfo command gives information about the cluster status, by default listing all the partitions. Partitions group computing nodes into logical sets, which serves various functionalities such as interactivity, visualization and batch processing.

A partition is a group of nodes. A partition can be made up of nodes with a specific feature/functionality, such as nodes equipped with GPU accelerators (gpu partition). A partition can have specific parameters, such as how long jobs can run. So partitions can be thought as "queues" in other batch systems. Partitions may overlap.

sinfo by default prints information aggregated by partition and node state. As shown above, there are four partitions namely c01_25, c26, c27 and gpu. The partition marked with an asterisk is the default one. Except there are two lines with the node state 'mix', which means some CPU cores occupied, all other nodes are idle.

See two useful sinfo command examples: 1. the first one lists those nodes in idle state in the gpu partition; 2. the second outputs information in a node-oriented format.

Check job status - [squeue, sstat, sacct]

The squeue command lists jobs which are in a state of either running, or waiting or completing etc. It can also display jobs owned by a specific user or with specific job ID.

Run 'man sinfo' or 'man squeue' to see the explanations for the results.

With the job ID in hand, we can track the job status through its lifetime. The job first appears in the Slurm queue in the PENDING state. Then when its required resources become available, the job gets priority for its turn to run, and is allocated resources, the job will transit to the RUNNING state. Once the job runs to the end and completes successfully, it goes to the COMPLETED state; otherwise it would be in the FAILED state. Use squeue -j <jobID> to check a job status.

Most of the columns in the output of the squeue command are self-explanatory. 

The column "ST" in the middle is the job status, which can be :

• • PD - pending: waiting for resource allocation

  • S  - suspended

  • R  - running

  • F  - failed: non-zero exit code or other failures

  • CD - completed: all processes terminated with zero exit code

  • CG - completing: in the completing process, some processes may still be alive

The column "NODELIST(REASON)" in the end is job status due to the reason(s), which can be :

• • JobHeldUser:            (obviously) 

  • Priority:               higher priority jobs exist

  • Resources:              waiting for resources to become available

• • BeginTime:              start time not reached yet

  • Dependency:             wait for a depended job to finish

  • QOSMaxCpuPerUserLimit:  number of CPU core limit reached

You may select what columns to display, in a width specified with an integer number between %. and a letter, %.10M.

Run the command sstat to display various information of running job/step. Run the command sacct to check accounting information of jobs and job steps in the Slurm log or database. There is a '–-helpformat' option in these two commands to help checking what output columns are available.

Type "man <command>" to look up detailed usage on the manual pages of command squeue, sstat and sacct.

Cancel a job - [scancel]

Things can go wrong, or in a way unexpected. Should you decide to terminate a job before it finishes, scancel is the tool to help.

Slurm Job Results

Job results includes the job execution logs (standard output and error), and of course the output data files if any defined when submitting the job. Log files should be created in the working directory, and output data files in your specified directory. Examine log files with a text viewer or editor, to gain a rough idea of how the execution goes. Open output data files to see exactly what result is generated. Run sacct command to see resource usage statistics. Should you decide that the job needs to be rerun, submit it again with sbatch with a modified version of batch script and/or updated execution configuration. Iteration is one characteristic of a typical data analysis!

SLURM Environment Variables

To get the list of SLURM_* variables, you may run a job to check, e.g. srun sh -c 'env | grep SLURM | sort' . The command 'man sbatch' explains what these variables stand for. Below are a few frequently used:

• • SLURM_JOB_ID             -  the job ID

  • SLURM_SUBMIT_DIR         -  the job submission directory

  • SLURM_SUBMIT_HOST        -  name of the host from which the job was submitted

  • SLURM_JOB_NODELIST       -  names of nodes allocated to the job

  • SLURM_ARRAY_TASK_ID      -  job array job index

  • SLURM_JOB_CPUS_PER_NODE  -  CPU cores on this node allocated to the job

  • SLURM_NNODES             -  number of nodes allocated to the job

Exceeded Step Memory Limit Warning Message

The current Slurm implementation utilizes Linux Control Groups (cgroups) for resource containment. If necessary please see this page at kernel.org for a detailed description of cgroups.

If you get the correct outputs, please just ignore this warning message - "slurmstepd: error: Exceeded job memory limit at some point". You can also check job exit state to confirm. For reference there is some explanation in the bug report.