HPC Batch & Interactive Job

Batch Job Submission

Batch job submission is recommended when submitting a job in the HPC. The job script, or SLURM script, is used to submit the job. In general, a Slurm script is a bash script that contains SBATCH directives to request resources for the job, file manipulations commands to handle job files, and execution parts for running one or more programs that constitute the job. Note that the SBATCH directives may not have blank lines between them -- the first blank line is interpreted as the end of the directive section.  Generally, bash scripts may have blank lines and lines beginning with hash (#) are treated as comments, ingnored by the bash interpreter. https://www.gnu.org/software/bash/manual/bash.html

When the job is submitted using "sbatch" command, it assigns a unique "JobID" for the job.

Consider the SLURM script (example.slurm) below:

Using scratch space 

(see HPC Temporary and Scratch Space):

#!/bin/bash

#SBATCH -o test.o%j

#SBATCH --time=00:30:00      # 30 minutes of wall time

#SBATCH -N 1                 # 1 Node

#SBATCH -c 1                 # 1 processor

#SBATCH -mem=2gb             # Assign 2gb memory; default in 1gb

module load <software-module>

cp -r executable <inputFile1> <inputFile2> ...<inputFileN> $PFSDIR

cd $PFSDIR

./executable <arg1> <arg2> ... <argN> 

cp -ru * $SLURM_SUBMIT_DIR

The job file requests 30 minutes of wall time (time=00:30:00), requests one processor (n 1) in one node (N 1) for processing, and specifies with ".o%j" that stderr and stdout be redirected to the file name test.o<jobid>, respectively. These SLURM options are followed by bash commands to be executed by a shell after the job is submitted on a compute node. Load the software module to set the environment variables, copy the files (executable,inputFile1, inputFile2, ..) required for your job to the temporary (scratch) directory ($PFSPDIR) associated with the job, execute there, and copy all (*) output files back to the working directory ($SLURM_SUBMIT_DIR). The executable may require arguments (arg1 arg2 ..).

Notes: For small amount of scratch data, you can use $TMPDIR instead of $PFSDIR but still $PFSDIR is recommended. Also, to remove Scratch Files Immediately, add the following to the end of the job scripts

rm -rf "$PFSDIR"/*

Submit the job:

sbatch example.slurm

We recommend you to use the scratch space, however, if your files are huge and you think it takes time to copy to PFSDIR, you can directly run the job from your working directory as showed:

Without using scratch space (NOT RECOMMENDED)

#!/bin/bash

#SBATCH -o test.o%j

#SBATCH --time=00:30:00

#SBATCH -N 1

#SBATCH -c 1

#SBATCH --mem=2gb

module load <software-module>

./<executable>

For OpenMP jobs, request the number of processors "<x>" as required.

#SBATCH -N 1 -c <x>

For MPI jobs, request the number of nodes "<n>" and tasks "<i>" and processors-per-task "<x>" as required and use "mpiexec" or "mpirun" along with your executable as showed:

#SBATCH -N <n> -n <i> -c <x>

...

mpiexec <mpi-executable>

Interactive Job Submission

See the appropriate Graphical Access section, if you would like to have graphical interface. 

The following command request any node with the default request.

srun --pty /bin/bash

You will be assigned a compute node (prompt similar to (<CaseID>@compxxx ~). You job will have a unique JobID. By default, the granted resources will be -N 1 -n 1 --mem=1gb --time=10:00:00

For GUI access:

srun --x11 -N 1 -c 2 --time=1:00:00 --pty /bin/bash

Note: This will provide two processors (-c 2) on a single node (-N 1) for 1 hour (--time=1:00:00), with graphical windows (--x11) ready. If you want to request, let's say, 100 hrs, you can do it as --time=4-04:00:00 in the the format: "days-hours:minutes:seconds".

Check the default memory and walltime by using:

scontrol show job <JobID> | grep MinMemory

scontrol show job <JobID> | grep TimeLimit

output:

MinCPUsNode=1 MinMemoryCPU=1000M MinTmpDiskNode=0

   RunTime=01:30:36 TimeLimit=10:00:00 TimeMin=N/A

To run interactive jobs with 2 processor in 1 node with <m>gb memory of any node type: 

srun -N 1 -c 2 --mem=<m>gb --pty /bin/bash

Please request the nodes and processors appropriately to accommodate <m>gb looking here.

To reserve 2 processors on 2 nodes (total of 4 processors):

 srun --time=32:00:00 -N 2 -n c --pty /bin/bash

To request 1 processor in partition <P>:

srun -p <P> --pty /bin/bash

Requesting Specific Nodes

To request a specific node, use the --nodelist option.

Request a specific node:

srun -p gpu -C gpu2v100 --nodelist=gpu059t --gres=gpu:1  --pty bash

It is requesting gpu059t. It is useful when we want to use a specific node only. Please be mindful that you might have to wait until the node is free to use if someone else is running a job in that node.

Exclude a particular node or multiple nodes from running your jobs:

srun --exclude=compt227,compt229 --pty bash

srun --exclude=compt[230-240] --pty bash

If you want to reserve the whole node, you can use, --exclusive

srun --exclusive --pty bash

Request an smp node in smp partition (-p smp).

srun -p smp -c 32 --mem=500gb --pty bash

Removing Scratch Space

Removing scratch space manually

rm -rf /scratch/pbsjobs/pbsjobs.<jobID>.hpc

Removing Scratch Files Immediately

It is very easy to Delete the /scratch job files before the job is completed, just add the following to the end of the PBS job scripts

rm -rf "$PFSDIR"/*

High Memory Jobs

You need to assign appropriate memory to run your high memory job. It may get terminated before the completion when memory resource is not met (see exit codes). Calculate the amount of memory/processor from HPC Resource View and use the appropriate queue and the number of processors. Please also refer to Memory Estimation Guide. Memory leak can also be the culprit for high memory requirement. Please refer to HPC Guide to Debugging Segmentation Fault.

Example (serial job):

if you require about 23 gb of memory for your serial job, then you can use 6 processors. 

srun -N 1 -c 6 --mem=23gb --pty /bin/bash

You may want to reserve the whole node (--exclusive) if you don't know in advance how much memory your job requires.  Please also use --mem=0  flag if you need to request all available memory in that node

srun --pty --exclusive --pty bash

Example (Parallel job):

If you have a parallel job that requires 120gb of memory, one option is using 2 nodes with as many cpu-core as are available (match the node resources).

srun -N 2 -c 24 --mem=120gb --pty /bin/bash  # example applies to thread-parallel.

srun -N 2 -n 24 --mem=120gb --pty /bin/bash  # example applies for MPI parallel.

Job Submission using different Accounts

If you are affiliated with more than one group and want to run jobs using account group that is not your default (primary) group, use the -A option in SLURM script as showed:

#SBATCH -A <groupAccount>

To see the list of your affiliated shares, use the following command. You will see you groups.

$ i

****Your SLURM's CPU Quota****

           account_1      128

           account_2       24

You need to include the following line in the slurm file:

#SBATCH -A account_2