Garpur-Slurm

How to properly submit jobs on Garpur

sbatch -J testRRRR example-slurm-script.sh

Submitting a job using a Slurm jobscript using sbatch (similar command as qsub in PBS system).
The -J flag is optional but sets the Slurm jobname (can also be set inside jobscript). example-slurm-script.sh is the name of the jobscript.

The example-slurm-script.sh Slurm jobscript is specific to the kind of program that the user is running and possibly specific to the user as well.
Below is the contents of an example script 

IMPORTANT: It is very important to make efficient use of Garpur and to not slow down the cluster, to copy input information for your program over to the local scratch on the node (/scratch), have the jobscript change directory (cd) to the scratch directory and launch the program from the scratch directory. If this step is omitted then the program will run remotely on the node but will constanly write all of its large temporary files back to the submission directory (that may be on /users/home or /users/work) and this creates a lot of network traffic that slows down the use of Garpur for everybody. It will also slow down the job itself.

By using the script below and modify it according to your needs, you make sure that your job runs effectively on the cluster.


#!/bin/bash

# Usage of this script:
#sbatch -J jobname example-slurm-script.sh

# About SBATCH lines below:
# Choose number of nodes (N, computers), number of tasks per node (CPU cores), and walltime (hours:minutes:seconds). Also queue: normal, himem, himem-bigdisk, gpu.
# himem is intended for really memory intensive jobs, himem-bigdisk for jobs that require a lot of memory and lots of disk space for scratch (has 3 TB local harddrive). Generally you should use the normal queue. gpu queue is intended for GPU (Graphics processing Unit) jobs.
# The example below will submit to the normal queue a job that reserves 1 core (task) on 1 node (N) of the cluster. The maximum time that the job can run is here set to 2 hours. After 2 hours the queuing system will forcefully cancel the job so make sure your job has finished within that time.
#SBATCH -N 1
#SBATCH --tasks-per-node=1
#SBATCH --time=2:00:00
#SBATCH -p normal

#Setting variable job to the Slurm job name
job=$SLURM_JOB_NAME

#Location of scratch directory. Here using the local scratch dir: /scratch that is present on each node. Do not use /users/work or /users/home.
scratchlocation=/scratch


#Control OpenMP or MKL parallelization of your program here if needed. Othewise leave set to 1.
export MKL_NUM_THREADS=1
export OMP_NUM_THREADS=1

# Load modules here that are necessary for your job. Add more module-loading lines if needed.
module load intel/compiler/2017.2

#Define PATH and LD_LIBRARY_PATH manually here if needed
export PATH=/path/to/dir:$PATH
export LD_LIBRARY_PATH=/path/to/dir/lib:$LD_LIBRARY_PATH

#This creates user scratch directory if it does not exist on the node
if [ ! -d $scratchlocation/$USER ]
then
  mkdir -p $scratchlocation/$USER
fi

# Change programname variable below  from myprogram to the name of the program that you are running. e.g. programname=matlab
#This helps identifying the scratch directories created.
programname=myprogram

#Create temporary directory job-scratch directory inside $scratchlocation/$USER on the node.
#tdir here becomes the specific scratch directory for the job (deleted at the end of job)
tdir=$(mktemp -d $scratchlocation/$USER/$programnamejob__$SLURM_JOBID-XXXX)
#Makes the tdir job-scratch dir readable by others and so that its size can be shown.
chmod +xr $tdir

# Go to temporary scratch dir. This step means that there will be no network traffic while job is running if using local scratch
cd $tdir

#Exiting if tdir does not exist.
if [ ! -d "$tdir" ]; then
echo "Scratch dir  does not exist on this node"
echo "Maybe because scratchlocation $scratchlocation does not exist on node??"
echo "Or because permissions did not allow creation of folder?"
echo "Exiting..."
exit
fi

#Copying files from submission dir to scratch dir that are necessary for the job.
#Avoid copying everything, copy only the stuff that is necessary.

#Copying file that has the name given at submission and has an .inp ending from submission dir to tdir.
cp $SLURM_SUBMIT_DIR/$jobname.inp $tdir/
#Copying all files with an .xyz ending
cp $SLURM_SUBMIT_DIR/*.xyz $tdir/


# Copy job and node info to beginning of outputfile
echo "Job execution start: $(date)" > $SLURM_SUBMIT_DIR/$job.out
echo "Walltime is: $SBATCH_TIMELIMIT" >> $SLURM_SUBMIT_DIR/$job.out
echo "Shared library path: $LD_LIBRARY_PATH" >> $SLURM_SUBMIT_DIR/$job.out
echo "Path environment: $PATH" >> $SLURM_SUBMIT_DIR/$job.out
echo "Slurm Job ID is: ${SLURM_JOBID}" >> $SLURM_SUBMIT_DIR/$job.out
echo "Slurm Job name is: ${SLURM_JOB_NAME}" >> $SLURM_SUBMIT_DIR/$job.out
echo "Scratchdir is: $tdir" >> $SLURM_SUBMIT_DIR/$job.out
echo "Slurm nodes in use: $SLURM_NODELIST" >> $SLURM_SUBMIT_DIR/$job.out

#These lines print how much space is available on local scratch BEFORE job begins
header=$(df -h | grep Filesy)
scratchsize=$(df -h | grep scratch)
echo "Scratch size is (before job run):" >> $SLURM_SUBMIT_DIR/$job.out
echo "$header" >> $SLURM_SUBMIT_DIR/$job.out
echo "$scratchsize" >> $SLURM_SUBMIT_DIR/$job.out



#Start the job here.
#Output file (standard output created by the program) is here written directly to submit directory. While this creates some network traffic, it should be small.
#If your program creates a lot of output here then redirect the output to the scratch location instead.
/path/to/program $job.inp >> $SLURM_SUBMIT_DIR/$job.out


#These lines print how much space is available on local scratch AFTER job begins
header=$(df -h | grep Filesy)
scratchsize=$(df -h | grep scratch)
echo "" >> $SLURM_SUBMIT_DIR/$job.out
echo "Scratch size is (after job, before scratch deletion):" >> $SLURM_SUBMIT_DIR/$job.out
echo "$header" >> $SLURM_SUBMIT_DIR/$job.out
echo "$scratchsize" >> $SLURM_SUBMIT_DIR/$job.out

# The job has now finished (or crashed). Now copy important stuff back to the submission directory.
#Avoid copying everything back, copy only the data you really need.
cp $tdir/*.xyz $SLURM_SUBMIT_DIR
cp $tdir/*.loc $SLURM_SUBMIT_DIR

#Important. This deletes the temporary scratch directory once the job is complete.
rm -rf $tdir
Comments