Scratch spaces are spaces for your temporary jobs files. The files and directories will be removed after 14 days from their creation.
You can choose between the provided variables $PFSDIR (/scratch/pbsjobs) and $TMPDIR (/tmp) to run your jobs.
srun --pty /bin/bash
echo $PFSDIR $TMPDIR
output:
/scratch/pbsjobs/job.<JOBID>.hpc /tmp/job.<JOBID>.hpc
Besides /scratch/pbsjobs, we also provide the project workspace /scratch/users, which is offered to Member only.
Scratch is mostly only for running jobs on the cluster
Scratch is not backed up
Any file on the /scratch/pbsjobs and /scratch/users directory is subject to deletion (based on their modified time not the upload time) after 14 days if untouched. Use Linux "touch" command to extend the deletion time ONLY if necessary.
Use $PFSDIR instead of $TMPDIR if you are using huge files or storage.
Use $TMPDIR on the nodes that have local SSDs for faster performance: gpu2080, gpu2v100, and gpu4v100.
Avoid creating huge number of files in a single directory. It will affect panasas (storage) performance especially when your job is accessing such directories frequently. This is called Directory Cache Thrashing. Refer to Panasas Storage Guideline for Huge Directory
There is a quota of 1TB in place for the /scratch/pbsjobs space. So, you may want to remove the temporary files in scratch space immediately at the end of your job script using:
rm -rf $PFSDIR/*
Member has 1TB group quota limit, and may request additional quota by contacting us.
Guest has a nominal 10GB group quota limit, and cannot request for any additional allocation.
The scratch directory files (e.g. foo.tar.gz) if untarred/unzipped needs to have -m option to update the files so that they do not get deleted after another 14 days.
tar -zxvfm foo.tar.gz
rm -rf /scratch/pbsjobs/job.<jobID>.hpc
if directory is empty:
find /scratch/pbsjobs -maxdepth 1 -type d -user <CaseID> -delete
if directory is not empty:
find /scratch/pbsjobs -maxdepth 1 -type d -user <CaseID> -exec rm -r {} \;
other flags for 'find' exist, including '-mtime', which allows to select directories by age in days.
It is very easy to Delete the /scratch job files before the job is completed, just add the following to the end of the job scripts
rm -rf "$PFSDIR"/*
pan_df -hP /scratch/* # currently not showing correct result; check quotapan
output:
Filesystem Size Used Avail Use% Mounted on
panfs://192.168.223.10/scratch 19T 9.1T 9.2T 50% /scratch/pbsjobs
panfs://192.168.223.10/scratch 9.1T 1.3T 7.9T 15% /scratch/users