Scratch Spaces

Scratch Spaces

Scratch spaces are spaces for your temporary jobs files. The files and directories will be removed after 14 days from their creation. 

You can choose between the provided variables $PFSDIR (/scratch/pbsjobs) and $TMPDIR (/tmp) to run your jobs.

srun --pty /bin/bash

echo $PFSDIR $TMPDIR

output:

/scratch/pbsjobs/job.<JOBID>.hpc /tmp/job.<JOBID>.hpc

Besides /scratch/pbsjobs, we also provide the project workspace /scratch/users, which is offered to Member only.

Guidance

• •   Scratch is mostly only for running jobs on the cluster

  •   Scratch is not backed up

  •   Any file on the /scratch/pbsjobs and /scratch/users directory is subject to deletion after 14 days if untouched.

  • Use $PFSDIR instead of $TMPDIR if you are using huge files or storage.

  • Use $TMPDIR on the nodes that have local SSDs for faster performance: gpu2080, gpu2v100, and gpu4v100.

  • Avoid creating huge number of files in a single directory. It will affect panasas (storage) performance especially when your job is accessing such directories frequently. This is called Directory Cache Thrashing. Refer to Panasas Storage Guideline for Huge Directory

Quota

/scratch/pbsjobs

There is a quota of 1TB in place for the /scratch/pbsjobs space. So, you may want to remove the temporary files in scratch space immediately at the end of your job script using:
rm -rf $PFSDIR/*

/scratch/users

Member has 1TB group quota limit, and may request additional quota by contacting us.
Guest has a nominal 10GB group quota limit, and cannot request for any additional allocation.

Managing Scratch Space

Updating Scratch Files

The scratch directory files (e.g. foo.tar.gz) if untarred/unzipped needs to have -m option to update the files so that they do not get deleted after another 14 days.

tar -zxvfm foo.tar.gz

Removing scratch space manually

rm -rf /scratch/pbsjobs/job.<jobID>.hpc

Removing folders under scratch belonging to a user

• if directory is empty:

find /scratch/pbsjobs -maxdepth 1 -type d -user <CaseID> -delete

• if directory is not empty:

find /scratch/pbsjobs -maxdepth 1 -type d -user <CaseID> -exec rm -r {} \;

other flags for 'find' exist, including '-mtime', which allows to select directories by age in days.

Removing Scratch Files Immediately

It is very easy to Delete the /scratch job files before the job is completed, just add the following to the end of the job scripts

rm -rf "$PFSDIR"/*

Scratch Usage

pan_df -hP /scratch/* # currently not showing correct result; check quotapan

output:

Filesystem Size Used Avail Use% Mounted on

panfs://192.168.223.10/scratch   19T 9.1T 9.2T 50% /scratch/pbsjobs

panfs://192.168.223.10/scratch  9.1T 1.3T 7.9T 15% /scratch/users