Storage Quota

Folders with Quota or Storage Limits

Important Notes

• The quota in HPC is termed a 'hard' quota. When the full usage allowed by the quota is reached, further information may not be added to disk until the usage has been reduced below 95% of the quota.  This reduces the scenario in which small reductions in usage take place, writing resumes, and the quota is again reached -- particularly important in managing usage in a multi-user environment.

• The storage that has snapshot feature implemented may not release free storage immediately. So, avoid quota violation as much as possible. Keep monitoring and managing your disk space.

/home

Your most important storage location is /home as it is where you will be when you log into the cluster. You will also likely keep your most important current files in this location. Therefore, it is important to keep the usage of this space organized and managed below the group quota.

Reducing /home usage:

• Delete the data that is no longer necessary

• Compress the folders into <filename>.tar.gz, so the size is kept to the minimum.

• You can move the data from your /home to a different location by using several methods, including this simple command:
  rsync -avz <source-path-to-directory> <destination-path-to-same-directory>

• The 'newgrp' command may be used in the shell to chance the group ownership of the files and directories generated in a session.  The group ownership determines which group has that usage against it for the quota.  When belonging to more than one group, it's up to you to manage this ownership setting.

/home recovery from accidental deletion

If you accidently delete your files in your /home directory during the last 30 days, you can retrieve the files from the snapshots residing at your home directory /home/<CaseID>/.snapshot. The snapshots are taken every day for the last 7 days, and every week for the last 4 weeks. For other details and backup information, visit Recovering Files. 

/scratch

Please find more information about /scratch on its own page.

/mnt/pan or /mnt/vstor

If you lease a storage volume in the HPC storage, it will be allocated at:

/mnt/pan/<Userspace>

where the <Userspace> is most like: /SCHOOL_DEPT_PI'sID

If you lease a storage volume in the Research Storage, it will be allocated at: /mnt/vstor/<Userspace>

Monitoring Disk Usage & Quota

Quick View

Check /home and /scratch space usage for your group including you. Note that it gets updated at 6:03am, 6:03am, 11:03 am, 3:03pm, 11:03pm. Use panfs_quota below for instantaneous usage.

quotachk <CaseID>

output:

Output: partition, user/group, usage, %of soft quota, % of hard quota, # of files

    /home            user:<caseID>          431.72 GB       0       0  365.50 K       0       0

    /scratch/pbsjobs user:<caseID>                  0       0       0      58.0       0       0

    /scratch/users   group:hpcadmin                 0       0       0       1.0       0       0

    /home            group:hpcadmin         621.19 GB   88.71   67.52 1663.93 K       0       0

    /scratch/pbsjobs group:hpcadmin          80.96 GB   10.12    8.10     103.0       0       0

...

Out of 621.19 GB space at /home used by hpcadmin group, the user has used 431.72GB. The group has used 80.96GB of scratch space

Check the detailed list of /home usage for all users in a group (updated once a day).  Though you can't get 

quotagrp <PI_CaseID>

output:

UserID  Member       Total     LgstDir  Path

<user1> Yes      415.569GB   359.106GB  /home/<user1>/projects

<user2> Yes      152.037GB   140.788GB  /home/<user2>/bench

...

Total Quota Usage: 721.47GB

%Soft Quota Used : 103.07

%Hard Quota Used : 78.42

The quotagrp command gives information on the largest folder that occupy the space on /home, so if you need to reduce usage, you can start looking at removing files/folders from that folder.  For information on /scratch and other mount spaces,  refer to other sections in this document.

For more information on usage and output format, use --help flag.

quotagrp --help

Getting the quota usage on /home, /scratch/pbsjobs and /scratch/users

Our policies currently do not allow the expansion of /home but we can expand the /mnt/pan space in Panasas.

Find out your disk usage using the command at the prompt. This gives the instantaneous current usage.

panfs_quota /home /scratch/pbsjobs /scratch/users # for personal user usage

panfs_quota -G /home /scratch/pbsjobs /scratch/user # for group usage

Output example for the personal usage :

<bytes> <soft> <hard> : <files> <soft> <hard> : <path to volume> <pan_identity(name)>

148757434368 unlimited unlimited : 443458 unlimited unlimited : /home/<CaseID> uid:5xxxxx(CaseID)

The example above gives ~149GB of personal usage. Also, note that the user having ~443k files is over the limit of 100-200k.

Output example for the group usage: 

       <bytes>       <soft>       <hard> : <files>    <soft>    <hard> : <path to volume> <pan_identity(name)>

  297113653248 700000000000 920000000000 :  605488 unlimited unlimited : /home/<CaseID> gid:123xxx(GroupID)

Here, the Soft quota is 700GB and hard quota is 920GB for GroupID and the group has already used about 297gb.

To get the usage in the specific group, you can use the following command

find /home/<caseID> -maxdepth 1 -type d -group <PI/class-Group> -exec du -sh {} \;   # for home

/scratch quota

There is a quota of 1TB in place for the /scratch/pbsjobs space. You may want to make room at your scratch space although they automatically get deleted after 14 days. Check your group scratch quota, similar to /home above with the following commands:

panfs_quota /scratch/pbsjobs  # for personal group usage

panfs_quota -G /scratch/pbsjobs # for group usage and quota

Identifying large scratch folders

To determine which /scratch/pbsjobs directories are holding data owned by your account, the following command is useful:

find /scratch/pbsjobs/ -user <caseid> -exec du -sh {} + 2>/dev/null > <output.log>

where 'du -sh {} +' examines the usage in the directories identified by 'find'.  The output may be optionally directed to a file as above: <output.log>. And because there are thousands of files in /scratch/pbsjobs that all have access restricted to only the owner, the error messages that occur when trying to access those directories is redirected by '2>/dev/null'. Sample output in <output.log> file is as follows:

35G     /scratch/pbsjobs/job.11142335.hpc

171G    /scratch/pbsjobs/job.12043224.hpc

2.7G    /scratch/pbsjobs/job.10529234.hpc

57G     /scratch/pbsjobs/job.11193469.hpc

91G     /scratch/pbsjobs/job.11190376.hpc

(Note: Based on the size of your scratch space, this command may take some time to complete)

If you want to delete the temporary files in the scratch space, please include the "rm" command in your job script as described in Scratch Space.

If you want to remove all the folders in the /scratch/pbsjobs that belong to you and you do not have any jobs running in the cluster, this is the command

find /scratch/pbsjobs -maxdepth 1 -user <CaseID> -name "*" | xargs rm -rf

Checking Usage and Capacity on Leased Storage 

/mnt/pan

Sometimes, you may want to view your mounted storage (e.g. /mnt/pan/<Userspace>)

$ pan_df -H /mnt/pan/Data<ID> 

output:

Filesystem                        1K-blocks   Used Total Available Use% Mounted on

panfs://192.168.223.10/pan                    146T  160T     5.7T  97% /mnt/pan/cryoem/

Note: -H gives decimal values TeraBytes instead of TebiBytes. 

For more readable format including soft and hard quota. Volume can be looked up at /scratch/users/admin/quotapan.lst which only gets updated a few times a day.

quotapan /pan/cryoem

output:

                                                 Space       Soft      Hard

Volume               BladeSet       RAID         Used       Quota      Quota      Status

/pan/cryoem           AS20      Object RAID6+  153.06 TB   150.00 TB   160.00 TB   Online

Note that when the hard quota is hit, usage needs to fall below 95% of the hard quota, for users to be able to write to this volume again.

/mnt/vstor

Check usage against the quota with quotarstor (similar to the other quota scripts). The command requires a unique string from the path to pull the correct record from the reference file:

For example, to see the quota for ajs215, use 'ajs215' (case insensitive):

$ quotavstor ajs215

Usage: quotavstor PI_groupID/LastName

        "name": "AJS215",

        "hard_limit": 11000000000000,

        "used_capacity_tb": 9.973,

/mnt/rds

From the login node, type:

df -H | grep <PI>

output:

192.168.223.xxx:/pool                       81T     0   81T   0% /mnt/rds/<PI>

192.168.223.xxx:/pool/xxxLab                61T   45T   17T  73% /mnt/rds/<PI>/xxxLab

192.168.223.xxx:/pool/xxxLab             60T   57T  3.8T  94% /mnt/rds/<PI>/xxxLab  # at 94% 

Important Notes: Once the usage is at 100%, you can't write to the disk and you will also get the "disk quota exceeded" error msg. Also, RDS storage can't be extended. You need to purchase another system. However, you can look into getting alternative additional spaces in /mnt/vstor. Please manage your disk space following the section "Managing Disk Usage" well before hitting the 100% limit.  

Also,  the total storage doesn't reflect the snapshots usage. In the example below, though the total space was 60.0TB, about 19 TB was allocated for snapshot.

NAME                                               AVAIL   USED    USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD

pool/xxxLab                                         0B    60.0T     19.2T   40.8T             0B         0B 

DUG utility

DUG is a utility similar to linux "du" that focuses on summarizing usage by group or owner. 

dug -h -u -n -t 1  <directory-path>   # get info of flags running "dug --help" ; Example of directory path "/mnt/pan/courses/sxg125_csds438"

output:

=================== Sub Directories ====================

/mnt/pan/courses/sxg125_csds438

                  sxg125  64K

...

                  jyf6  192K

                   Total  119G

Managing Disk Usage

Compressing Files

Important Notes:  Please use hpctransfer or hpcviz (i.e. ssh hpctransfer or ssh hpcviz) if you are running tar/zip command on huge files/directories.

Create a Test.tar.gz file (zipped file) out of Test directory using the command:

$tar -czf Test.tar.gz Test         #Test and Test.tar.gz have been used as an example

Make sure that you have both Test.tar.gz file and Test directory in your home directory. Now, remove the Test directory using rm command:

$rm -rf Test 

Later on if you want to unzipped your file to analyze data, you can use:

$tar -xzf Test.tar.gz

This will recreate a Test directory for you.

If you want to extract a single file or few file, you can do that using:

tar -xzvf delme.tar.gz delme/pop2000.481.txt

Here, only file pop2000.481.txt is extracted to delme directory from delme.tar.gz.

For parallel compression, refer to HPC pigz Guide.

If you want to view what are the files inside the .tar.gz file:

tar tvf delme.tar.gz

Deleting Files

It is very easy to Delete the /scratch job files before the job is completed, just add "rm -rf "$PFSDIR"/* " to the end of the job scripts after the cp command as showed:

#!/bin/bash

#SBATCH ...

...

cp -ru * $SLURM_SUBMIT_DIR  # Copy all the output files from the /scratch space to your working directory

rm -rf "$PFSDIR"/*                      # Empty the scratch space immediately to avoid quota violation

To delete the /scratch job files later:

find /scratch/pbsjobs -maxdepth 1 -user -name "*" | xargs rm -rf

Transferring Files to your local storage

Visit HPC site Transferring File @HPC

Purchasing Additional Storage

Please contact us with your requirements if you need to purchase more space at hpc-supportATcase.edu

Visit Access Policy.