NERSC Guidelines
Getting Started
General
NERSC is a DOE funded computing facility housed at Lawrence Berkeley National Laboratory. The main cluster, Hopper, has over 150,000 compute cores and 200 TB of memory. Any information not found here is most likely available at www.nersc.gov, which is a really good resource.
About the group allocation
1.5 million hours for 2013 group allocation
Hours can be allocated to anyone within the group that has a NERSC account
Annual review cycle (deadlines)
Creating an account
Send the following information to the PI proxy for the group (currently Kyle):
Full first and last name
Preferred user name
Country of citizenship
Email address
Telephone number
The PI proxy can add a new user via NIM:
Select Actions -> Add/Revive User
Enter users last name
If user does not exist, then select "Add new user to NIM"
NIM (NERSC Information Management)
The NIM website, nim.nersc.gov, is where personal information for each account can be accessed. There you can check your account balance, change settings, etc.
New Users: First Steps
Setting your default shell
By default, a new user's shell is set to csh. This can by changed using the following:
Log onto NIM
Go to the "Actions" drop-down menu, select "Change Shell"
Select your shell for each machine
Setting rc and profile files
The following table gives the files that are write protected and the files that should be edited instead. For each, the edited file (.bashrc.ext) is automatically sourced by the protected file (.bashrc).
Access to VASP binaries
Contact the PI proxy to get access to the VASP binaries that have been compiled by the NERSC admins and optimized for their systems. The PI proxy should follow the steps at www.nersc.gov/users/software/applications/materials-science/vasp.
Once you have access to the VASP binaries, you can use them by loading the vasp module using module load vasp (this can go into your .bashrc.ext file).
As an alternative, we have compiled our own VASP binaries. These should be located in the group shared directory.
Information for Users
Group shared directory
All members of the group have read/write/exe access to /global/homes/k/kylemich/share so that common files and programs can be stored there. *Please keep in mind that this is in Kyle's home directory, so keep the files small and delete those that are not necessary whenever possible.*
Common executables, including VASP, are located in /global/homes/k/kylemich/share/hopper/bin. It would be a good idea to include this in your path using (in the bash shell) export PATH=$PATH:/global/homes/k/kylemich/share/hopper/bin, which could also be set in your .bashrc.ext file.
Common commands
ssh username@hopper.nersc.gov to login to hopper, where username is your own username
qstat or qstat -u username to view the entire queue or to the view the jobs in the queue for username
qsub jobname to submit the job script in jobname
module list to view all modules that are currently loaded
module avail to view all modules that are available
module load modulename to load the module modulename
Available software at NERSC
There are a number of programs that have been installed by the NERSC admins. To see a complete list, go to http://www.nersc.gov/users/software/applications.
Queueing policies
Become familiar with the queue policies at http://www.nersc.gov/users/computational-systems/hopper/running-jobs/queues-and-policies. Note that you can explicitly submit a job as low priority and only be charged for half the hours that you use (although it will take longer to get through the queue). You can also submit as a high priority job to move through the queue more quickly, although you will be charged for twice the hours that you use.
It is possible to bundle jobs together and, assuming that you reach the medium job limit, be charged for only 60% of the hours that you use. If you do bundle jobs, try to ensure that each will finish at nearly the same time to avoid wasting hours when some jobs have finished but others are still running.
Acknowledging NERSC
Please add the following acknowledgment to any papers you write that use NERSC resources. This is very important!
This research used resources of the National Energy Research Scientific Computing Center, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231.
Submission scripts
General information
Each node has 24 cores.
When bundling jobs, mppwidth should be the total number of cores and each exe call should use some fraction of this (see example script)
For more information, see http://www.nersc.gov/users/computational-systems/hopper/running-jobs/example-batch-scripts.
Single job
#!/bin/bash
#PBS -q regular
#PBS -l mppwidth=24
#PBS -l walltime=12:00:00
# Set the executable and number of cores per job
exe=vasp
nproc=24
# Move to working directory and submit job
cd $PBS_O_WORKDIR
aprun -n $nproc $exe >& stdout &
# Wait for completion
wait
Bundled job
#!/bin/bash
#PBS -q regular
#PBS -l mppwidth=48
#PBS -l walltime=12:00:00
# Set the executable and number of cores per job
exe=vasp
nproc=24
# Move to working directory
cd $PBS_O_WORKDIR
# Move to first directory and submit
cd DIR1
aprun -n $nproc $exe >& stdout &
# Move to second directory and submit
cd ../DIR2
aprun -n $nproc $exe >& stdout &
# Wait for completion
wait