Best Practices on Greene

Scheduling jobs in an efficient and fair manner can be a challenging task in a multi-user environment. Here we provide some recommendations.

What resources to ask for (and how much)?

When asking for compute resources in the batch script, never ask for more than you need.

There are two important reasons listed below. Imagine you need only 2 CPUs but request 10.

• • You will wait longer for job to start: It is easier for scheduler to find 2 CPUs than 10

  • You will make other users to not be able to run jobs: 8 cores will be sitting idle and no other users will be able to use them.

The same argument applies to other types of resources: RAM, GPUs (and potentially other 'TRES' in SLURM terminology).

Very often, even for parallel codes, it makes sense to request fewer CPU cores. Main example of this is -- your parallel job doesn't scale well. For example, imagine that using 12 cores instead of 4, you can reduce execution time from 1 hour (on 4 cores) to 45 minutes (on 12 cores). Yes, 45 minutes is faster, but you are using 3 times as many resources. Additionally, it may happen that your job with 4 cores will only wait in the queue for 5 minutes, but the 12-core job would need to wait for 20 minutes. This completely  offsets all the gains from using 8 more cores. In the end you are to judge, what is really needed, but you should always make this type of considerations when creating a job script. 

We recommend you don't specify a specific GPU unless needed. If it isn't very important to run your job on a particular GPU device (for example a powerful V100), then request any GPU with

And only when you are absolutely sure that you need a V100, use #SBATCH --gres=gpu:v100:1. V100s are often in high demand and average wait time for these GPUs is higher than for any other type of GPU devices. It may be (and very often is) beneficial to run your job on a less powerful accelerator, but spend less time waiting in the queue for your job to start.   

Do NOT run CPU heavy jobs on login nodes 

Login nodes designed use

•  Logging In

• Running low-compute operations like copying files, installing packages, etc

• Submitting compute jobs using Slurm (see Getting Started)

Important: check if you are wasting RAM and CPU!

Another useful command that allows you to better understand how resources were utilized by completed jobs is seff:

This example shows statistics on a completed job, that was ran with a request of 1 cpu core and 10Gb of RAM. While CPU utilization was 100%, RAM utilization was very poor -- only 2.2GB out of requested 10GB was used. This job's batch script should definitely be adjusted to something like #SBATCH --mem=2250MB

Check resources usage of a currently running job

Run job. When it is scheduled you can use squeue to figure out node name, where job is scheduled.

From login node, do

Take a look how fully you use CPUs and how much RAM your jobs are using.

For a GPU job also run

Take a look how much GPU processing power your job is using.

It may happen that your code does not scale well, and it is better to use 1 or 2 GPUs instead of 4

You can also take a look at GPU memory utilization.

RAM specification in sbatch file is for CPU RAM, not GPU memory!

Please request only as much RAM as your job needs!

Is my job scalable? How efficiently I use multiple CPUs/GPUs

Every code is different. Test it!

• • Run your job with 1 CPU, then 2 CPUs, then 4 CPUs, etc. Make a plot! How much improvements you get from adding more CPUs?

  • Run your job with 1 GPu, then 2 GPUs, then 4 GPUs, etc. Make a plot! How much improvements you get from adding more GPUs?

  • Most of our nodes don't have NVlink, which may make scaling across GPUs much less efficient comparing to systems which do

Why my jobs are queued?

To understand why your job is waiting in the queue you can run 

Last column of the output would indicate a reason. You can find out more about squeue output format from man squeue. 

The column "NODELIST(REASON)" in the end is job status due to the reason(s), which can be :

• • Priority: higher priority jobs exist

  • Resources: waiting for resources to become available

  • BeginTime: start time not reached yet

  • Dependency: wait for a depended job to finish

  • QOSMaxCpuPerUserLimit: number of CPU cores limit is reached

  • JobArrayTaskLimit: limit of tasks in the JobArray is reached

 For a more complete list of possible values of REASON, please refer to man squeue under the section JOB REASON CODES.

Limits on resources you can request

Within SLURM there are multiple limits defined on different levels and applied to different objects. Some of the important limits are listed here. 

Number of jobs per user

2000

Job lifetime

Limited to 7 days (168 hours), but you can request an extension by emailing HPC team (hpc@nyu.edu).

CPU, GPU, RAM

These limits depend on the time you request for the job: "short queue" (under 48 hours, or 2 days) or "long queue" (under 168 hours, or 7 days).
These limits may be updated by the HPC team, based on the cluster usage patterns. In order to obtain the up to date numbers you have to run the following command

In the output look at MaxWall to determine queue you are interested in (short or long). Now look under MaxTRESPU to find limits for CPU, RAM, GPU.

Here is an example of output you may get

How many CPU cores per GPU

These limits are frequently updated by the HPC team, based on the cluster usage patterns. 
Due to this, the numbers below are not exact and should only be used as general guidelines. 

Here are some of these limits:

From this table you can for example see, that a job asking for 8 V100 GPUs will not be queued. Another example is that requests for 2 V100s and 48 cores will also not be granted. 

How to find more information on my jobs?

Some other useful SLURM commands that can help to get information about running and pending jobs are

How busy is the cluster?

Please refer to the Systems Status page to see the number of jobs in the queue and other metrics. Often your jobs are queued for a simple reason -- cluster is very busy and there aren't enough resources available.

Best way to submit large number of similar jobs

The correct way to submit such jobs is to use the array job functionality of SLURM. This reduces load on the scheduler system.

Don't make your own loops to do this kind of work.

You can find a detailed description on how to submit such jobs here.

Web scraping, Data mining from websites

If you need to do web scraping, don't do that on Greene. It is not allowed due to security concerns and waste of CPU resources (most of the time a job would spend on downloading files, instead of using CPU). Please contact us, and we will advise on a better workflow for your project.

Error handling

We recommend using #!/bin/bash -e instead of plain #!/bin/bash, so that the failure of any command within the script will cause your job to stop immediately rather than attempting to continue on with an unexpected environment or erroneous intermediate data. It also ensures that your failed jobs show a status of FAILED in sacct output.

When a batch job is finished it produces an exit code (among other useful data). To view the error code of the job you can use: 

When reaching out to the HPC team asking for help with failing jobs, it is useful to find an exit code from the job at question. 

Check efficiency of your jobs

Please check on this page

SSH Issues

Some people may experience connection warnings while connection to Greene, and connections being terminated too soon. 

This can be addressed by entering the following into ~/ssh/config 

More information on SSH can be found on the SSH Tunneling page.