Getting Started on Greene

Accessing the Greene Cluster

There are several ways one can use to access the cluster. We only show one example below. Please refer to the Accessing HPC page to read about other options.

SSH using Terminal or Command Prompt

Simply open a terminal (Linux, Mac) or the Command Prompt (Windows 10) and enter the commands:

ssh <NetID>@gw.hpc.nyu.edu ## you can skip this step if you  are on the NYU Network or using the NYU VPN
ssh <NetID>@greene.hpc.nyu.edu

NOTE: When you are asked to enter password, just type it (letters will not be displayed), and then hit "Enter"

Computations on Greene

There are several types of jobs that can be run on the Greene cluster. The traditional type of workload for HPC is called a "batch job" - where a job is sent to the system to execute a function or program without further user input or interaction. 

Users can submit batch jobs to the SLURM scheduler (a software which manages the jobs queue on Greene). 

For other job types please read here.

Writing Your Code

Using a text editing program such as vim, nano, or emacs, create a python script. Below is an example called 'hello-world.py':

import osprint("Hello from the compute node", os.uname()[1])

The script loads the os library in order to print out the name of the server that the script is run on. 

Step 1. Allocating Resources Using SLURM

To run our python script we need to submit a batch job to SLURM to allocate the necessary compute resources. SLURM expects file in the following format in order to execute the job. The file contains both commands specific for SLURM to interpret as well as programs for it execute. Below is a simple example of a batch job to run the python script we just created, the file is named "hello-python.sbatch":


#!/bin/bash
#SBATCH --nodes=3                        # requests 3 compute servers
#SBATCH --ntasks-per-node=2              # runs 2 tasks on each server
#SBATCH --cpus-per-task=1                # uses 1 compute core per task
#SBATCH --time=1:00:00
#SBATCH --mem=2GB
#SBATCH --job-name=python-hello-world
#SBATCH --output=python-hello-world.out

module purge
module load python/intel/3.8.6

python ./hello-world.py

You then submit the job with the following command:

$ sbatch hello-python.sbatch

The command will result in the job queuing as it awaits resources to become available (which varies on the number of other jobs being run on the cluster). You can see the status of your jobs with the following command:

$ squeue -u $USER

There are many more ways how you can allocate resources in SLURM. Please read here for more details.

Reading the Job Output

Once the job has completed, you can read the output of your job in the python-hello-world.out file. This is where logs regarding the execution of your job can be found, including errors or system messages. You can print the contents to the screen from the directory containing the output file with the following command:

$ cat python-hello-world.out

Python natively will only run on a single CPU core. This is why the output contains only one printed statement, even though we requested several nodes and 2 tasks per node. Your output may look something like the following:

Hello from the compute node cs240.nyu.cluster

We can make our script use all the resources allocated by SLURM. We can accomplish this by parallelizing the batch script, which we will do in the next step. 

Step 2. Running Your Script on Multiple Nodes/Cores 

After we allocated multiple nodes resources (Step 1), we can run scripts on multiple nodes.
One of the most popular software tools to run code on multiple nodes/cores at HPC is "Open MPI". 

Here is a simple example of using Open MPI with the job we ran before


#!/bin/bash
#SBATCH --nodes=3                        # requests 3 compute servers
#SBATCH --ntasks-per-node=2              # runs 2 tasks on each server
#SBATCH --cpus-per-task=1                # uses 1 compute core per task
#SBATCH --time=1:00:00
#SBATCH --mem=2GB
#SBATCH --job-name=python-hello-world
#SBATCH --output=python-hello-world.out

module purge
module load python/intel/3.8.6
module load openmpi/intel/4.0.5           # NOTE: we are loading "openmpi" module here

srun python ./hello-world.py            # NOTE: we can use "srun" or "mpirun" in front of python command

The mpirun command will execute our hello-world.py script using each requested node/core. You can see it in the output:

Hello from the compute node cs240.nyu.cluster
Hello from the compute node cs239.nyu.cluster
Hello from the compute node cs240.nyu.cluster
Hello from the compute node cs239.nyu.cluster
Hello from the compute node cs241.nyu.cluster
Hello from the compute node cs241.nyu.cluster

Using mpirun, the batch job ran the hello-world.py script on three nodes on two separate CPU cores.

Step 3. Parallelizing the Script

Now you can modify your program, so it can leverage available resources. 

As an example, in Python, you can use package mpi4py. 

First install package by executing in terminal

cd /scratch/<NetID>
module load python/intel/3.8.6
python3 -m venv ./test_venv
source ./test_venv/bin/activate
module load openmpi/intel/4.0.5
pip install mpi4py

Create a new python file 'hello-world_parallel.py' (more examples can be found here)

import osfrom mpi4py import MPI
comm = MPI.COMM_WORLDrank = comm.Get_rank()size = comm.Get_size()host = os.uname()[1]print(f"hello from process {rank} on host {host}")

data = (rank+1)**2data_collected = comm.gather(data, root=0)
if rank == 0:    sum = 0    for i in range(len(data_collected)):        sum = sum + data_collected[i]    print(f"on rank {rank} I've calculated sum = {sum}")

MPI.Finalize()
# compare the result to:# sum([i**2 for i in range(N+1)])# where N is a number of MPI ranks you are using

And modify batch file


#!/bin/bash
#SBATCH --job-name=pythonMPIexample
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=2
#SBATCH --cpus-per-task=1
#SBATCH --mem=2GB
#SBATCH --time=01:00:00

module purge
module load python/intel/3.8.6
module load openmpi/intel/4.0.5

source /scratch/<NetID>/test_venv/bin/activate
srun /scratch/<NetID>/test_venv/bin/python ./hello-world_parallel.py

GPU Jobs

To request one GPU, use SBATCH directives in job script

#SBATCH --gres=gpu:1
#SBATCH --gres=gpu:v100:1  ## To request specific GPU (v100 or rtx8000)

We recommend to use Singularity images containing pre-installed software for GPU jobs. You can use overlay to install additional packages as described here. Here is an example with an overlay image containing Miniconda and  PyTorch  'overlay-10GB-400K-tensorflow-pytorch.ext3'

srun --pty -c 2 --mem=10GB --gres=gpu:rtx8000:1 /bin/bash # request an RTX8000 GPU node
SINGULARITY_IMAGE=/scratch/work/public/singularity/ubuntu-20.04.1.sif

OVERLAY_FILE=/scratch/work/public/examples/greene-getting-started/overlay-15GB-500K-pytorch.ext3## Note: this overlay file has PyTorch installed using command
## conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c nvidia
## Run interactively on the GPU node with singularity exec --nvsingularity exec --nv --overlay $OVERLAY_FILE $SINGULARITY_IMAGE /bin/bashsource /ext3/miniconda3/bin/activate
## Exit the Singularity sessionSingularity> exit
## Exit interactive sessionexit

Now we can run SLURM job script 'run-test.SBATCH', that will start our Singularity Image and call the 'torch-test.py' script.
Note: If you started an interactive job with singularity above, exit it first. 

sbatch /scratch/work/public/examples/greene-getting-started/run-test.SBATCH

Further Examples/Instructions

There are other kinds of jobs that can be run on Greene like interactive jobs and jobs arrays. The links below give further information on the usage of SLURM.

Examples available on Greene

/scratch/work/public/examples

Graphical User Interface (GUI) Tools (using Open OnDemand)

Open OnDemand is a tool that allows users to launch Graphical User Interfaces (GUIs) based applications are accessible without modifying your HPC environment. 

Login

Software available through Open OnDemand 


Software

You can view software available on Greene as well as instructions on installing new software at the Greene Software page.

Best Practices

Please read