Accessing HPC

Introduction

There are several ways to interact with the Greene HPC cluster. Similar to other Linux clusters, the most common method of connection is via a Command Line Interface (CLI). A CLI is a program that allows you to create and delete files, run programs, and navigate through directories and files via a typed prompt. On Mac, the built-in CLI application is called Terminal. While some Windows 10 machines support a Linux Subsystem, which allows for similar functionality, a popular tool used to connect to a Linux server is a free application called PuTTY. 

The following sections will outline basic ways to connect to the Greene cluster. Access to the clusters is primarily handled via the Secure Shell Protocol, or ssh. Below we outline ways to connect via ssh on Mac, Linux, and Windows machines.

Remote Connections with the NYU VPN & HPC Gateway Server

If you are connecting from a remote location that is not on the NYU network (your home for example), you have two options: 

• VPN Option: set up your computer to use the NYU VPN. Once you've created a VPN connection, you can proceed as if you were connected to the NYU net.  

• Gateway Option: go through our gateway servers (example below). Gateways are designed to support only a very minimal set of commands and their only purpose is to let users connect HPC systems without needing to first connect to the VPN.

You do not need to use the NYU VPN or gateways if you are connected to the NYU network (wired connection in your office or WiFi) or if you have VPN connection initiated. In this case you can ssh directly to the clusters.

Quickstart

Please see this page to see how to configure your ssh client. For Linux and Mac machines, the configuration file is the ~/.ssh/config file on your computer. These are the basic lines that should be added to your ~/.ssh/config file:

Host greene.hpc.nyu.edu dtn.hpc.nyu.edu gw.hpc.nyu.edu

  StrictHostKeyChecking no

  ServerAliveInterval 60

  ForwardAgent yes

  UserKnownHostsFile /dev/null

  LogLevel ERROR

Host hpcgwtunnel

  HostName gw.hpc.nyu.edu

  ForwardX11 no

  StrictHostKeyChecking no

  LocalForward 8027 greene.hpc.nyu.edu:22

  UserKnownHostsFile /dev/null

  User <Your NetID>

Host greene

  HostName localhost

  Port 8027

  ForwardX11 yes

  StrictHostKeyChecking no

  UserKnownHostsFile /dev/null

  LogLevel ERROR

  User <Your NetID>

Configuring SSH Config on Windows Machines

For Windows, you can now follow a similar configuration setup. Using Powershell you can use the following to create and open a Windows SSH config file:

# Create the config file with Powershell

New-Item -Path $HOME\.ssh\config -ItemType File

# Open config File with Notepad

C:\WINDOWS\System32\notepad.exe $HOME\.ssh\config

You can now ssh using the cmd terminal.

VPN Option

If you're on the VPN or the main NYU network on campus you can SSH directly to greene like so:

ssh <Your NetID>@greene.hpc.nyu.edu

And then authenticate with your password. 

Tunneling (Gateway) Option

If you want to use ssh tunneling via the gateway it's slightly different, though - then you need to add this to your ~/.ssh/config file:

And then ssh like so:

1. Open a terminal

2. Connect to the gateway via the ssh command:
   ssh hpcgwtunnel

3. Authenticate with your password and Duo MFA

4. Open a second terminal

5. Connect to the Greene login nodes via the ssh command:
   ssh greene

6. Authenticate

Command Line Interface (Use Terminal)

Mac & Linux Access

To connect to the gateway servers, simply open a terminal application and enter the following command:

After typing in your password you will be logged in to the cluster. Once this connection is established, you can make one more hop and connect to one of the HPC clusters:

Windows CMD

Windows 10 users have several options. First, the CMD program on updated Windows 10 machines should contain an ssh client, allowing you to log into Greene or Hudson the same way as with a Linux terminal. 

Windows WSL2

If you run Windows 10, you can install WSL, and then install Ubuntu or other Linux distribution (for example, from Microsoft Store). You will have a fully functional Ubuntu with terminal and can connect to cluster using instructions provided above for Linux/Mac users 

• Instructions on WSL installation: https://docs.microsoft.com/en-us/windows/wsl/install-win10

Tips:

• One of many options to get terminal that support tabs, etc. is to install 'Windows Terminal' from Microsoft Store.

• If you are using WSL 2 (Windows subsystem for Linux), you may not be able to access internet when Cisco AnyConnect VPN, installed from exe file, is activated. A potential solution: uninstall Cisco AnyConnect and install AnyConnect using Microsoft Store, and then setup new VPN connection using settings described on IT webpage.

Setting Up SSH Keys

Instead of typing your password every time you need to log in, you can also specify an ssh key.

• Only do that on the computer you trust

• Generate ssh key pair (terminal in Linux/Mac or cmd/WSL in Windows): https://www.ssh.com/ssh/keygen/

• Note the path to ssh key files. Don't share key files with anybody - anybody with this key file can login to your account

• Log into cluster using regular login/password and then add the content of generated public key file (the one with .pub) to $HOME/.ssh/authorized_keys on cluster

• Next time you will log into cluster no password will be required. 

For additional recommendations on how to configure your SSH sessions, see this page.

PuTTY (Windows)

There are many SSH clients for Windows OS, but we recommend using PuTTY SSH if you have not already. Once it is installed, launch PuTTY and configure new session "Session" category as in the screenshot below: 

Here we are instructing PuTTY to connect to host gw.hpc.nyu.edu on port 22 using SSH protocol (note, that this interface allows you to save this connection configuration for future). Just like for Linux and Mac users, if you are connecting from the outside of NYU network, you need to go through the gateway servers.

Once you click "Open", a terminal window with prompt for password will pop up. Enter your NetID password and you should be authorized on the gateway server. Gateways are designed to support only a very minimal set of commands and their only purpose it to let users access HPC systems. Once you are there type in an ssh command that will let you connect to Greene cluster :

A new command line interface window will open up that prompts you for your password on the gateway server, from there you can connect to Greene by entering the following:

Open OnDemand (Web-based Graphical User Interface)

Open OnDemand is an open source project funded by the National Science Foundation (NSF). Open OnDemand is designed to create easier access to users to interface with HPC systems. Originally developed by Ohio Supercomputer Center (OSC), used by many universities around the world, and now servicing the NYU Greene HPC cluster.

Open OnDemand has a variety of convenient tools to manage files, access the command line, manage and monitor jobs, and launch interactive applications, such as Jupyter Notebooks, RStudio sessions, and even full Linux Desktops. 

Features Include:

• Easy file management - upload and download files, view HTML and pictures without downloading

• Command-line shell access without any SSH client locally installed

• Job management and monitoring

• Full Linux desktop experience without X11

• Interactive Apps such as JupyterHub and RStudio without the need for port forwarding

Open OnDemand (OOD) is accessible to all users with a valid NYU HPC account while on-campus network or through a VPN.

To access OOD visit: https://ood.hpc.nyu.edu (VPN Required)

Access the Shell

Under the clusters menu you can select the Greene Shell Access option to access the Linux shell. No local SSH client is required.

Interactive Applications

GUI based applications are accessible without the need for port or X11 forwarding. Select the Interactive Apps menu, select the desired application, and submit the job based on required resources and options. 

Troubleshooting Connections to Open OnDemand

A common issue that can occur is receiving an error that the Open OnDemand page cannot be reached. Sometimes this can indicate that the service is down, but often this is an issue with the the local browser cache. You can test this by opening a private browser window and seeing if https://ood.hpc.nyu.edu will load. If it does, try deleting the cache for https://ood.hpc.nyu.edu in your browser history to resolve this issue.

In Chrome, this can be done by navigating to this page in your settings:

chrome://settings/content/all?searchSubpage=ood&search

The link above will automatically search for the Open OnDemand site data and cookies. You can then simply click on the trashcan icon to delete the site cache.

Once done, try navigating again to https://ood.hpc.nyu.edu and the site should load. For other issues please email hpc@nyu.edu.