SSH Configuration and X11 Forwarding
Avoiding Man in the Middle Warning
If you see this warning:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!Someone could be eavesdropping on you right now (man-in-the-middle attack)!It is also possible that a host key has just been changed.
Do not be alarmed - this is an issue that occurs because Greene has multiple login nodes (log-1, log-2, and log-3) that greene.hpc.nyu.edu resolves to.
To avoid this warning, you can add these lines to your SSH configuration file. Using your favorite editor, open the file "~/.ssh/config" and place the following lines in it:
Host *.hpc.nyu.edu StrictHostKeyChecking no UserKnownHostsFile /dev/null LogLevel ERRORSSH Timeout Errors
Some people may experience connection warnings while connecting to Greene, and connections being terminated too soon.
This can be addressed by entering the following into ~/.ssh/config
# Increase alive interval host *.hpc.nyu.edu ServerAliveInterval 60 ForwardAgent yes StrictHostKeyChecking no UserKnownHostsFile /dev/null LogLevel ERRORSSH Tunneling (Mac, Linux)
Setting up your workstation for SSH tunneling will make logging in and transferring files significantly easier, and installing and running an X server will allow you to use graphical software on the HPC clusters. X server is a software package that draws on your local screen windows created on a remote computer (such as an NYU HPC).
Linux users have X set up already. Mac users can download and install XQuartz.
1. Set up a tunnel you can reuse
To avoid repeatedly setting up a tunnel, you can write the details of the tunnel into your SSH configuration file. Using your favorite editor, open the file "~/.ssh/config" and place the following lines in it:
# first we create the tunnel, with instructions to pass incoming# packets on ports 8027 and 8028 through it and to specific locationsHost hpcgwtunnel HostName gw.hpc.nyu.edu ForwardX11 no LocalForward 8027 greene.hpc.nyu.edu:22 User <Your NetID>
# next we create an alias for incoming packets on the port# The alias corresponds to where the tunnel forwards these packets
Host greene HostName localhost Port 8027 ForwardX11 yes StrictHostKeyChecking no UserKnownHostsFile /dev/null LogLevel ERROR User <Your NetID>
Create this file/directory In case you don't have it. Make sure that ".ssh" directory has correct permissions (it should be "700" or "drwx------"). If needed, set permissions with:
chmod 700 ~/.sshYou may also need to setup permissions on your local computer
chmod 700 $HOMEchmod 700 $HOME/.ssh## to be safe, all files inside ~/.ssh should be set 600chmod 600 ~/.ssh/*
2. Start the tunnel
To create the tunnel, ssh to it with the following command:
$ ssh hpcgwtunnelImportant: you must leave this window open for the tunnel to remain open. It is best to start a new terminal window for subsequent logins.
3. Log in via the tunnel
Open a new terminal window and use ssh to log in to the cluster, as shown below.
$ ssh greene # to get to Greene clusterNote that you must use the short name defined above in your .ssh/config file, not the fully qualified domain name:
Creating a once-off tunnel.
Alternatively, you can set up a once-off tunnel without editing .ssh/config by running the following command:
$ ssh -L 8027:greene:22 NetID@gw.hpc.nyu.edu # to set up a tunnel$ ssh -Y -p 8027 NetID@localhost
This is the equivalent to running "ssh hpcgwtunnel" in the reusable tunnel instructions, but the port forwarding is specified on the command line.
Tunneling (Windows)
Step 1: Creating the tunnel
First open Putty and prepare to log in to gw.hpc.nyu.edu. If you saved your session during that process, you can load it by selecting from the "Saved Sessions" box and hitting "Load". Don't hit "Open" yet!
Under "Connection" -> "SSH", just below "X11", select "Tunnels
3. Write "8026" (the port number) in the "Source port" box, and "greene.hpc.nyu.edu:22" (the machine you wish to tunnel to - 22 is the port that ssh listens on) in the "Destination" box.
4. Click "Add". You can repeat step 3 with a different port number and a different destination. If you do this you will create multiple tunnels, one to each destination.
5. Before hitting "Open", go back to the "Sessions" page, give the session a name ("hpcgw_tunnel") and hit "Save". Then next time you need not do all this again, just load the saved session.
6. Hit "Open" to login in to gw.hpc.nyu.edu and create the tunnel. A terminal window will appear, asking for your login name (NYU NetID) and password (NYU password). Windows may also ask you to allow certain connections through its firewall - this is so you can ssh to port 8026 on your workstation - the entrance to the tunnel.
Note: You can add other NYU hosts to the tunnel by adding a new source port and destination and clicking "Add". For example, you could add "Source port = 8025" and "Destination = EXAMPLE.hpc.nyu.edu:22", then press "Add". You would then perform Step 2 (below) twice - once for greene on port 8026 and once for an example server on port 8025.
Using your SSH tunnel: To log in via the tunnel, first the tunnel must be open. If you've just completed Step 1, it will be open and you can jump down to "Step 2: Logging in via your SSH tunnel". If you completed Step 1 yesterday, and now want to re-use the tunnel you created, first start the tunnel:
Starting the tunnel: During a session, you need only do this once - as long as the tunnel is open, new connections will go over it.
Start Putty.exe (again, if necessary), and load the session you saved in settings during procedure above
Hit "Open", and log in to the bastion host with your NYU NetID and password. This will create the tunnel.
Step 2: Logging in via your SSH tunnel
Start the second Putty.exe. In the "Host Name" box, write "localhost" and in the "Port" box, write "8026" (or whichever port number you specified when you set up the tunnel in the procedure above). We use "localhost" because the entrance of the tunnel is actually on this workstation, at port 8026.
2. Go to "Connections" -> "SSH" -> "X11" and check "Enable X11 forwarding"
3. Optionally, give this session a name (in "Saved Sessions") and hit "Save" to save it. Then next time instead of steps 1 and 2 you can simply load this saved session.
4. Hit "Open". You will again get a terminal window asking for your login (NYU NetID) and password (NYU password). You are now logged in to the HPC cluster!
X11 Forwarding
In rare cases when you need to interact with GUI applications on HPC clusters, you need to enable X11 forwarding for your SSH connection. Mac and Linux users will need to run the ssh commands described above with an additional flag:
ssh -Y <NYU_NetID>@greene.hpc.nyu.eduHowever, Mac users need to install XQuartz, since X-server is no longer shipped with the macOS (you can find it here).
Windows users will also need to install X server software. We recommend two options out there. We recommend installing Xming. Start Xming application and configure PuTTY to support X11 forwarding:
Stata X11 Forwarding Example
Please make sure an X server is installed and running on your workstation before running a program like Stata on the HPC cluster.
Establish connection
Note: example given is for Stata, but the same will apply for other programs
For Linux and Mac users
If your workstation or laptop is inside the NYU-Net, you can login directly to the HPC cluster login node using SSH. Open up a terminal window and issue the following SSH command. Make sure to specify the -Y option with the SSH command.
The same instructions apply if you are outside the NYU-Net, but you use VPN to access the NYU Network.
$ ssh -Y <net_id>@greene.hpc.nyu.eduStart an interactive session with X11 forwarding enabled with the following command:
$ srun --x11 --pty /bin/bashYou will be redirected to one of the compute nodes. Load "stata" module:
$ module avail stata-------- /share/apps/modulefiles ------ stata/16.1 stata/17.0$ module load stata/17.0
Finally, run Stata with X forwarding and Multiprocessing:
$ xstata-mpNote: Make sure you exit the node once you are done using Stata to free up the node. You can exit the node by typing the command $exit
For Windows Users
You will need Xming and Putty to run Stata on Windows.
Step 1: Download Xming and PuTTY if you don't have them. You will require both.
Step 2: Update the settings in PuTTY as shown in the image above.
Step 3: Log into the HPC cluster with PuTTY
Step 4: Make sure Xming is running.
Step 5: Type in the following commands in your PuTTY session:
$ srun --x11 --pty /bin/bashLoad the Stata module:
$ module load stata/17.0Run Stata:
$ xstata-mpThe Stata window should pop up as below:
Note: Make sure you exit the node once you are done using Stata to free up the node. You can exit the node by typing the command $exit