Jupyter

Jupyter is a very useful web browser IDE for Python that is base on notebook format, similar to Mathematica or R markdown. Jupyter can be spawned on the compute node and the webpage can be accessed from the user through the SSH tunnel. Jupyter is a Python package, so the execution would require a Python module load, what we suggest is python/3.6.6 as that one has more complete packages, including tensorflow.

There are three ways to get it running.

1. Connect to the cluster using 2 terminals: 1) to request an interactive node and to start jupyter, 2) to establish the ssh tunnel to the compute node. Once both terminals are set, open the browser on your local computer using the open jupyter port.

2. Use the X2Go or other X11 access to the cluster, then request an interactive node with X11 (refer to HPC Guide to Visual Access). On this compute node, start jupyter and subsequently the browser to access jupyter.

   1. (Note: If you need access to GPU nodes, srun --x11 -p gpu --gres=gpu:1 --pty bash)

   2. Through the OnDemand Web Portal (https://ondemand.case.edu)

   3. You could do `jupyter notebook > messages.log 2>&1 &` to capture all output to file. Then you could continuously display the contents with `tail -f messages.log` if you want to monitor what is going on, or stop that with ^C and use the shell.

This is the first method: Cluster graphics

• 1. Login to the cluster i.e. ssh <CaseID>@markov.case.edu

  2. Request a compute node. You can select a specific partition if necessary. The following provides default setting, including 10 hour duration.

     1. srun --pty bash

  3. You will be assigned a compute node. Now, Load python module

• • 1. module swap intel gcc

    2. module load python/3.6.6

• 4. Start Jupyter on the interactive node. 

• • 1. $ jupyter notebook --no-browser --ip=$(hostname -i) --port=9999

    2. [I 09:04:54.059 NotebookApp] Serving notebooks from local directory: /home/<CaseID>

    3. [I 09:04:54.060 NotebookApp] The Jupyter Notebook is running at:

    4. [I 09:04:54.060 NotebookApp] http://192.168.209.62:9999/?token=89aa52a04b834866361724d69382a33a9872405a6070bb1b

    5. [I 09:04:54.060 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

    6. [C 09:04:54.065 NotebookApp]

    7. Copy/paste this URL into your browser when you connect for the first time, to login with a token:

    8. http://192.168.209.62:9999/?token=89aa52a04b834866361724d69382a33a9872405a6070bb1b

• 5. In Linux/Mac, open a new terminal and create a tunnel to the Jupyter notebook using SSH. You can do the same using Putty on Windows.

  6. <ip-address> is the IP address from the interactive node (hostname -i), and also the bolded italic part above. A prompt will not appear after entering your password because of the "-N" option, but the port will be forwarded.

• • 1. ssh -N -L 9999:<ip-address>:9999 <caseID>@markov.case.edu

• 7. Open a web browser from your PC and navigate to http://localhost:9999 and enter the token number printed in your first terminal (<ip>:9999/?token=<token-number>) when prompted.

Here's the step-by-step instruction of the first method on windows using putty:

Accessing jupyter notebook:

# access command line: windows + R, cmd

ssh -X <CaseID>@markov.cwru.edu

srun --pty bash

module swap intel gcc

module load python/3.7.0

jupyter notebook --no-browser --ip=$(hostname -i) --port=9999

# open another command line: windows + R, cmd

ssh -N -L 9999:<ip-address>:9999 <CaseID>@markov.case.edu  # port forwarding

# enter password (will not update with any lines)

# access new internet window and enter

http://localhost:9999

# enter token number from first cmd

Note: SSH port forwarding is a mechanism in SSH for tunneling application ports from the client machine to the server machine, or vice versa. 

SSH port forwarding command "ssh -L ..." opens a connection to the markov.case.edu jump server, and forwards any connection to port 9999 on the local machine to port 9999 on <ip-address>.

This is the second method: X2go Desktop

• 1. Login to the cluster i.e. ssh into markov.case.edu with X-Forwarding with -X flag.

  2. Request a compute node. You can select a specific partition if necessary.

     1. srun --x11 -t 2:00:00 -c 2 --mem=8gb --pty bash

  3. You will be assigned a compute node. Now, Load python module.

• • 1. module swap intel gcc

    2. module load python/3.6.6

• 4. Start jupyter notebook. 

• • 1. jupyter notebook --ip=127.0.0.1 &

• 5. output:

• • 1. [I 16:42:11.550 NotebookApp] Serving notebooks from local directory: /home/<CaseID>

    2. [I 16:42:11.550 NotebookApp] The Jupyter Notebook is running at:

    3. [I 16:42:11.550 NotebookApp] http://127.0.0.1:8888/?token=<token>

    4. [I 16:42:11.550 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

    5. [W 16:42:11.555 NotebookApp] No web browser found: could not locate runnable browser.

    6. [C 16:42:11.555 NotebookApp]

    7. Copy/paste this URL into your browser when you connect for the first time,

    8. to login with a token:

    9. http://127.0.0.1:8888/?token=<token>

• 6. Press Enter and open the firefox (ignore the warnings). First unload the base module as it is conflicting with firefox

• • 1. module unload base

    2. firefox &

• 7. Copy/paste this bolded URL into your browser when you connect for the first time, to login with a token as instructed above.

OnDemand Web Portal

• 1. Navigate to http://ondemand.case.edu and from the Interactive Apps menu, Start the 'Markov Desktop' app

  2. In the desktop, open a terminal

  3. Use the usual commands

     1. module swap intel gcc

     2. module load python/3.6.6

  4. Start Jupyter:

• • 1. jupyter notebook --ip=127.0.0.1 &

• 5. Open a browser (use the menu if you like, or open from another terminal)6

  6. Copy the bold URL with token to the firefox url input:

• • 1. http://127.0.0.1:8888/?token=​<token-string>