JupyterHub

The NYU HPC team, in close collaboration with course instructors and departmental educational technologists, maintains and provides access to a centralized JupyterHub environment to support courses using Google Cloud Platform (GCP)

Benefits of JupyterHub on GCP

• • Use JupyterHub on GCP allows to support high availability

  • No need for HPC account (NYU email is sufficient)

  • Helps separate research HPC environment and teaching JupyterHub environment

  • Self service approach allowing instructors to control various parameters of deployment dynamically

What is provided

• • R, Python, Julia kernels.

  • Tools like SQL, MongoDB, Spark, TensorFlow, PyTorch can be used

  • Custom environments for packages installation (using conda)

  • IDE options: Classical Notebook, JupyterLab, RStudio

  • Designated storage to share large data files with students

  • (large; writable only by instructors/TAs)

  • Persistent storage for students’ home directory (limited size)

  • Sync from github private repositories

  • NBgrader (only in Classical Notebook IDE option)

    • How to create assignments, setup auto-grading, and more: link

    • Watch video about NBgrader use

  • GPUs are available as an option

What is controlled by an instructor

• • List of authorized students (whitelist)

    • May be set to remain open for a limited time duration

  • List of admins/TAs

  • IDE for student pods: Classical Notebook, JupyterLab, RStudio

  • Persistent storage need for students

  • RAM for student pods

  • GPU allocation if needed for student pods

  • Link to github repository

  • Custom R/Python conda environments created by instructors themselves.

  • An instructor can specify a version of R or Python while creating an environment

Instructions and Support

• To access instructions on how to use the system

  • • Submit intake form and wait for an approval

    • Wait for your class environment to become active

    • Instructions will be available in settings webiste

• • NYU group (accessible only to Instructors and TAs) is used to ask/answer/read questions and answers

  • You will be invited to the group, when your class is created

  • Link to support group

  • In cases of unusual specific requirements, we will work with you one-on-one

  • The HPC team will support designated/named instructors/TA’s.

  • In any case, HPC team can't respond to questions from individual students (hundreds of them!)

Who is paying

• • For the current and the next semester, costs will be covered by NYU IT Research Technology budget. This may change in future semesters.

  • Be considerate while requesting more resources (RAM, GPU, etc.), as this will increase the cost. You can estimate the cost using this app

Intake form

• • Please submit the course in-take form if you would like to use the service

  • Link to Intake Form

Maintenance

• • If significant changes to the environment are necessary, work be done during the semester recess

Usage & cost Monitoring

• • In case an actual utilization goes significantly above your estimation, we may need to contact you and figure out whether to continue with the current config

Decommissioning

• • Student environment and data will be removed approximately 2 weeks after end of semester

  • Instructor environment and data may be kept active between semester if requested and approved

  • Save files in shared and instructor home directories if needed

  • Hint: in order to easily save a directory, you can tar it in terminal and then download from Jupyter/RStudio UI interface

  • tar -czvf name-of-archive.tar.gz /path/to/directory-or-file

  • Save list of packages and their versions

    • If you installed all the packages using conda install (and nothing using 'pip install' or "install.packages") you can use

      • conda list -e > requirements.txt  ## saves list of packages installed by conda

      • conda create --name <env> --file requirements.txt

    • If you did install anything with 'pip install', use the following to command to track both: packages installed by conda and by pip

      • conda env export > environment.yaml ## saves to file info about env name, conda packages, pip packages

      • conda env create -f environment.yaml

    • If you did use install.packages

      • Either keep track of such packages yourself or save list of packages and their versions with this command

write.csv(data.frame(installed.packages())[c("Package", "Version")], "r_packages.txt")

• • • • Then install those manually in the next semester.

    • A more automatic way would be to use the renv package.

    • Make sure to download and save file requirements.txt/environment.yaml/r_packages.txt file !

Who is using now

• • look here

FAQ

• Please look for info on settings site provided to instructors/TAs