Singularity with Conda

What is Singularity?

Singularity is a free, cross-platform and open-source program that creates and executes containers on the HPC clusters. Containers are streamlined, virtualized environments for specific programs or packages. Singularity is an industry standard tool to utilize containers in HPC environments. Containers allow for the support of highly specific environments and further increase scientific reproducibility and portability. Using Singularity containers, researchers can work in the reproducible containerized environments of their choice can easily tailor them to their needs.

Instructions for Jupyter Notebook Kernels

If you plan to use your Singularity overlay with the Open OnDemand Jupyter Notebooks, please see this page.

Using Singularity Overlays for Miniforge (Python & Julia)

Preinstallation Warning

If you have initialized Conda in your base environment (your prompt on Greene may show something like (base) [NETID@log-1 ~]$) then you must first comment out or remove this portion of your ~/.bashrc file:

The above code automatically makes your environment look for the default shared installation of Conda on the cluster and will sabotage  any attempts to install packages to a Singularity environment. Once removed or commented out, log out and back into the cluster for a fresh environment.

Miniforge Environment PyTorch Example

Conda environments allow users to create customizable, portable work environments and dependencies to support specific packages or versions of software for research. Common conda distributions include Anaconda, Miniconda and Miniforge. Packages are available via "channels". Popular channels include "conda-forge" and "bioconda".  In this tutorial we shall use Miniforge which sets "conda-forge" as the package channel. Traditional conda environments, however, also create a large number of files that can cut into quotas. To help reduce this issue, we suggest using Singularity, a container technology that is popular on HPC systems. Below is an example of how to create a pytorch environment using Singularity and Miniforge.

Create a directory for the environment

Copy an appropriate gzipped overlay images from the overlay directory. You can browse available images to see available options

In this example we use overlay-15GB-500K.ext3.gz as it has enough available storage for most conda environments. It has 15GB free space inside and is able to hold 500K files
You can use another size as needed.

Choose a corresponding Singularity image. For this example we will use the following image

For Singularity image available on nyu HPC greene,  please check the singularity images folder

For the most recent supported versions of PyTorch, please check the PyTorch website. 

Launch the appropriate Singularity container in read/write mode (with the :rw flag)

The above starts a bash shell inside the referenced Singularity Container overlayed with the 15GB 500K you set up earlier. This creates the functional illusion of having a writable filesystem inside the typically read-only Singularity container.

Now, inside the container, download and install miniforge to /ext3/miniforge3

Next, create a wrapper script /ext3/env.sh using a text editor, like nano.

The wrapper script will activate your conda environment, to which you will be installing your packages and dependencies. The script should contain the following:

Activate your conda environment with the following:

If you have the "defaults" channel enabled, please disable it with

Now that your environment is activated, you can update and install packages:

To confirm that your environment is appropriately referencing your Miniforge installation, try out the following:

Install packages

You may now install packages into the environment with either the pip install or conda install commands. 

First, start an interactive job with adequate compute and memory resources to install packages. The login nodes restrict memory to 2GB per user, which may cause some large packages to crash.

After it is running, you’ll be redirected to a compute node. From there, run singularity to setup on conda environment, same as you were doing on login node.

We will install PyTorch as an example:

For the latest versions of PyTorch please check the PyTorch website.

You can see the available space left on your image with the following commands:

Now, exit the Singularity container and then rename the overlay image. Typing 'exit' and hitting enter will exit the Singularity container if you are currently inside it. You can tell if you're in a Singularity container because your prompt will be different, such as showing the prompt 'Singularity>'

Test your PyTorch Singularity Image

Note: the end ':ro' addition at the end of the pytorch ext3 image starts the image in read-only mode. To add packages you will need to use ':rw' to launch it in read-write mode.

Using your Singularity Container in a SLURM Batch Job

Below is an example script of how to call a python script, in this case torch-test.py, from a SLURM batch job using your new Singularity image

torch-test.py:

Now we will write the SLURM job script, run-test.SBATCH, that will start our Singularity Image and call the torch-test.py script.

run-test.SBATCH:

You will notice that the singularity exec command features the '--nv flag' - this flag is reguired to pass the CUDA drivers from a GPU to the Singularity container.

Run the run-test.SBATCH script

Check your SLURM output for results, an example is shown below

Optional: Convert ext3 to a compressed, read-only squashfs filesystem

Singularity images can be compressed into read-only squashfs filesystems to conserve space in your environment. Use the following steps to convert your ext3 Singularity image into a smaller squashfs filesystem.

Here is an example of the amount of compression that can be realized by converting:

Notice that it saves over 3GB of storage in this case, though your results may vary.

Use a squashFS Image for Running Jobs

You can use squashFS images similarly to the ext3 images.

Adding Packages to a Full ext3 or squashFS Image 

If the first ext3 overlay image runs out of space or you are using a squashFS conda enviorment, but need to install a new package inside, please copy another writable ext3 overlay image to work together.

Open the first image in read only mode

Note: Click here for information on how to configure your conda environment.

Please also keep in mind that once the overlay image is opened in default read-write mode, the file will be locked. You will not be able to open it from a new process. Once the overlay is opened either in read-write or read-only mode, it cannot be opened in RW mode from other processes either. For production jobs to run, the overlay image should be open in read-only mode. You can run many jobs at the same time as long as they are run in read-only mode. In this ways, it will protect the computation software environment, software packages are not allowed to change when there are jobs running. 

Julia Singularity Image

Singularity can be used to set up a Julia environment.

Create a directory for your julia work, such as /scratch/<NetID>/julia, and then change to your home directory. An example is shown below.

Copy an overlay image, such as the 2GB 100K overlay, which generally has enough storage for Julia packages. Once copied, unzip to the same folder, rename to julia-pkgs.ext3

Copy the following wrapper script in the directory

Now launch writable Singularity overlay to install packages

Now exit from the container to launch a read only version to test (example below)

You can make the above code into a julia script to test batch jobs. Save the following as test-knitro.jl

You can add additional packages with commands like the one below (NOTE: Please do not install new packages when you have Julia jobs running, this may create issues with your Julia installation)

Run a SLURM job to test with the following sbatch command (e.g. julia-test.SBATCH)

Then run the command with the following:

Once the job completes, check the SLURM output (example below)

Using CentOS 8 for Julia (for Module Compatibility)

Building on the previous Julia example, this will demonstrate how to set up a similar environment using the Singularity CentOS 8 image for additional customization. Using the CentOS 8 overlay allows for the loading of modules installed on Greene, such as Knitro 12.3.0

Copy overlay image

The path in this example is /scratch/<NetID>/julia/julia-pkgs.ext3

To use modules installed into /share/apps you can make two directories

Now, in this example, the absoulte paths are as follows

To launch Singularity with overlay images in writable mode to install packages 

Implement a wrapper script /ext3/env.sh

Load julia via the wrapper script and check that it loads properly

Run julia to install packages

Set up a similar test script like the test-knitro.jl script above. Name it test.jl:

Now implement a wrapper script named julia into ~/bin, the overlay image is in readonly mode

Make the wrapper executable

Test your installation with a SLURM job example. The following code has been put into a file called test-julia-centos.SBATCH

Run the above with the following:

Read the output (example below)

Installing New Julia Packages Later

Implement another writable julia-writable with overlay image writable in order to install new Julia packages later

Check the writable image

Install packages to the writable image

If you do not need host packages installed in /share/apps, you can work with Singularity OS image 

download Julia installation package from https://julialang-s3.julialang.org/bin/linux/x64/1.5/julia-1.5.3-linux-x86_64.tar.gz

install Julia to /ext3, setup PATH properly. It will be easy to move to other servers in future.