Running Spark in an Interactive Jupyter Notebook

The Spark Standalone Cluster application allows you to bring up a Spark instance on Greene and interact with it live.

Connect to the Open OnDemand Web Dashboard.  In the dropdown menu in the header, select Interactive Apps > Spark Standalone Cluster.  Set the cluster's parameters using the provided form, and then click Launch.

You will be automatically redirected to an info card for the session, or you can find it again under My Interactive Sessions. Once the cluster has finished startup, you should see the following:

The Host link will take you to a shell interface with the remote machine running your cluster.

The Standalone Cluster Web UI link provides monitoring and usage information, as well as the cluster's SPARK_URL for directing batch jobs as outlined in Running Multi-Node Spark with Singularity.

The Jupyter Notebook Environment link takes you to your home or scratch file directory on Greene, where you will be able to create or open Jupyter notebooks that run against your standalone cluster.

For code examples, see Big Data Tutorial: Spark.

Adding custom python libraries to Spark Standalone Cluster

Build an overlay with your python libraries

Adding python libraries to the Spark Standalone Cluster requires a custom overlay which is compatible with the existing Spark Standalone images.  

If you have built a Miniforge environment before, e.g. by following the PyTorch Example tutorial, this process should be familiar.  However, instead of building installations from scratch, we will start by cloning an existing pyspark image, and install additional packages on top of it.  Finally, before saving we will clean up files and packages in the overlay that might overwrite data in the Spark Standalone images.

Create a working directory.

Copy an appropriately sized overlay image. In this example we use overlay-15GB-500K.ext3.gz.  It has 15GB free space inside and is able to hold 500K files

Launch a Singularity container to open the overlay in read/write mode.

Next, inside the container, download and install Miniforge to /ext3/miniforge3.

Create a wrapper script to activate your base conda environment.  Open /ext3/env.sh and write the following:

Activate the conda environment, update, and clean. Finally, exit singularity to continue the installation on a compute node.

Clone a pyspark image and install additional packages

First, start an interactive job with adequate compute and memory resources to install packages. This will allow you to install large packages which might otherwise crash on the login node with its 2GB memory limit.

Once a node is assigned, open your overlay in read/write mode and activate the base environment.

Now, clone an existing pyspark environment to use as base. This command uses the file flag -f to create a new conda environment from an existing config file. The prefix flag -p creates the environment in its own top-level directory, /ext3/pyspark.  Make sure you use the same pyspark version here that you intend to use when starting up the Spark Standalone Cluster.

Activate the new environment and install the libraries you need using conda or pip.

To double check your installations, you can open an interactive python window within the container.  Confirm that your custom libraries are installed with the correct version.

Once you are happy with your python environment, clean up files in /ext3 that would conflict with the default Spark Standalone images. This is necessary to make your overlay compatible with the Spark Standalone Cluster.

As a final optional step, compress your overlay into a squashfs filesystem to save space.

Add the overlay when initializing Spark Standalone Cluster

When launching a Spark Standalone Cluster from the Open OnDemand dashboard, include the path to your custom overlay at the bottom of the form.  You should then be able to access your custom libraries from any Jupyter notebooks run against the cluster.  The same overlay can be re-used in as many cluster instances as needed.