Migrating from Rider

General Concepts

The software library available through modules on Pioneer is generally richer and more up to date than on Rider, but the way it is organized is significantly different. Keep in mind the following key points while migrating a job or workflow from Rider to Pioneer.

Module Hierarchy

The software modules on Rider were organized hierarchically, so you needed to load certain modules before you could load other modules. On Pioneer, the modules are all flat, so you just load the application you want directly. For example on Rider,  you first need to load gcc 6.3.0 before you can load Python 2.8.6:

module load gcc/6.3.0 python/3.8.6

On Pioneer, you just load Python directly, and it handles loading any dependent modules:

module load Python/3.8.6

Matching Dependencies

A second key point to note is that since the modules load all of their dependencies, you must plan to load compatible modules if you need a software stack beyond a single module. Matching the dependencies is fairly simple in most cases, where we only need to match the toolchain that was used to build the software. The toolchain is included in the full version string for the modules, e.g. highlighted in bold below:

module spider R

...

     Versions:

        R/4.0.3-foss-2020b

        R/4.1.0-foss-2021a

        R/4.1.2-foss-2021b

        R/4.1.3-foss-2021a

        R/4.2.0-foss-2021b

        R/4.2.1-foss-2021a

        R/4.2.1-foss-2021b

        R/4.2.1-foss-2022a

        R/4.2.2-foss-2022a

        R/4.2.2-foss-2022b

If we want to load Tensorflow and R, we need to load versions from compatible toolchains, so we also look at Tensorflow:

module spider TensorFlow

...

     Versions:

        TensorFlow/2.6.0-foss-2021a-CUDA-11.3.1

        TensorFlow/2.7.1-foss-2021b-CUDA-11.4.1

        TensorFlow/2.11.0-foss-2022a-CUDA-11.7.0

We see that Tensorflow 2.11 and R 4.2.2 were both compiled with the foss-2022a toolchain, and thus we can load them both without generating conflicts:

module load R/4.2.2-foss-2022a TensorFlow/2.11.0-foss-2022a

As a general rule of thumb, if you see messages like "Reloading the following modules..." output to the terminal, you have a conflict in your module versions.

Python

Multiple versions of Python are available:

module spider Python

     Versions:

        Python/2.7.18-GCCcore-9.3.0

        Python/2.7.18-GCCcore-10.2.0

        Python/2.7.18-GCCcore-10.3.0-bare

        Python/2.7.18-GCCcore-11.2.0-bare

        Python/2.7.18-GCCcore-11.3.0-bare

        Python/3.7.4-GCCcore-8.3.0

        Python/3.8.2-GCCcore-9.3.0

        Python/3.8.6-GCCcore-10.2.0

        Python/3.9.5-GCCcore-10.3.0-bare

        Python/3.9.5-GCCcore-10.3.0

        Python/3.9.6-GCCcore-11.2.0-bare

        Python/3.9.6-GCCcore-11.2.0

        Python/3.10.4-GCCcore-11.3.0-bare

        Python/3.10.4-GCCcore-11.3.0

        Python/3.10.8-GCCcore-12.2.0-bare

        Python/3.10.8-GCCcore-12.2.0

These can be loaded directly, e.g. module load Python/3.10.4

In addition, several widely used Python packages have dedicated modules, such as Tensorflow and SciPy. Loading the package specific modules loads the compatible version of Python, e.g.:

[stm@hpc8 ~]$ module load SciPy-bundle/2022.05

[stm@hpc8 ~]$ python

Python 3.10.4 (main, Sep 12 2022, 18:12:09) [GCC 11.3.0] on linux

Type "help", "copyright", "credits" or "license" for more information.

>>> 

We encourage the use of virtual environments when there is a need to build a highly customized software stack, such as specific versions of libraries, or libraries beyond what are available in the default install. 

R

There are multiple version of R:

module spider R

     Versions:

        R/4.0.3-foss-2020b

        R/4.1.0-foss-2021a

        R/4.1.2-foss-2021b

        R/4.1.3-foss-2021a

        R/4.2.0-foss-2021b

        R/4.2.1-foss-2021a

        R/4.2.1-foss-2021b

        R/4.2.1-foss-2022a

        R/4.2.2-foss-2022a

        R/4.2.2-foss-2022b

These can be loaded directly, e.g. module load R/4.2.2. 

There are also a modules available specifically for Bioconductor that load the compatible supporting version of R, e.g.:

[stm@hpc8 ~]$ module load R-bundle-Bioconductor/3.15

[stm@hpc8 ~]$ R

R version 4.2.2 (2022-10-31) -- "Innocent and Trusting"

Copyright (C) 2022 The R Foundation for Statistical Computing

Platform: x86_64-pc-linux-gnu (64-bit)

...

Additional packages can be installed in the user home directory the same as on Rider.

MATLAB

There are multiple versions of MATLAB, which can be loaded directly, the same as on Rider

module spider matlab

     Versions:

        matlab/R2020b

        matlab/R2021b

        matlab/R2023a