Working with Modules on Pioneer

Software Installation on Pioneer

The Pioneer cluster will mount the /home directories in common with the Markov cluster, however, because Pioneer servers run the RedHat 8 operating system (RHEL 8) the installed software will be on a different location in the High Performance Computing Storage (Panasas file system). The 'mount point' for this storage will have the same name (e.g. /usr/local") as with and Markov, but it is different software, compiled under RedHat 8.


We rely primarily on EasyBuild to install software for the Pioneer computing cluster, and implement module naming such that; 1)  all module files are directly available for loading; and, 2)  each module name uniquely identifies a particular installation. This is different from the hierarchical approach to module management used on Markov cluster. 


Since all the modules are now available to load at all times, it is important to ensure that the environment remains consistent, and the module command remains the best means to do so.


Toolchains

Toolchains are used to provide software built from the same 'base' package versions, and therefore benefitting from the associated consistency and community testing. New toolchains are released approximately twice each year. Case HPC will maintain the 'foss' and 'intel' toolchains.



Lmod and the 'flat' hierarchy

We install our software as modules and we manage them using Lmod.

"Lmod is a Lua based module system that easily handles the MODULEPATH Hierarchical problem. Environment Modules provide a convenient way to dynamically change the users' environment through modulefiles. This includes easily adding or removing directories to the PATH environment variable. Modulefiles for Library packages provide environment variables that specify where the library and header files can be found." Source: TACC Lmod


Module Paths

Modules installed with EasyBuild will have a common module path: /usr/local/easybuild/modules/all

The MODULEPATH variable is set by default to reference the EasyBuild path. Custom software, specifcially that installed from source in a personal directory (e.g. /home/<caseid>/..) may have Lmod modules established. The MODULEPATH will require modification to make those modules available to use for managing the shell environment for this custom software. 


Optimization Trees

The CWRU HPC has nodes with a variety of capabilities, including the cpu instructions sets. To manage code installed to run on either a small set of specialized nodes, or to run generally on a broad set of nodes, we are using using module trees. The current trees and their capabilities are listed:


To switch trees, there are two approaches. Both adjust the MODULEPATH environment variable.

The <tree> is a placeholder, with the target to be taken from the list of trees above (e.g. easybuild_allnodes)

Supplementary Software

The latest versions of licensed software and other system software supplementing that available from EasyBuild will be installed in another location, from the file system root location "/usr/local/software". For example, Matlab will be found under "/usr/local/software/matlab/<release-version>", as is the case on the Markov cluster filesystem. This path is also included by default in MODULEPATH.

Managing Modules

Module Load and Unload

Simply stated, a module is a packaging of definitions of environment variables into a script. In general, there is a module defined for each application, and that module defines the environment appropriately for that application. To use the application, you need to load the module first using the module command:

module load <software-module>

For an example, if the software-module is "matlab", it becomes "module load matlab".

If there are more versions, let's say matlab/R2019b, then it becomes:

module load matlab/R2019b

To load the sotware-module, use the command:

module load <software-module>

To unload it, use:

module unload <software-module>

Module List

To see which modules you have loaded, use:

module list

All Modules Can Be Loaded - Manage environment with care

Contrary to previous clusters, Modules on Pioneer are organized without hierarchy, and the module names contain information about toolchains and compilers used in the build. For example, 

module avail Python/   (note the trailing '/')

[mrd20@hpc8 ~]$ module avail Python/

------------------------------------- /usr/local/easybuild/modules/all --------------------------------------------

   Boost.Python/1.71.0-gompic-2019b Python/3.7.4-GCCcore-8.3.0    

   Python/2.7.16-GCCcore-8.3.0                    Python/3.8.2-GCCcore-9.3.0    

   Python/2.7.18-GCCcore-9.3.0                 Python/3.8.6-GCCcore-10.2.0

   Python/2.7.18-GCCcore-10.2.0                Python/3.9.5-GCCcore-10.3.0 (D)

   flatbuffers-python/1.12-GCCcore-10.2.0     protobuf-python/3.14.0-GCCcore-10.2.0


All modules with 'python/' or 'Python/' are listed. This example shows the distinction between the GCCcore prepared modules, and the 'gompic' modules. Without the trailing '/', modules having "python" appearing anywhere in the name will be listed. 


Possible Compatibility Issues

For compatibility reasons, try to load the dependency modules built with appropriate toolchains and versions. For example, the dependency modules are using the same GCCcore-12.2.0.


Let's consider a scenario where you want to use PyYAML module. Let's say you decide to use the version "PyYAML/6.0-GCCcore-12.3.0". Then you need to check the possible dependency modules (using "module list" after loading the module). Now, you will be using "Python/3.11.3-GCCcore-12.3.0" version of Python. The other version of Python can have compatibility issues. Now, if you want to use SciPy module, you will be choosing "module show SciPy-bundle/2023.07-gfbf-2023a" that uses the same Python version.

Searching All Modules

The distinction between using 'avail' and 'spider' is now reduced, as all modules will be listed, according to what directories are included in the MODULEPATH variable (echo $MODULEPATH).  The two commands provide different degrees of information, with 'module avail' providing a listing of module names, and "module spider <name>" the set of modules, a description of the software, and further steps to gain more information. Following from the previous example, use "module spider gompi/' and "module spider gompic/" to better understand 'gompic':


[mrd20@hpc8 ~]$ module spider gompi/


Description:  GNU Compiler Collection (GCC) based compiler toolchain, including OpenMPI for MPI support.

  Versions:    gompi/2019b    gompi/2020a   gompi/2020b   gompi/2021a


[mrd20@hpc8 ~]$ module spider gompic/


Description:  GNU Compiler Collection (GCC) based compiler toolchain along with CUDA toolkit, including OpenMPI for MPI support with CUDA features enabled.

  Versions:    gompic/2019b    gompic/2020a   gompic/2020b


As previously, for detailed information about a specific package (including how to load the modules) use the module's full name.   For example:


  $ module spider gompi/2021a


Note that with the 'flat hierarchy' and extended module names, tab-completion of module names is available in the shell.

Differences in Modules from Markov

GPUS and CUDA

To use CUDA on Pioneer, there are three module types to be aware of:  CUDA, CUDAcore, and cuDNN.  CUDA loads the compiler and the appropriate CUDAcore.  The cuDNN must be managed separately to implement the cudnn libraries of interest for the job, compatible with the CUDAcore version selected.