Module System In-Depth

We rely primarily on EasyBuild to install software for the HPC computing clusters, and implement module naming such that; 1)  all module files are directly available for loading; and, 2)  each module name uniquely identifies a particular installation.

Since all the modules are available to load at all times, it is important to ensure that the environment remains consistent, and the module command remains the best means to do so.

Toolchains

Toolchains are used to provide software built from the same 'base' package versions, and therefore benefitting from the associated consistency and community testing. New toolchains are released approximately twice each year. Case HPC will maintain the 'foss' and 'intel' toolchains.

• The foss toolchain consists of all open source components (hence the name: "FOSS" stands for Free & Open Source Software): GCC, Open MPI, OpenBLAS, ScaLAPACK and FFTW.

• The intel toolchain consists of the Intel C, C++ and Fortran compilers (on top of a GCC version controlled through EasyBuild) alongside the Intel MPI and Intel MKL libraries.

Lmod and the 'flat' hierarchy

We install our software as modules and we manage them using Lmod.

"Lmod is a Lua based module system that easily handles the MODULEPATH Hierarchical problem. Environment Modules provide a convenient way to dynamically change the users' environment through modulefiles. This includes easily adding or removing directories to the PATH environment variable. Modulefiles for Library packages provide environment variables that specify where the library and header files can be found." Source: TACC Lmod

Module Paths

Modules installed with EasyBuild will have a common module path: /usr/local/easybuild/modules/all

The MODULEPATH variable is set by default to reference the EasyBuild path. Custom software, specifcially that installed from source in a personal directory (e.g. /home/<caseid>/..) may have Lmod modules established. The MODULEPATH will require modification to make those modules available to use for managing the shell environment for this custom software. 

Optimization Trees

The CWRU HPC has nodes with a variety of capabilities, including the cpu instructions sets. To manage code installed to run on either a small set of specialized nodes, or to run generally on a broad set of nodes, we are using using module trees. The current trees and their capabilities are listed:

• allnodes: (default)  should run without error on all the HPC nodes, though without the latest cpu architecture support.

• icosa:      optimized to use all cpu architecture support on the icosa192gb feature nodes.

• aisc:        optimized for use with the aisc partition nodes (limited access).

To switch trees, there are two approaches. Both adjust the MODULEPATH environment variable.

• Approach 1

  • export MODULEPATH=/usr/local/easybuild_<tree>/modules/all

• Approach 2

  • module unuse /usr/local/easybuild_allnodes/modules/all

  • module use /usr/local/easybuild_<tree>/modules all

The <tree> is a placeholder, with the target to be taken from the list of trees above (e.g. easybuild_allnodes)

Supplementary Software

The latest versions of licensed software and other system software supplementing that available from EasyBuild will be installed in another location, from the file system root location "/usr/local/software". For example, Matlab will be found under "/usr/local/software/matlab/<release-version>". This path is also included by default in MODULEPATH.

Managing Modules

Module Load and Unload

Simply stated, a module is a packaging of definitions of environment variables into a script. In general, there is a module defined for each application, and that module defines the environment appropriately for that application. To use the application, you need to load the module first using the module command:

module load <software-module>

For an example, if the software-module is "matlab", it becomes "module load matlab".

If there are more versions, let's say matlab/R2023a, then it becomes:

module load matlab/R2023a

To load the sotware-module, use the command:

module load <software-module>

To unload it, use:

module unload <software-module>

Module List

To see which modules you have loaded, use:

module list

All Modules Can Be Loaded - Manage environment with care

Contrary to previous clusters, Modules on Pioneer are organized without hierarchy, and the module names contain information about toolchains and compilers used in the build. For example, 

module avail Python/   (note the trailing '/')

[mrd20@hpc8 ~]$ module avail Python/   #  partial output

------------------------------------- /usr/local/easybuild_allnodes/modules/all -----------------------

   GitPython/3.1.40-GCCcore-12.3.0      Python/3.10.4-GCCcore-11.3.0

   LIBSVM-Python/3.30-foss-2022a        Python/3.10.8-GCCcore-12.2.0-bare

   Python/3.7.4-GCCcore-8.3.0           Python/3.10.8-GCCcore-12.2.0

   Python/3.8.2-GCCcore-9.3.0           Python/3.11.3-GCCcore-12.3.0

   Python/3.8.6-GCCcore-10.2.0          Python/3.11.5-GCCcore-13.2.0

   Python/3.9.5-GCCcore-10.3.0-bare     Python/3.12.3-GCCcore-13.3.0          (D)

All modules with 'python/' or 'Python/' are listed. This example shows the distinction between the GCCcore prepared modules, and the 'gompic' modules. Without the trailing '/', modules having "python" appearing anywhere in the name will be listed. 

Possible Compatibility Issues

For compatibility reasons, try to load the dependency modules built with appropriate toolchains and versions. For example, the dependency modules are using the same GCCcore-12.2.0.

• module purge                                                # purge all the modules before loading any new modules

• module load Python/3.10.8-GCCcore-12.2.0

• module load Tk/8.6.12-GCCcore-12.2.0

• module load Tcl/8.6.12-GCCcore-12.2.0

Let's consider a scenario where you want to use PyYAML module. Let's say you decide to use the version "PyYAML/6.0-GCCcore-12.3.0". Then you need to check the possible dependency modules (using "module list" after loading the module). Now, you will be using "Python/3.11.3-GCCcore-12.3.0" version of Python. The other version of Python can have compatibility issues. Now, if you want to use SciPy module, you will be choosing "module show SciPy-bundle/2023.07-gfbf-2023a" that uses the same Python version.

Searching All Modules

The distinction between using 'avail' and 'spider' is now reduced, as all modules will be listed, according to what directories are included in the MODULEPATH variable (echo $MODULEPATH).  The two commands provide different degrees of information, with 'module avail' providing a listing of module names, and "module spider <name>" the set of modules, a description of the software, and further steps to gain more information. Following from the previous example, use "module spider gompi/' and "module spider gompic/" to better understand 'gompic':

[mrd20@hpc8 ~]$ module spider gompi/

Description:  GNU Compiler Collection (GCC) based compiler toolchain, including OpenMPI for MPI support.

  Versions:      gompi/2020a, gompi/2020b, gompi/2021a, gompi/2021b, gompi/2022a

                              gompi/2022b, gompi/2023a, gompi/2023b, gompi/2024a

As previously, for detailed information about a specific package (including how to load the modules) use the module's full name.   For example:

  $ module spider gompi/2021a

Note that with the 'flat hierarchy' and extended module names, tab-completion of module names is available in the shell.