Markov Software

Available Software

This section provides high level information regarding software applications available on the HPC cluster. To get more information, go to HPC Software Guide.  For information specific to the Pioneer cluster, we offer Pioneer Software Guide.

Searching and Loading the Software

• • To get the versions of the software installed in HPC, use the command "module spider <software>".

  • The path of the executable and libraries of the <software> can be obtained using the command "module display <software>"

Library Dependency

• • The default libraries are available at /usr/lib, /usr/lib64, /usr/local/lib, /usr/local/lib64

  • The module system provides a simple means to include or exclude software-specific libraries in the runtime environment.

  • In the case of software that is not available through the module system, you can install it in your home directory. Check our HPC Tutorials. If it is a generic software and you are having trouble installing it after following the instructions from the HPC Tutorials, please contact us.

Licensed Software

• • If the Software available on HPC is licensed, you will need to explicitly request access via email to  hpc-supportATcase.edu

  • To maintain the licensed software for you, you would need to contact the vendor and find out about the software and its feasibility on the HPC (license model and control). We would ask that your PI acquires HPC membership (see HPC access policy).

Software Categories

To have the complete alphabetical listing of the available software, please go to HPC Software Guide.

• Programming/Computing Languages

• Compilers

• Cryo-EM

• Deep Learning

• Visualization

• Molecular Dynamics

• Genome Analysis

• Container Platforms

• IDEs

Accelerated Computing

There are software and hardware methods to increase the efficiency of software. The starting point is that the software must have methods to allow acceleration through parallel execution using multiple cpu-cores, or to compute using a GPU. Without the software built to make use of the hardware, the acceleration can not benefit from hardware resources alone. Researchers need to develop an understanding of such capabilities available in the software that they rely on.

Comment on Parallel Computing with Python

The situation with Python and parallelism is complicated. What you read is correct, and the central problem is the CPython interpreter uses a Global Interpreter Lock (GIL) to enforce only one thread is executing Python code at a time. The ways to get around that are to 1) use processes, 2) use libraries that can release the GIL under appropriate conditions.

The most popular library for parallel processing in Python is multiprocessing: https://docs.python.org/3/library/multiprocessing.html   It provides a high level interface that look like threads, but it is actually spawning multiple processes to avoid the GIL limitations.  Note that the multiprocessing library does not detect the correct number of CPUs when Python is running as an HPC job. You will need to design your code in a way that you can pass in the number of CPUs the job has been allocated.

Some numpy routines can release the GIL when they will be executing native instructions compiled from C: https://stackoverflow.com/questions/36479159/why-are-numpy-calculations-not-affected-by-the-global-interpreter-lock.  Any Python library that links to native code (has a compiled component) could use this same approach, so there are probably other examples.

Adding Jupyter to the requirements may complicate things as there are several Stackoverflow questions around multiprocessing not working in Jupyter, e.g. https://stackoverflow.com/questions/50937362/multiprocessing-on-python-3-jupyter

But there are also many suggestions and examples for how to make it work.