Python

Important Notes

If you want to install python packages/modules, refer to Python Module Installation Guide.
To create isolated environments for running Python., refer to Python Virtual Environment Guide
The system software library is managed through modules, including the modules to activate different Python versions. To learn more you can refer to Module System Basicis and Module System In-Depth
The current clusters runs Redhat 8 version of Linux, so environments and personal modules that were installed on older versions of the clusters may need reinstalled.

Python on Pioneer

EasyBuild leads to component modules

We use the EasyBuild (EB) software installation framework for Pioneer modules, allowing us to update Python versions with greater consistency. As a result of using EB, Python is provided in component modules, including Python, SciPy-bundle (numpy, scipy, pandas, ...), matplotlib, h5py, and more. The Python modules feature a 'bare' alternative that only features 'core' Python modules, when maximum flexibility in choosing modules is desired. The more complete Python is required to also load other component modules, featuring ~125 commonly used modules.

Installed versions

All the available versions of Python. This applies for other applications as well.

module avail python/

module spider python/

output (4 Nov 2022):

----------------------------- /usr/local/easybuild_allnodes/modules/all -----------------------------

Biopython/1.79-foss-2021a Python/3.9.6-GCCcore-11.2.0-bare

IPython/7.25.0-GCCcore-10.3.0 Python/3.9.6-GCCcore-11.2.0

IPython/7.26.0-GCCcore-11.2.0 (D) Python/3.10.4-GCCcore-11.3.0-bare

Python/2.7.18-GCCcore-9.3.0 Python/3.10.4-GCCcore-11.3.0 (L,D)

Python/2.7.18-GCCcore-10.2.0 flatbuffers-python/2.0-GCCcore-10.3.0

Python/2.7.18-GCCcore-11.2.0-bare flatbuffers-python/2.0-GCCcore-11.2.0 (D)

Python/3.8.2-GCCcore-9.3.0 protobuf-python/3.14.0-GCCcore-10.2.0

Python/3.8.6-GCCcore-10.2.0 protobuf-python/3.17.3-GCCcore-10.3.0

Python/3.9.5-GCCcore-10.3.0-bare protobuf-python/3.17.3-GCCcore-11.2.0 (D)

Python/3.9.5-GCCcore-10.3.0 spglib-python/1.16.0-fosscuda-2020b

Where:

L: Module is loaded

D: Default Module

We can see that the default version of Python is 3.10.4-GCCcore-11.3.0 (toolchain foss/2022a).

Identify modules in Python

Load the appropriate python modules, for example

module load Python/3.10.4-GCCcore-11.3.0

Check python modules. Use pip3 for python version 3.

pip3 list

output: (4 Nov 2022)

appdirs 1.4.4

asn1crypto 1.5.1

atomicwrites 1.4.0

attrs 21.4.0

Babel 2.10.1

...

Requesting Resources and Running Jobs

Running NumPy/Scipy Job

Interactive Job

In the bash terminal, request a compute Node (For Markov, use the appropriate Slurm flags -A and -p):

srun --pty /bin/bash

Load the Installed version of Python module

Run Python version 3 using python3:

python3

In the Python Interactive Window, try the following. Install the numpy and scipy packages; recommended to use Python Virtual Environment.

>>> import numpy;

>>> numpy.version.version

'1.12.0.dev0+59a7b25'

>>> import scipy;

>>> scipy.version.version

'0.18.0.dev0+54c0d5a'

>>> quit()

Run from the Terminal

Copy the job file "job.slurm" and python file "test.py" from /usr/local/doc/NUMPY-SCIPY to your home/course directory

cp /usr/local/doc/NUMPY-SCIPY/* .

Run

python test.py

output:

NumPy Version:

1.12.0.dev0+59a7b25

SciPy Version:

0.18.0.dev0+54c0d5a

[-3.14159265 0. 3.14159265]

Batch Job

Copy the job file "job.slurm" and python file "test.py" from /usr/local/doc/NUMPY-SCIPY to your home directory.

cp /usr/local/doc/NUMPY-SCIPY/* .

Submit the job. Please make sure to use the correct slurm flags in the job.slurm especially for the Markov (class)

sbatch job.slurm

Find the output as showed in the output file " slurm.o<JobID>

NumPy Version:

1.12.0.dev0+59a7b25

SciPy Version:

0.18.0.dev0+54c0d5a

[-3.14159265 0. 3.14159265]

Page updated

Report abuse