Python
Important Notes
If you want to install python packages/modules, refer to Python Module Installation Guide.
To create isolated environments for running Python., refer to Python Virtual Environment Guide
Modules are managed differently on Pioneer, refer to Working with Modules on Pioneer
Pioneer runs Redhat 8 version of linux, so environments and personal modules need installed anew.
Python on Pioneer
EasyBuild leads to component modules
We use the EasyBuild (EB) software installation framework for Pioneer modules, allowing us to update Python versions with greater consistency. As a result of using EB, Python is provided in component modules, including Python, SciPy-bundle (numpy, scipy, pandas, ...), matplotlib, h5py, and more. The Python modules feature a 'bare' alternative that only features 'core' Python modules, when maximum flexibility in choosing modules is desired. The more complete Python is required to also load other component modules, featuring ~125 commonly used modules.
Installed versions
All the available versions of Python. This applies for other applications as well.
module avail python/
module spider python/
output (4 Nov 2022):
----------------------------- /usr/local/easybuild_allnodes/modules/all -----------------------------
Biopython/1.79-foss-2021a Python/3.9.6-GCCcore-11.2.0-bare
IPython/7.25.0-GCCcore-10.3.0 Python/3.9.6-GCCcore-11.2.0
IPython/7.26.0-GCCcore-11.2.0 (D) Python/3.10.4-GCCcore-11.3.0-bare
Python/2.7.18-GCCcore-9.3.0 Python/3.10.4-GCCcore-11.3.0 (L,D)
Python/2.7.18-GCCcore-10.2.0 flatbuffers-python/2.0-GCCcore-10.3.0
Python/2.7.18-GCCcore-11.2.0-bare flatbuffers-python/2.0-GCCcore-11.2.0 (D)
Python/3.8.2-GCCcore-9.3.0 protobuf-python/3.14.0-GCCcore-10.2.0
Python/3.8.6-GCCcore-10.2.0 protobuf-python/3.17.3-GCCcore-10.3.0
Python/3.9.5-GCCcore-10.3.0-bare protobuf-python/3.17.3-GCCcore-11.2.0 (D)
Python/3.9.5-GCCcore-10.3.0 spglib-python/1.16.0-fosscuda-2020b
Where:
L: Module is loaded
D: Default Module
We can see that the default version of Python is 3.10.4-GCCcore-11.3.0 (toolchain foss/2022a).
Identify modules in Python
Load the appropriate python modules, for example
module load Python/3.10.4-GCCcore-11.3.0
Check python modules. Use pip3 for python version 3.
pip list
output: (4 Nov 2022)
appdirs 1.4.4
asn1crypto 1.5.1
atomicwrites 1.4.0
attrs 21.4.0
Babel 2.10.1
...
Requesting Resources and Running Jobs
Use the regular request for a compute node.
This is just an example:
srun --pty bash
Creating virtual environment
$ module load Python/3.8.6-GCCcore-10.2.0
$ python -m venv p3venv-pioneer
$ source ~/p3venv-pioneer/bin/activate
(p3venv-pioneer) $ pip install twill
Collecting twill
Downloading twill-3.1-py3-none-any.whl (46 kB)
|████████████████████████████████| 46 kB 827 kB/s
Collecting lxml<5,>=4.9
Downloading lxml-4.9.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (7.1 MB)
|████████████████████████████████| 7.1 MB 4.4 MB/s
Collecting pyparsing<4,>=3.0
Downloading pyparsing-3.1.1-py3-none-any.whl (103 kB)
|████████████████████████████████| 103 kB 28.4 MB/s
Collecting requests<3,>=2.27
Downloading requests-2.31.0-py3-none-any.whl (62 kB)
|████████████████████████████████| 62 kB 350 kB/s
Collecting urllib3<3,>=1.21.1
Downloading urllib3-2.0.4-py3-none-any.whl (123 kB)
|████████████████████████████████| 123 kB 71.4 MB/s
Collecting idna<4,>=2.5
Downloading idna-3.4-py3-none-any.whl (61 kB)
|████████████████████████████████| 61 kB 38 kB/s
Collecting charset-normalizer<4,>=2
Downloading charset_normalizer-3.2.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (199 kB)
|████████████████████████████████| 199 kB 94.7 MB/s
Collecting certifi>=2017.4.17
Downloading certifi-2023.7.22-py3-none-any.whl (158 kB)
|████████████████████████████████| 158 kB 71.9 MB/s
Installing collected packages: lxml, pyparsing, urllib3, idna, charset-normalizer, certifi, requests, twill
Successfully installed certifi-2023.7.22 charset-normalizer-3.2.0 idna-3.4 lxml-4.9.3 pyparsing-3.1.1 requests-2.31.0 twill-3.1 urllib3-2.0.4
WARNING: You are using pip version 20.2.1; however, version 23.2.1 is available.
You should consider upgrading via the '/home/<CaseID>/p3venv-pioneer/bin/python -m pip install --upgrade pip' command.