Python

Important Notes

• If you want to install python packages/modules, refer to Python Module Installation Guide.

• To create isolated environments for running Python., refer to Python Virtual Environment Guide 

• Modules are managed differently on Pioneer, refer to Working with Modules on Pioneer

• Pioneer runs Redhat 8 version of linux, so environments and personal modules need installed anew.

Python on Pioneer

EasyBuild leads to component modules

We use the EasyBuild (EB) software installation framework for Pioneer modules, allowing us to update Python versions with greater consistency. As a result of using EB, Python is provided in component modules, including Python, SciPy-bundle (numpy, scipy, pandas, ...), matplotlib, h5py, and more. The Python modules feature a 'bare' alternative that only features 'core' Python modules, when maximum flexibility in choosing modules is desired. The more complete Python is required to also load other component modules, featuring ~125 commonly used modules.

Installed versions

All the available versions of Python. This applies for other applications as well.

module avail python/ 

module spider python/ 

output (4 Nov 2022):

----------------------------- /usr/local/easybuild_allnodes/modules/all -----------------------------

   Biopython/1.79-foss-2021a                Python/3.9.6-GCCcore-11.2.0-bare

   IPython/7.25.0-GCCcore-10.3.0            Python/3.9.6-GCCcore-11.2.0

   IPython/7.26.0-GCCcore-11.2.0     (D)    Python/3.10.4-GCCcore-11.3.0-bare

   Python/2.7.18-GCCcore-9.3.0              Python/3.10.4-GCCcore-11.3.0          (L,D)

   Python/2.7.18-GCCcore-10.2.0             flatbuffers-python/2.0-GCCcore-10.3.0

   Python/2.7.18-GCCcore-11.2.0-bare        flatbuffers-python/2.0-GCCcore-11.2.0 (D)

   Python/3.8.2-GCCcore-9.3.0               protobuf-python/3.14.0-GCCcore-10.2.0

   Python/3.8.6-GCCcore-10.2.0              protobuf-python/3.17.3-GCCcore-10.3.0

   Python/3.9.5-GCCcore-10.3.0-bare         protobuf-python/3.17.3-GCCcore-11.2.0 (D)

   Python/3.9.5-GCCcore-10.3.0              spglib-python/1.16.0-fosscuda-2020b

  Where:

   L:  Module is loaded

   D:  Default Module

We can see that the default version of Python is 3.10.4-GCCcore-11.3.0 (toolchain foss/2022a).

Identify modules in Python

Load the appropriate python modules, for example

module load Python/3.10.4-GCCcore-11.3.0

Check python modules. Use pip3 for python version 3.

pip list

output: (4 Nov 2022)

appdirs                           1.4.4

asn1crypto                        1.5.1

atomicwrites                      1.4.0

attrs                             21.4.0

Babel                             2.10.1

...

Requesting Resources and Running Jobs

Use the regular request for a compute node.

This is just an example:

srun --pty bash

Creating virtual environment

$ module load Python/3.8.6-GCCcore-10.2.0

$ python -m venv p3venv-pioneer

$ source ~/p3venv-pioneer/bin/activate

(p3venv-pioneer) $ pip install twill

Collecting twill

  Downloading twill-3.1-py3-none-any.whl (46 kB)

     |████████████████████████████████| 46 kB 827 kB/s

Collecting lxml<5,>=4.9

  Downloading lxml-4.9.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (7.1 MB)

     |████████████████████████████████| 7.1 MB 4.4 MB/s

Collecting pyparsing<4,>=3.0

  Downloading pyparsing-3.1.1-py3-none-any.whl (103 kB)

     |████████████████████████████████| 103 kB 28.4 MB/s

Collecting requests<3,>=2.27

  Downloading requests-2.31.0-py3-none-any.whl (62 kB)

     |████████████████████████████████| 62 kB 350 kB/s

Collecting urllib3<3,>=1.21.1

  Downloading urllib3-2.0.4-py3-none-any.whl (123 kB)

     |████████████████████████████████| 123 kB 71.4 MB/s

Collecting idna<4,>=2.5

  Downloading idna-3.4-py3-none-any.whl (61 kB)

     |████████████████████████████████| 61 kB 38 kB/s

Collecting charset-normalizer<4,>=2

  Downloading charset_normalizer-3.2.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (199 kB)

     |████████████████████████████████| 199 kB 94.7 MB/s

Collecting certifi>=2017.4.17

  Downloading certifi-2023.7.22-py3-none-any.whl (158 kB)

     |████████████████████████████████| 158 kB 71.9 MB/s

Installing collected packages: lxml, pyparsing, urllib3, idna, charset-normalizer, certifi, requests, twill

Successfully installed certifi-2023.7.22 charset-normalizer-3.2.0 idna-3.4 lxml-4.9.3 pyparsing-3.1.1 requests-2.31.0 twill-3.1 urllib3-2.0.4

WARNING: You are using pip version 20.2.1; however, version 23.2.1 is available.

You should consider upgrading via the '/home/<CaseID>/p3venv-pioneer/bin/python -m pip install --upgrade pip' command.