Python

Python - NumPy/SciPy

Important Notes

• • Refer to Python Module Installation Guide if you want to install python packages/modules

  • Refer to Python Virtual Environment Guide to create isolated environments for running Python.

NumPy

NumPy(http://numpy.scipy.org/) is a package for scientific computing with Python which contains:

• •     a powerful N-dimensional array object

  •     sophisticated (broadcasting) functions

  •     tools for integrating C/C++ and Fortran code

  •     useful linear algebra, Fourier transform, and random number capabilities

  •     capability to seamlessly and efficiently integrate with a wide variety of databases

Scipy

SciPy (pronounced "Sigh Pie": (http://www.scipy.org/)) is open-source software for mathematics, science, and engineering. It is also the name of a very popular conference on scientific programming with Python. The SciPy library depends on NumPy, which provides convenient and fast N-dimensional array manipulation. The SciPy library is built to work with NumPy arrays, and provides many user-friendly and efficient numerical routines such as routines for numerical integration and optimization.

Installed versions

All the available versions of Python. This applies for other applications as well.

module avail python

module spider python

output:

--------------------------------------- /usr/local/share/modulefiles ------------------------------------

python/2.7.8 (D) python/2.7.10 

We can see that the default version of Python is 2.7.8.

The python modules built with gcc can be accessed by switching to gcc environment and module avail

module swap intel gcc

module avail python

Identify modules in Python

Load the appropriate python modules (version 2 or 3). 

module load python/version

Check python modules. Use pip3 for python version 3.

pip freeze

output:

alabaster==0.7.10

apptools==4.4.0

asn1crypto==0.24.0

Run Python interactively. Use python3 for python version 3.

python

Running NumPy/Scipy Job

Interactive Job

In the bash terminal, request a compute Node:

srun --pty /bin/bash

Load the Python module:

module load python/<version>

Run Python version 2 using python and version 3 using python3:

python

In the Python Interactive Window, try the following:

>>> import numpy;

>>> numpy.version.version

'1.12.0.dev0+59a7b25'

>>> import scipy;

>>> scipy.version.version

'0.18.0.dev0+54c0d5a'

>>> quit()

Run from the Terminal

Copy the job file "job.slurm" and python file "test.py" from /usr/local/doc/NUMPY-SCIPY to your home directory

cp /usr/local/doc/NUMPY-SCIPY/* .

Run

python test.py

output:

NumPy Version:

1.12.0.dev0+59a7b25

SciPy Version:

0.18.0.dev0+54c0d5a

c:

[-3.14159265 0. 3.14159265]

Batch Job

Copy the job file "job.slurm" and python file "test.py" from /usr/local/doc/NUMPY-SCIPY to your home directory

cp /usr/local/doc/NUMPY-SCIPY/* .

Submit the job:

sbatch job.slurm

Find the output as showed in the output file " slurm.o<JobID>

NumPy Version:

1.12.0.dev0+59a7b25

SciPy Version:

0.18.0.dev0+54c0d5a

c:

[-3.14159265 0. 3.14159265]

Troubleshooting

Issue: encountered "Illegal instruction (core dumped)"

Solution: 

(i) Make sure that you have requested enough memory with slurm --mem flag.

(ii) Python init files - __init__.py and __init__.pyc, conflicted with python script. so they need to be deleted.

References:

1. Python Home

2. Tutorial