Embedded Files
In order to be able to install new Python packages and make your work reproducible, please use virtual environments.
There is more than one way to create a private environment in Python.
Create project directory and load Python module
## Find python version you need Create project directory and load Python module
module avail python
## created directory for your project and cd there
mkdir /scratch/$USER/my_project
cd /scratch/$USER/my_project
## load python module (different versions available)
module load python/intel/3.8.6
Automatic deletion of your files
Automatic deletion of your files
This page describes the installation of packages on /scratch. One has to remember, though, that files stored in the HPC scratch file system are subject to the HPC Scratch old file purging policy: Files on the /scratch file system that have not been accessed for 60 or more days will be purged (read more).
Thus you can consider the following options
Reinstall your packages if some of the files get deleted
You can do this manually
You can do this automatically. For example, within a workflow of a pipeline software like Nextflow
Pay for "Research Project Space" - read more here
Use Singularity and install packages within a corresponding overlay file - read more here
Create virtual environment
Create virtual environment
It is advisable to create private environment inside the project directory. This boosts reproducibility and does not use space in /home/$USER
virtualenv
virtualenv
virtualenv is a tool to create isolated Python environments
Since Python 3.3, a subset of it has been integrated into the standard library under the venv module.
Note: you may need to install virtualenv first, if it is not yet installed (instructions)
Now create new virtual environment in current directory
Empty
OR
inherit all packages from those installed on HPC already (and available in PATH after you load python module)
## Create an EMPTY virtual environmentvirtualenv venv
## Create an virtual environment that inherits system packagesvirtualenv venv --system-site-packages
venv
venv
venv is package shipped with Python3. It provides subset of options available in virtualenv tool (link).
python3 -m venv venvCreate new virtual environment in current directory
Empty
OR
inherit all packages from those installed on HPC already (and available in PATH after you load python module)
mkdir /scratch/$USER/my_project
cd /scratch/$USER/my_project
##EMPTY
## (use venv command to create environment called "venv")
python3 -m venv venv
## Inhering all packages
python3 -m venv venv --system-site-packages
Install packages. Keep things reproducible
## activatesource venv/bin/activate## install packagespip install <package you need>## If package was inherited, but you want to install it in your own env anywaypip install <package you need> --ignore-installed## export list of packages (to report together with paper and/or to reproduce environment on another computer)pip freeze > requirements.txt## restorepip install -r requirements.txtInstall packages. Keep things reproducible
Close an Activated Virtual Environment
Close an Activated Virtual Environment
If you have activated a virtual environment, you can exit it with the following command:
deactivateUse with sbatch
Use with sbatch
When you use this env in sbatch script, please use
module purge;source venv/bin/activate;
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK;
python python_script.py
If you use mpi
mpiexec bash -c "module purge;source venv/bin/activate;
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK;
python python_script.py"
Page updated
Report abuse