Embedded Files
- Using Miniconda for HPC Environments
- Why Miniconda?
Anaconda is recommended if it allows the user to install and configure their environment. However, its large footprint and potential conflicts make it less suitable for most HPC environments. Instead, the HPC administration team advises using Miniconda to manage Conda environments.
📌 Important Note:
If you have previously installed Conda via an installer, it may have initialized itself in your ~/.bashrc file, which can cause conflicts in the HPC environment. It is recommended that you check and remove these entries if necessary.- Step 1: Load the Miniconda Module
Before using Conda, load the Miniconda module:
module load Miniconda3/22.11.1-1
- Step 2: Create a Conda Environment
Use the --prefix option to define the environment location. A recommended practice is to place it under your home directory:
conda create --prefix /home/<caseID>/mycondenv
📌 Warning:
Old Python libraries at ~/.local/lib/python* may conflict with this setup. If you encounter issues, consider removing them.- Step 3: Check Environment Information
List available Conda environments:
conda info -e
Your environment should be listed as:
<prefix-path>/mycondenv
- Step 4: Activate and Deactivate Conda Environment
To activate your newly created Conda environment:
source activate <prefix-path>/mycondenv
You will see the environment name in your shell prompt:
(<prefix-path>/mycondenv) [<caseID>@hpc6 ~]$
To deactivate the environment:
conda deactivate
- Step 5: Check Installed Packages
To view installed packages:
conda list
Example output:
# Name Version Build Channel
_libgcc_mutex 0.1 main
- Step 6: Improve Dependency Resolution Speed
For a faster dependency solver, install conda-libmamba-solver:
conda install conda-libmamba-solver
Then set libmamba as the default solver:
export CONDA_SOLVER=libmamba
- Step 7: Install Python Packages
To install a specific Python package (e.g., numpy). Check the repository for packages.
conda install numpy
- Step 8: Remove Conda Packages or Environment
To remove a package or environment completely:
conda remove --prefix <prefix-path>/mycondenv --all
- Step 9: Create a Reproducible Python Environment
To ensure reproducibility, create a requirements file:
conda list -e > requirements.txt
Install the required packages from the file:
conda install --file requirements.txt
- Use Case
Check the process of installing IBM CPLEX on HPC using Conda.
Page updated
Report abuse