Setup Conda Env in HPC Server

Configuring Conda environments on a High-Performance Computing (HPC) server can be tricky, particularly when integrating environment activation into bash scripts. This article provides a clear, step-by-step guide on how to set up and activate conda environments in a shared HPC environment.

1. Start by checking if the conda module is available on your HPC server:

module avail conda

2. Create a new conda environment with a specified name (you can also specify a Python version)

conda create -n <env_name> python=<version>

3. List the Conda environments to see their locations.

conda info --envs

You will notice that the base environment and newly created environments are located in different directories.

source <env_location>/bin/activate

Why is this a problem?

The conda activate <env> command works well in interactive shell sessions, where it sets up the environment as expected. However, when used in bash scripts, there are challenges because the script executes in a new subshell. This subshell doesn’t always retain the environment changes made by conda activate. To address this, you need to source the conda setup script in your bash script. This ensures that the necessary environment variables are correctly set up for the conda commands to function properly within the script.

4. Create a bash script (e.g., run_job.sh) for your HPC job

#!/bin/bash
#SBATCH --job-name=medium_run
#SBATCH --partition=medium
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=2

module load conda/conda
source <base_env_path>/bin/activate
conda activate <env_name>

Key Points:

1. Load Conda Module: Use module load conda/conda to make sure the conda module is available in your session.
2. Source Conda Script: source <base_env_path>/bin/activate sets up the necessary environment variables for conda commands.
3. Activate Environment: conda activate <env_name> activates your environment.