Common Software

Python Environment

Python is a versatile and widely used programming language that’s particularly popular in scientific computing, data analysis, and artificial intelligence. To manage Python packages efficiently, we recommend using Conda, which is a powerful package manager and environment management system.

Why Use Conda?

Conda allows you to create isolated environments that can have their own specific versions of Python and packages installed. This is particularly useful in HPC environments where you may be working on multiple projects with different dependencies. By using Conda, you can:

Avoid conflicts between package versions.
Easily share your environment with others, ensuring reproducibility.
Install packages without needing administrative privileges.

How to Use It?

To get started with Conda, you can follow this workflow:

1. Load Conda Module

module load conda

Note that module load conda will load the default version. If you want a specified version, you can view the available versions via module spider conda and load it with the full name.

module load conda/2023a

2. Create Conda Environment from YAML File

You can specify all the packages and their versions in a YAML file. This is particularly useful for sharing environments or ensuring reproducibility.

Here’s an example YAML:

name: myenv
channels:
  - defaults
dependencies:
  - numpy
  - pandas
  - matplotlib

Once you have a YAML file (here using myenv.yml as example), you can run

conda env create -f myenv.yml

After that, the environment myenv is ready to be activated. You can check it by using conda env list.

Output

# conda environments:
#
myenv                    /fs1/home/username/.conda/envs/myenv
python                   /fs1/home/username/.conda/envs/python
pytorch                  /fs1/home/username/.conda/envs/pytorch
base                  *  /fs1/software/sstack/rhel_8/stacks/conda/2023a
amptk-1.5.5              /fs1/software/sstack/rhel_8/stacks/conda/2023a/envs/amptk-1.5.5

3. Using Conda virtual environments in your Slurm script

After your virtual environment myenv is created, you can use it in your Slurm script because your program depends on the packages contained in the environment.

In your Slurm script, there are two lines you’d want to add right after the declaration of Slurm directives, module load conda and conda activate my_env. myenv is the name of the virtual environment which was created earlier.

Here’s an example Slurm script using virtual environment myenv:

#!/bin/bash

#SBATCH --job-name=jobname   		## Name of the job
#SBATCH --output=job_output.out  	 ## Output file
#SBATCH --time=10:00           		## Job Duration
#SBATCH --ntasks=1             		## Number of tasks (analyses) to run
#SBATCH --cpus-per-task=1      		## The number of threads the code will use
#SBATCH --mem-per-cpu=100      		## Real memory(MB) per CPU required by the job.

## Load the python interpreter
module load conda
conda activate myenv

## Execute the python script, assuming your python script is program.py
srun -n 1 python program.py

On line 11, the conda module has been loaded. On line 12, the custom conda environment, my_env has been activated which contains the packages and dependencies, that the project requires to run.

Learn More

For the full workflow and more detailed documentation on building/using Conda environments, please check Conda Virtual Environments.

R

Open OnDemand

The Interactive App RStudio is available in Open OnDemand.

You can use it directly via Submission Environment - Module by selecting different versions of R prepared by the HPC team.

You can also use your own container by change the Submission Environment to be Container.

In Container Submission Environment, you can select your .SIF file.

Container Modules

For languages like R and Julia, although they can also be managed using Conda, sometimes, more complex environment configurations or system-level dependencies may be required, and this is where the containerized solution provided by Apptainer can better ensure project portability and consistency.

In addition, in the need to ensure complete consistency of the environment, or when running in a high-performance computing environment, Apptainer’s advantages are even more obvious.

To find the default R containers on Discovery, you can use the module spider command as shown below:

module --terse --redirect spider r | grep SIF

Output:

r/4.1.2-RStudio_Server_2022.02.2.485-SIF
r/4.1.3-RStudio_Server_2022.02.2.485-SIF
r/4.2.0-RStudio_Server_2022.02.2.485-SIF
r/4.2.3-RStudio_Server-VSCode-Jupyter-SIF
r/4.3.2-RStudio_Server-VSCode-Jupyter-SIF

All modules that use Apptainer containers have a name that ends with -SIF.

Customize Your Container

You can customize your own Apptainer container by creating a definition ( .def ) file.

Once you have your definition file ready, you can use the Apptainer command line tool to build a new SIF file.

For more detailed tutorial, please check Customize Your R Apptainer.

Julia

Open OnDemand

The Interactive App Julia is available in Open OnDemand.

You can use it directly via Submission Environment - Module by selecting different versions of Julia prepared by the HPC team.

You can also use your own container by change the Submission Environment to be Container.

In Container Submission Environment, you can select your .SIF file.

Container Modules

To find the default Julia containers on Discovery, you can use the module spider command as shown below:

module --terse --redirect spider julia | grep SIF

Output:

julia/1.6.7-SIF
julia/1.8.5-SIF
julia/1.9.2-SIF
julia/1.10.0-SIF

All modules that use Apptainer containers have a name that ends with -SIF.

Customize Your Container

You can customize your own Apptainer container by creating a definition ( .def ) file.

Once you have your definition file ready, you can use the Apptainer command line tool to build a new SIF file.

For more detailed tutorial, please check Customize Your Julia Apptainer.

MATLAB

Open OnDemand

The Interactive Apps MATLAB (Xfce) and MATLAB (Web) are available in Open OnDemand.

MATLAB (Xfce) will launch the graphical interface (GUI) for MATLAB on a compute node.
MATLAB (Web) will launch the web server (proxy) version of MATLAB on a compute node.

You can use them directly via Submission Environment - Module by selecting different versions of MATLAB prepared by the HPC team.

You can also use your own container by change the Submission Environment to be Container.

In Container Submission Environment, you can select your .SIF file.

Container Modules

To find the default MATLAB containers on Discovery, you can use the module spider command as shown below:

module --terse --redirect spider matlab | grep SIF

Output:

cst_studio_suite/2021-SIF
cst_studio_suite/2022-SIF

All modules that use Apptainer containers have a name that ends with -SIF.