Creating a Virtual Environment

To get started, first you need to load the Conda module which is a tool that aims to simplify package management and deployment of data science and machine learning tools.

1. Login to Discovery and run

module load conda

2. Check out existing virtual environments

Before creating conda virtual environments on Discovery, there are a few list of already created virtual environments you can use right out of the box which are tailored to suit specific project needs. Some of these virtual environments are TensorFlow (with GPU support), PyTorch and QIIME. But if you require none of those, you can go ahead and create yours.

Show the list of all existing environments

conda env list

Output

# conda environments:
#
base                  *  /software/anaconda/anaconda3
alfalfa_gbs              /software/anaconda/anaconda3/envs/alfalfa_gbs
amptk-1.4.2              /software/anaconda/anaconda3/envs/amptk-1.4.2
hsc_prediction           /software/anaconda/anaconda3/envs/hsc_prediction
pytorch                  /software/anaconda/anaconda3/envs/pytorch
qiime2-2019.10           /software/anaconda/anaconda3/envs/qiime2-2019.10
redbiom                  /software/anaconda/anaconda3/envs/redbiom
soilsystems              /software/anaconda/anaconda3/envs/soilsystems
tensorflow-1.15.0        /software/anaconda/anaconda3/envs/tensorflow-1.15.0
tensorflow-2.0.0         /software/anaconda/anaconda3/envs/tensorflow-2.0.0
tensorflow-gpu-1.15.0    /software/anaconda/anaconda3/envs/tensorflow-gpu-1.15.0
tensorflow-gpu-2.0.0     /software/anaconda/anaconda3/envs/tensorflow-gpu-2.0.0

The output above contains a list of virtual environments on Discovery and their respective locations. The asterisk * symbol on the base environment specifies that the base is the current active virtual environment. Actually, the base is the default VE.

3. Create the environment

There are different ways to create a Conda virtual environment:

(1) Pre-defined ENV (Recommended)

Creating a Conda Environment Using a YAML File.

You can specify all the packages and their versions in a YAML file. This is particularly useful for sharing environments or ensuring reproducibility.

Here’s an example YAML:

name: myenv
channels:
  - conda-forge
dependencies:
  - python=3.6
  - numpy
  - pandas

Once you have a YAML file (here using myenv.yml as example), you can run

$ conda env create -f myenv.yml

After that, the environment myenv is ready to be activated.

You can also export your ENV as a YAML file, see Clone & Export for more details.

(2) Build from scratch

syntax: conda create -n <your_environment> --no-channel-priority -c <channel_name> <package_name>

This command only creates the environment without installing any packages.

$ conda create -n my_env

(3) Build & Install

This command creates the environment and also installs some essential packages for working with Python, all on one line. You can also specify the version of python you would like to install.

$ conda create -n myenv python=3.6 numpy pandas -c conda-forge

The myenv is the name of the virtual environment, python, numpy, and pandas are the specified packages and version you want to install in this new environment.

Flags Explained

Flag

Description

-n or --name

The name of the virtual environment

-c or --channel

Conda packages are downloaded from remote channels, which are URLs to directories containing the conda packages.

Now, conda will take a little while to search for the package(s) you specified to download and install it to your home directory /home/yourusername/.conda/envs/my_env. After you get the prompt Proceed ([y]/n)?, please hit the y and then the Enter key afterward to continue with the installation.

Once this phase is done the end of the output printed on your console should look like the one below.

...

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate my_env
#
# To deactivate an active environment, use
#
#     $ conda deactivate

4. Activate your newly created environment

To start using the packages installed in your environment, you have to activate the environment you just created using the command below.

syntax: conda activate <your_environment>

$conda activate my_env

After the environment has been activated, you would notice your shell prompt on the console changed from:

[yourusername@discovery-l1 ~]$

to

(my_env) [yourusername@discovery-l1 ~]$

This shows that you are currently in the my_env workspace. Also when you run the command conda env list, you should see the asterisk * symbol on the my_env line.

5. Show the list of installed packages

Use the conda list command to show the list of packages installed.

(my_env) [yourusername@discovery-l1 ~]$ conda list

You should get an output like the one below.

...

pip                       20.1.1                   py36_1
python                    3.6.10               h7579374_2
readline                  8.0                  h7b6447c_0
setuptools                49.2.0                   py36_0
sqlite                    3.32.3               h62c20be_0
tk                        8.6.10               hbc83047_0

...

Instead of showing the entire list, you can use the grep command to quickly search and verify if a given package is installed.

conda list | grep -i wheel

If after going through the installed packages, and you discovered that the package you want isn’t part of the installed essentials. Then, you can search for the package on the conda channels and install afterward.

6. Installing additional packages to your environment

For the Python programming example, search for and install the package → Scipy, a free and open-source Python library used for scientific computing and technical computing

(1-1) Search for the package(s):

syntax: conda search <package_name(s)>

(my_env) [yourusername@discovery-l1 ~]$ conda search scipy

It should print a list of scipy versions as well as their channels respectively. Whether you require the latest version of the package or not, choose the version that suits your purpose and also specify it’s channel.

(1-2) Search for the package(s) by channel:

syntax: conda search -c <channel> <package_name(s)>

(my_env) [yourusername@discovery-l1 ~]$ conda search -c conda-forge scipy

The first -c argument is of higher priority than the second, therefore priority decreases from left to right. conda-forge is a community channel made up of thousands of contributors.

Although specifying the channel is optional, it remains a good practice to specify the --channel or -c flag because it shows you a list of various versions of that package.

(1-3) Search for packages on Anaconda.org

Anaconda.org provides a very convenient package search function. You can use filters on the search page to find the package you need.

(2) Install the package(s):

syntax: conda install <package(s)>

(my_env) [yourusername@discovery-l1 ~]$ conda install scipy

Note that when installing a package, conda would also install all the dependencies required for that given package.

If the package installation fails, you will need to create your conda ENV from the beginning (see section 3. Create the environment) that contains the required packages.

7. Run the installed program

  • Next, launch the python CLI by typing python on your console.

    (my_env) [yourusername@discovery-l1 ~]$ python

    Now you should get an output like the one below.

    Python 3.10.5 | packaged by conda-forge | (main, Jun 14 2022, 07:04:59) [GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

    Notice the version of python installed is 3.6>

  • Next, import and use the scipy module.

    >>> from scipy.special import cbrt
>>> cb = cbrt([27, 64])
>>> print(cb)

[3. 4.]

    The output is [3. 4.].

Custom location for your Conda virtual environments

To use a custom location for your Conda virtual environments, specifically within a project directory, you can leverage the --prefix option when creating a new environment.

This option allows you to specify an exact path where the environment should be created, rather than using the default location ($HOME/.conda) for Conda environments.

For example, if you want to create a virtual environment called local-env

  • Using Relative Path (if you’re already in your project directory)

conda create --prefix ./local-env

  • Using Full Path

conda create --prefix /full/path/local-env

Note that Conda provides commands (such as conda env list) to quickly view and manage all environments created in the default location ($HOME/.conda).

Customizing the location will result in them not being viewed or managed by those commands.

Benefits

  • It makes it easy to tell if your project uses an isolated environment by including the environment as a subdirectory.

  • It makes your project more self-contained as everything, including the required software, is contained in a single project directory.

Exiting a conda environment

$ conda deactivate

Searching for a conda package

Syntax: conda search <package_name>

Example command to search for r-devtools

$ conda search r-devtools

Removing a Package From a conda Environment

Syntax:

conda remove --name <your_custom_environment_name> <package_name>

$ conda remove --name my_env scipy

The my_env is the name of the already existing environment and scipy is the name of the package which will be removed from the environment. This uninstalls the package together with its dependencies.

Delete an environment and everything in it

(1) from default location

If you want to destroy a given virtual environment that you created, you can use the command below which removes the environment and all the packages in it.

Syntax: conda env remove --name <your_custom_environment_name>

$ conda env remove --name my_env

(2) If it was created using --prefix (specified path)

If you used the --prefix option when creating a Conda virtual environment, the environment was created directly in the folder you specified.

You can remove it using the following command:

Syntax: conda env remove --prefix "your_custom_environment_path"

For example, if you are in the folder where the environment is:

$ conda env remove --prefix ./my_env

Before you delete a virtual environment, remember to make a backup first if there is important data or configuration in it.