Module Environments and Commands

What are Modules?

Modules are used to connect to different software packages to run your scripts. Different versions of the software can be switched quickly and easily using the module commands. A variety of popular software like R, Python, MatLab, etc are available in the Discovery.

What if the Module you are looking for isn’t available?

If the modules you are looking for isn’t available in the discovery, then you can submit the software request by filling the form in this link → (HPC software request). After the module is updated into the discovery, you will be notified via email.

When to use Modules?

For example, if you have a perl script that needs to be run, you first need to load the perl module to your environment. You can choose the desired version of the perl module from the list of available modules. You can load as many modules as you wish to.

But remember that when loading multiple versions of the same compiler, the module loaded last will always have precedence. The best way to avoid this confusion is to unload the module when it’s no longer needed.

Module Commands

The below table shows some useful module commands in discovery.

Commands Syntax Description

Commands	Syntax	Description
module avail	`module avail`	Shows all the available modules
module spider	`module spider <software-name>`	Retrieves the specific module
module load	`module load <software-name>`	Loads the specific module to your environment
module list	`module list`	Lists all the loaded modules
module swap	`module swap <m1> <m2>`	Unloads module m1 and loads module m2
module unload	`module unload <software-name>`	Gets rid of the specific module
module purge	`module purge`	Removes all the currently loaded modules
module help	`module help`	Prints the help information.

module avail

module avail

Shows all the available modules

module spider

module spider <software-name>

Retrieves the specific module

module load

module load <software-name>

Loads the specific module to your environment

module list

module list

Lists all the loaded modules

module swap

module swap <m1> <m2>

Unloads module m1 and loads module m2

module unload

module unload <software-name>

Gets rid of the specific module

module purge

module purge

Removes all the currently loaded modules

module help

module help

Prints the help information.

Module Avail

To view the list of available modules in discovery, run the following command.

Syntax: module avail

 module avail

Output

------------------------------------------------------------------------------------- /etc/modulefiles --------------------------------------------------------------------------------------
 mpi/mpich-x86_64    mpi/mpich-3.0-x86_64    mpi/openmpi-x86_64 (D)

-------------------------------------------------------------------------------- /software/anaconda/modules ---------------------------------------------------------------------------------
 alfalfa_gbs    anaconda     anaconda3         pbbioconda    qiime2-2019.10    soilsystems              tensorflow-gpu/2.0.0 (D)    tensorflow/2.0.0 (D)
 amptk/1.4.2    anaconda2    hsc_prediction    pytorch       redbiom           tensorflow-gpu/1.15.0    tensorflow/1.15.0

----------------------------------------------------------------------------- /software/custom/el7/modulefiles ------------------------------------------------------------------------------
 bayescan/2.1    fastsimcoal2/2.6.0.3    gcc/4.9.4    openmpi/1.5.5-transat    petsc/3.2-p7-transat    transat/5.7.1

------------------------------------------------------------- /fs1/software/spack/0.13/share/spack/modules/linux-centos7-x86_64 -------------------------------------------------------------
 adios2/2.5.0-gcc-9.2.0-openmpi-24g24ah                       netcdf-c/4.7.2-gcc-9.2.0-openmpi-vjoppci                         r-data-table/1.12.2-gcc-9.2.0-tm3afks                 (D)
 ant/1.10.0-gcc-9.2.0-bvkawaf                          (D)    netcdf-fortran/4.5.2-gcc-9.2.0-openmpi-p2elujv            (D)    r-dbi/1.0.0-gcc-9.2.0-v5d3azu                         (D)
 argtable/2-13-gcc-9.2.0-3lwtbg7                       (D)    netcdf/4.7.2-gcc-9.2.0-openmpi-j22xar2                    (D)    r-deldir/0.1-23-gcc-9.2.0-hlmpvdb
 arpack-ng/3.7.0-gcc-9.2.0-openblas-openmpi-fp7yot3    (D)    netlib-lapack/3.8.0-gcc-9.2.0-hsgmmfb                            r-desc/1.2.0-gcc-9.2.0-rc3oxh6
 at-spi2-atk/2.26.2-gcc-9.2.0-ne7dldi                  (D)    netlib-lapack/3.8.0-gcc-9.2.0-7lemu3c                            r-devtools/2.1.0-gcc-9.2.0-mxzwt22
 at-spi2-core/2.28.0-gcc-9.2.0-ikgtwxh                 (D)    netlib-scalapack/2.0.2-gcc-9.2.0-openblas-openmpi-tvlazvq (D)    r-digest/0.6.20-gcc-9.2.0-ik2krtt                     (D)
 atk/2.30.0-gcc-9.2.0-d7uvfiw                          (D)    netlib-xblas/1.0.248-gcc-9.2.0-zjfxgwl                           r-doparallel/1.0.15-gcc-9.2.0-tsfoz7q
 augustus/3.3.2-gcc-9.2.0-6piwhgu                      (D)    nettle/3.4.1-gcc-9.2.0-s5j4zqp                            (D)    r-dosnow/1.0.18-gcc-9.2.0-nnq3aut
 autoconf-archive/2019.01.06-gcc-9.2.0-43fyr6f                ngsepcore/4.0.0-gcc-9.2.0-474wh3n                                r-dplyr/0.8.3-gcc-9.2.0-lzxcfzh
 autoconf/2.69-gcc-9.2.0-ytaohc4                              ninja/1.9.0-gcc-9.2.0-2qvepgo                             (D)    r-dplyr/0.8.3-gcc-9.2.0-2wfidt7                       (D)
 automake/1.16.1-gcc-9.2.0-4h2g7c4                            numactl/2.0.12-gcc-9.2.0-aj6bh6t                          (D)    r-e1071/1.7-2-gcc-9.2.0-lrbslzc
 bamtools/2.5.1-gcc-9.2.0-egeykzb                      (D)    nwchem/6.8.1-gcc-9.2.0-openblas-openmpi-ccgzl5e           (D)    r-ellipsis/0.2.0.1-gcc-9.2.0-nkwlj4t
 bcftools/1.9-gcc-9.2.0-oz7iegl                        (D)    oniguruma/6.1.3-gcc-9.2.0-fwjgg3u                         (D)    r-ellipsis/0.2.0.1-gcc-9.2.0-orbbzn3                  (D)
 bdftopcf/1.0.5-gcc-9.2.0-zsphr3k                             opari2/2.0.5-gcc-9.2.0-fobgwfp                                   r-evaluate/0.14-gcc-9.2.0-gyvdolg
 bdw-gc/8.0.0-gcc-9.2.0-p6hd7qd                               openbabel/2.4.1-gcc-9.2.0-m2yuj3d                                r-expm/0.999-4-gcc-9.2.0-7rjk2yu
 binutils/2.32-gcc-9.2.0-tzlq2cl                       (D)    openblas/0.3.7-gcc-9.2.0-epeitvj                          (D)    r-fansi/0.4.0-gcc-9.2.0-vxps4xj
 bison/3.4.2-gcc-9.2.0-xwnwxvw                         (D)    openfoam-org/7-gcc-9.2.0-openmpi-xfwbnsq                         r-feddata/2.5.7-gcc-9.2.0-openmpi-at4kcc7
 blast-plus/2.9.0-gcc-9.2.0-5bhhtpv                    (D)    openfoam/1912-gcc-9.2.0-openmpi-vmdetr2                          r-foreach/1.4.7-gcc-9.2.0-qwslptj
 boost/1.70.0-gcc-9.2.0-ndry3ar                        (D)    openjdk/1.8.0_202-b08-gcc-9.2.0-vn2k6b7                          r-foreign/0.8-72-gcc-9.2.0-epz7bi3
 bowtie2/2.3.2-gcc-4.8.5-dp2quvb                              openjdk/11.0.2-gcc-9.2.0-iuj2qln                          (D)    r-formatr/1.7-gcc-9.2.0-kmhhqhk
 bowtie2/2.3.5.1-gcc-9.2.0-4iv5wso                     (D)    openmpi/3.1.4-gcc-9.2.0-r6y3ifc                                  r-formula/1.2-3-gcc-9.2.0-gmhcjnh
 bpp-core/2.2.0-gcc-9.2.0-qrovxx3                      (D)    openmpi/3.1.4-gcc-9.2.0-7chkypj                                  r-fs/1.3.1-gcc-9.2.0-moko76e
 bpp-phyl/2.2.0-gcc-9.2.0-qf3itbg                      (D)    openmpi/3.1.6-gcc-9.2.0-skcle4f                                  r-gdata/2.18.0-gcc-9.2.0-utmgdvb
 bpp-seq/2.2.0-gcc-9.2.0-ipxokok                       (D)    openmpi/4.0.3-gcc-9.2.0-gl6vlx7                           (D)    r-gdistance/1.3-1-gcc-9.2.0-e7ceeyl
 busco/3.0.1-gcc-9.2.0-openmpi-65wij7u                        openssl/1.1.1d-gcc-9.2.0-c3qm3sp                                 r-ggmap/3.0.0-gcc-9.2.0-lorh5ro                       (D)
 bwa/0.7.17-gcc-9.2.0-lzcib6y                                 orca/4.2.1-gcc-9.2.0-openmpi-lqmvcro                      (D)    r-ggplot2/3.2.0-gcc-9.2.0-sxcv4yi
 bzip2/1.0.8-gcc-9.2.0-p6q2can                         (D)    pandoc/2.7.3-gcc-9.2.0-7hxzwvt                                   r-ggplot2/3.2.0-gcc-9.2.0-6pc4fx6                     (D)
 c-blosc/1.17.0-gcc-9.2.0-2xxpn5s                             pango/1.41.0-gcc-9.2.0-ab67i5l                                   r-gh/1.0.1-gcc-9.2.0-2ax2ets
...

The above output shows the list of all the available modules. If you want to search for a specific module using module avail, the following command can be used.

Syntax: module -r avail <regex-epression>

module -r avail '^gcc'

Output:

 ------------------------------------------------------------- /software/custom/el7/modulefiles --------------------------------------------------------------
 gcc/4.9.4

 --------------------------------------------- /fs1/software/spack/0.13/share/spack/modules/linux-centos7-x86_64 ---------------------------------------------
 gcc/8.3.0-gcc-9.2.0-czqf5ch    gcc/9.2.0-gcc-4.8.5-wqdecm4

------------------------------------------- /fs1/software/packages/spack/share/spack/modules/linux-centos7-x86_64 -------------------------------------------
 gcc/6.3.0-gcc-8.2.0-ceghynm    gcc/7.3.0-gcc-8.2.0-v7yiaer    gcc/8.2.0-gcc-4.8.5-sxbf4jq    gcc/8.2.0-gcc-8.2.0-7cqfkrl

--------------------------------------------------------- /fs1/software/packages/custom/modulefiles ---------------------------------------------------------
 gcc/latest    gcc/520    gcc/530 (D)

Where:
 D:  Default Module

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".

Output shows the list of available modules for gcc which starts with the string 'gcc' .

Module Spider

Module spider can also be used to search for any specific module. For example, if you want to search for any particular module like perl and only its associated versions, use the below command

Syntax: module spider <software-name>

module spider perl

Output:

 -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  perl:
  -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
     Versions:
        perl/5.26.2-gcc-4.8.5-7tmxubx
        perl/5.26.2-gcc-6.3.0-xkfjfvn
        perl/5.26.2-gcc-6.3.0-6wij3ok
        perl/5.26.2-gcc-8.2.0-fx6xlmi
        perl/5.26.2-gcc-8.2.0-ndoqgto
        perl/5.26.2
        perl/5.30.0-gcc-9.2.0-h7kut76
     Other possible modules matches:
        perl-algorithm-diff  perl-alien-build  perl-alien-libxml2  perl-bio-perl  perl-capture-tiny  perl-class-data-inheritable  perl-clone  perl-config-general  perl-data-dumper  ...

  -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  To find other possible module matches execute:

      $ module -r spider '.*perl.*'

  -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  For detailed information about a specific "perl" package (including how to load the modules) use the module's full name.
  Note that names that have a trailing (E) are extensions provided by other modules.
  For example:

     $ module spider perl/5.30.0-gcc-9.2.0-h7kut76
  -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

It displays the modules associated with the different versions of perl and now you can load the appropriate version.

Module Load

To load a specific version of the perl from the list of available modules, you can do it by running the module load command. The syntax for the module load command is the following.

Syntax: module load <software-name>

module load perl/5.26.2

You can also load several modules by typing the module names one by one in a single line. Run the below single-line command to load multiple modules.

Syntax: module load <sofware-name1>…<software-n>

module load perl julia ruby

The above command loads the latest version for perl, julia and ruby because the version isn’t specified. To check the list of loaded modules, module list command is used which is discussed in the next section.

Press the tab key in your keyboard to complete the software names and versions. The names sometimes contain special characters, spaces and rather than typing the complicated filename, you can start typing the beginning of the name, and then hit tab key afterward.

Module List

The module list command helps to list all the currently loaded modules in your environment.

Syntax: module list

module list

Output:

Currently Loaded Modules:
1) perl/5.30.0-gcc-9.2.0-h7kut76   2) julia/1.4.2-gcc-9.2.0-ncvp5em   3) ruby/2.6.2-gcc-9.2.0-vbzxuyf

The output shows the list of currently loaded modules.

Module Unload

To remove a specific module which you no longer need or you want to discard it, run the below command.

Syntax: module unload <software-name>

 module unload perl

Output:

Currently Loaded Modules:
1) ruby/2.6.2-gcc-9.2.0-vbzxuyf   2) mpi/openmpi-x86_64

Module perl has been successfully removed from your environment.

You have to run the module list command after the unload command to view the list of the currently loaded modules.

Module Purge

If you want to get rid of all the loaded modules in your environment, use the below command

Syntax: module purge

module purge

Output:

No modules loaded

The above output after running the module purge and list command shows that all the loaded modules have been removed from the environment successfully.

Note that Module unload is used to remove a specific module from your environment and module purge is used to get rid of all the modules.

Module Help

This command helps to explore all the options, commands and arguments associated with the module command.

Syntax: module help

module help

Output:

Usage: module [options] sub-command [args ...]

Options:
-h -? -H --help                   This help message
-s availStyle --style=availStyle  Site controlled avail style: system (default: system)
--regression_testing              Lmod regression testing
-D                                Program tracing written to stderr
--debug=dbglvl                    Program tracing written to stderr (where dbglvl is a number 1,2,3)
--pin_versions=pinVersions        When doing a restore use specified version, do not follow defaults
-d --default                      List default modules only when used with avail
-q --quiet                        Do not print out warnings
--expert                          Expert mode
-t --terse                        Write out in machine readable format for commands: list, avail, spider, savelist
--initial_load                    loading Lmod for first time in a user shell
--latest                          Load latest (ignore default)
--ignore_cache                    Treat the cache file(s) as out-of-date
--novice                          Turn off expert and quiet flag
--raw                             Print modulefile in raw output when used with show
-w twidth --width=twidth          Use this as max term width
...

User-defined Modules

Discovery users can install their own software and create modulefiles to use it for their computational tasks.

Create a directory called modulefiles and add it to the module path. Run the below commands:
```
mkdir $HOME/modulefiles && module use $HOME/modulefiles
```

Now, install the python version 3.9.0 which isn’t present in the Discovery’s system modules. Run the following commands one by one:

wget https://www.python.org/ftp/python/3.9.0/Python-3.9.0.tar.xz && tar xf Python-3.9.0.tar.xz

+ It downloads the python-3.9.0 software package and extracts it.

Change to the created directory called Python-3.9.0
```
cd Python-3.9.0
```
Generate the Makefile and run the Makefile to build and install the python package.
```
./configure --prefix=$HOME/pkg/python/3.9.0 && make && make install
```
--prefix flag is used to specify the location where you want to install to.
Now, switch to the modulefiles directory again and then create a directory called python inside it.
```
cd ~/modulefiles
mkdir python && cd python
```

Create a file called 3.9.0.lua.

Syntax: vi <your-version>.lua

vi 3.9.0.lua

Paste the below code inside the file. Save and close the file.

local home    = os.getenv("HOME")
local pkgName = myModuleName()
local version = myModuleVersion()
local pkg     = pathJoin(home,"pkg",pkgName,version,"bin")
whatis("Name: Python")
whatis("Version: 3.9.0")
whatis("Short Description: An user-defined module for python 3.9.0")
prepend_path("PATH", pkg)

Variables value

Variables	value
home	`/home/<your-username>`
pkgName	`python`
version	`3.9.0`
pkg	`home/<your-username>/pkg/python/3.9.0/bin`

home

/home/<your-username>

pkgName

python

version

3.9.0

pkg

home/<your-username>/pkg/python/3.9.0/bin

It adds the path in the pkg variable to the user’s path.

Now, run module avail command to verify that the Python/3.9.0 has been added successfully or not.

module avail python/3.9.0

Output:

--------------------------------------------------------------------------------- /home/<your-username>/modulefiles ---------------------------------------------------------------------------------
python/3.9.0


Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".

Output shows that the user-defined module for Python/3.9.0 has been created under the home/<your-username>/modulefiles directory.

To load the module, run the below commands:
```
module load python/3.9.0
```
Run module list command to find the list of loaded modules in your environment.
```
module list
```
Output:
```
Currently Loaded Modules:
1) python/3.9.0
```
Now, you can run and execute Python programs using the user-defined modules python/3.9.0

References

For more information regarding the personal module files, please refer to the Lmod page https://lmod.readthedocs.io/en/latest/020_advanced.html