Nix

Nix is a package manager and build system that parses reproducible build instructions specified in the Nix Expression Language. Nix expressions are pure functions taking dependencies as arguments and producing derivation specifying a reproducible build environment for the package. Nix stores the results of the build in unique addresses specified by a hash of the complete dependency tree, creating an immutable package store that allows for atomic upgrades, rollbacks, and concurrent installation of different versions of a package, essentially eliminating dependency hell.

Main Features

  1. Reproducible: this means that if a package works on one machine, it will also work on another.

  2. Declarative: you can share development and build environments for your projects, regardless of what programming languages and tools you’re using.

  3. Reliable: installing or upgrading one package can’t break other packages. This allows users to roll back to previous versions of installed packages after upgrading.

Managing Environment and Packages Using Nix

After you install Nix, you can use it to create new shell environments with programs that you want to use. Nix allows users to immediately use any program packaged with Nix, without installing it permanently. Also, you can manipulate or query Nix environments. You can also create reproducible shell environments given a declarative configuration in a Nix file.

In this tutorial you will learn how to create a new shell environment, use Nix packages without installing them. You will also know how to install Nix packages permanently using nix-env command and Nix files.

Using Packages without Installing

Nix allows you to create on-demands environments. You can use a packing inside that environment without installing it. This can be accomplished using the following command:

nix-shell -p <package-name>

For example, the following command creates a Nix environment and install nodejs in it. Then, you will be able to use the package inside the environment. Note that nodejs won’t be installed permanently and you won’t be able to use the package out of the environment.

nix-shell -p nodejs

The following output allows you use nodejs in the created shell.

Apptainer>

For example, to check installed version of Nodejs, run:

Apptainer> node --version
v18.17.1

Type exit or press CTRL-D to exit the shell, and Nodejs won’t be available anymore.

Running Commands in Nix Environments

You can run commands in Nix environments interactively or non-interactively.

To run a command interactively, you need to use --command option with nix-shell.

nix-shell -p <package-name> --command <cmd>

For example, to run node --version command in the Nix shell, run:

nix-shell -p nodejs --command "node --version"

This will show the version of Nodejs and then exit. To prevent this, add return at the end of the command.

nix-shell -p nodejs --command "node --version; return"

This will run the command and the shell won’t exit after running the command.

To run a command non-interactively, you need to use --run option with nix-shell.

nix-shell -p <package-name> --run <cmd>

This executes the command cmd in a non-interactive shell. If you hit Ctrl-C while the command is running, the shell exits.

Installing Nix Packages

You can install packages using nix-env or using a declarative configuration in a Nix file.

Query Available Packages

To show available packages for install, run the following:

$ nix-env --query  --available
0ad-0.0.26
0ad-data-0.0.26
0verkill-unstable-2011-01-13
0x-unstable-2022-07-11
0xtools-1.1.3
1am-20141106-git
1oom-1.0
1password-8.10.12-10.BETA
1password-8.10.9
1password-cli-2.20.0
2048-cli-unstable-2019-12-10
2048-cli-unstable-2019-12-10
2048-in-terminal-unstable-2022-06-13
20kly-1.5.0
2bwm-0.3
2d-array-export-to-quicklisp-502a46e2-git
2d-array-test-export-to-quicklisp-502a46e2-git
...

There is a set of over 80000 packages for the Nix package manager. You can search for packages here

You can use Nix expression to search for a package. For example, to show all packages with zip` in the name, run:

$ nix-env --query --available '.*zip.*'
bzip2-1.0.6
gzip-1.6
zip-3.0
…

Installing Packages Using nix-env

Yo can add packages to user environment. This packages are installed permanently. To install a package, run:

$ nix-env --install <package-name>

For example, to install Nodejs, run:

$ nix-env --install nodejs
installing 'nodejs-18.17.1'
this path will be fetched (10.34 MiB download, 94.74 MiB unpacked):
  /nix/store/nliha6yj2xmw5pghjfq2xqgwx77z77f5-nodejs-18.17.1-libv8
copying path '/nix/store/nliha6yj2xmw5pghjfq2xqgwx77z77f5-nodejs-18.17.1-libv8' from 'https://cache.nixos.org'...

To test the installed package, you can activate the environment and then test the package.

$ nix-shell -p

Then run,

Apptainer> $ node --version
v18.17.1

To show installed packages, you can run:

$ nix-env --query --installed
nix-2.17.0
nodejs-18.17.1
vim-9.0.1642

Uninstalling Packages

You can uninstall Nix packages using the nix-env command.

nix-env --uninstall <package-name>

For example, to uninstall nodejs-18.17.1, run:

nix-env --uninstall nodejs

You can show installed packages to confirm Nodejs was uninstalled successfully.

$ nix-env --query --installed
nix-2.17.0
vim-9.0.1642

Installing Packages Using Nix File (Declarative Shell Environments)

Nix declarative shell environments are powerful tools for managing software environments and dependencies in a reproducible and declarative manner. they’re created using the Nix package manager and its Nixpkgs repository. They can be applied in various contexts to manage software dependencies, configurations, and environments. The followings are some common use cases:

  • Development Environments: they can be used to create isolated development environments for different programming environments and frameworks. This is useful for projects with complex or specific dependencies.

  • Reproducible Builds: Declarative shell environments can be used to ensures that every build is reproducible by providing a consistent set of dependencies and build instructions. For example, This is useful for projects that need to be reproducible across different machines.

  • Continuous Integration (CI): they also can be used in CI pipelines to ensure that the same environment is used for testing and building software. This reduces the chances of build failures caused by differences in dependencies between the development environment and the CI environment.

Create a Nix File

The first step is creating a Nix file that contains the Nix expressions written in Nix language.]

Nix filename should end with .nix.

Create a Nix file and name it, for example, test-shell.nix and add the following content to it.

{ pkgs ? import <nixpkgs> {} }:

 pkgs.mkShell {
   name = "test-env"
   buildInputs = [
     pkgs.nodejs
   ];

 shellHook = ''
     echo hello! this is a test environment.
   '';
 }

The first line imports the nixpkgs package set. The helper function mkShell provided by the Nixpkgs library that creates an isolated development environment or shell environment with specific dependencies. buildInputs specifies the build-time dependencies required for building or running a software package. shellHook specifies a shell command or script that’s executed every time you enter the environment created by mkShell.

To enter the environment, run the following in the same directory as Nix file:

nix-shell test-shell.nix

This command will start downloading the missing packages and after the download completes, you are dropped into a new shell, which provides the packages specified in Nix file. This makes node available in $PATH. You can echo it and confirm that. Now you can run your commands in the Nix environment interactively as shown above.

You can also run commands non-interactively. For example,

$ nix-shell test-shell.nix  --run "node --version"
hello! this is a test environment.
v18.17.1

This prints the version of NodeJS that was installed using the Nix file.

Using Nix in Slurm Submission Script

After creating your Nix environment "Nix file," you can use it in your Slurm script because your program depends on the packages contained in the environment. In your Slurm script, there is a line you’d want to add right after the declaration of Slurm directives, module load <nix-module>. Where nix-module is the module name of the Nix stack that you installed before.

In this section, you create a new Declarative Shell Environment using a Nix file. For example, if you want to install some Python packages (Pandas and Matplotlib) that your project depends on using, create a Nix file with the following content.

{ pkgs ? import <nixpkgs> {} }:
  let
    my-python-packages = ps: with ps; [
      pandas
      matplotlib
      # other python packages
    ];
    my-python = pkgs.python3.withPackages my-python-packages;
 in my-python.env

Then, create a Python script (nix.py) and add the following content to it:

#!/usr/bin/env python3

import matplotlib as mplt
import pandas as pd

print("Matplotlib version is {}".format(mplt.__version__))
print("Pandas version is {}".format(pd.__version__))

Now, make the Python script executable:

chmod +x nix.py

To run the above python script using Slurm, you need to create a submission script subNix.sh with the following content:

#!/bin/bash

#SBATCH --job-name=nix        ## Name of the job
#SBATCH --output=nix_py.out       ## Output file
#SBATCH --time=10:00                ## Job Duration
#SBATCH --ntasks=1                  ## Number of tasks (analyses) to run
#SBATCH --cpus-per-task=1           ## The number of threads the code will use
#SBATCH --mem-per-cpu=1G           ## Real memory(MB) per CPU required by the job.

# load nix module of the installed nix stack
module load nix/nix5

srun nix-shell pythonTest.nix --run "./nix2.py"

To submit your Slurm script, run:

sbatch subNix.sh

After Slurm job completes, check the output of nix_py.out.

$ cat nix_py.out
Matplotlib version is 3.7.2
Pandas version is 2.0.3