Creating a Package

  • Before you start creating packages, you need to set the EDITOR variable, where you specify your preferred editor while using Spack. For example, to choose vim as an editor, you type:

    export EDITOR='/usr/bin/vim'
  • For more details on creating packages, visit Spack Documentation.

Adding a Package Repository

  • In this tutorial, you will learn how to re-create some already created packages. Therefore, you will add a package repository myproject, which you have created before, to avoid modifying Spack installation with the package you are creating. To add the myproject repository, type:

    spack repo add  $SPACK_ROOT/var/spack/repos/myproject

Creating the Package File

  • As mentioned before, Spack comes with thousands of build-in packages. Suppose you want to create a new package, Spack allows users to create new packages in their own package repositories. To create a new package, you type:

    spack create <url>
  • url is the link that contains the software repository.

  • The spack create command builds a new package from a template by taking the location of the package source code and using it to:

    1. Fetch the code.

    2. Create a package skeleton.

    3. Open the file up in your editor of choice

  • For example, to create a new package for mpileaks, you type:

    spack create https://github.com/LLNL/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz
  • Spack will create the package file in $SPACK_ROOT/var/spack/repos/myproject and name it package.py. Also, it opens the file using vim and the file looks like the following:

    # Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
    # Spack Project Developers. See the top-level COPYRIGHT file for details.
    #
    # SPDX-License-Identifier: (Apache-2.0 OR MIT)
    
    # ----------------------------------------------------------------------------
    # If you submit this package back to Spack as a pull request,
    # please first remove this boilerplate and all FIXME comments.
    #
    # This is a template package file for Spack.  We've put "FIXME"
    # next to all the things you'll want to change. Once you've handled
    # them, you can save this file and test your package like this:
    #
    #     spack install mpileaks
    #
    # You can edit this file again by typing:
    #
    #     spack edit mpileaks
    #
    # See the Spack documentation for more information on packaging.
    # ----------------------------------------------------------------------------
    
    from spack import *
    
    
    class Mpileaks(AutotoolsPackage):
        """FIXME: Put a proper description of your package here."""
    
        # FIXME: Add a proper url for your package's homepage here.
        homepage = "https://www.example.com"
        url      = "https://github.com/LLNL/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz"
    
        # FIXME: Add a list of GitHub accounts to
        # notify when the package is updated.
        # maintainers = ['github_user1', 'github_user2']
    
        version('1.0', sha256='2e34cc4505556d1c1f085758e26f2f8eea0972db9382f051b2dcfb1d7d9e1825')
    
        # FIXME: Add dependencies if required.
        # depends_on('foo')
    
        def configure_args(self):
            # FIXME: Add arguments other than --prefix
            # FIXME: If not needed delete this function
            args = []
            return args
  • Close the file and try to install it using the spack install command:

    spack install mpileaks
  • You will get the following error:

    >> 35    configure: error: unable to locate adept-utils installation
  • This error indicates that configure is unable to find the installation location of a dependency.

  • You need to edit the file and add:

    1. Some descriptions about the software.

    2. Required dependencies.

    3. The configuration arguments needed to build the package.

    4. Variants.

  • If you want to edit the package.py file, you can use the spack edit command:

    spack edit mpileaks

Adding Package Documentation

  • To add some documentations, you make the following changes:

    1. Remove the instructions between dashed lines at the top

    2. You add a proper description of the package. Replace the first FIXME comment with Tool to detect and report MPI objects like MPI_Requests and MPI_Datatypes.

    3. Update the homepage property with the correct link of the home page of the package. For this example, use https://github.com/LLNL/mpileaks

    4. Uncomment the maintainers property and add your GitHub user name.

  • Once those changes are done, the file looks like:

    from spack import *
    
    
    class Mpileaks(AutotoolsPackage):
        """Tool to detect and report MPI objects like MPI_Requests and MPI_Datatypes."""
    
        homepage = "https://github.com/LLNL/mpileaks"
        url      = "https://github.com/LLNL/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz"
    
        maintainers = ['altahat2003']
    
        version('1.0', sha256='2e34cc4505556d1c1f085758e26f2f8eea0972db9382f051b2dcfb1d7d9e1825')
    
        # FIXME: Add dependencies if required.
        # depends_on('foo')
    
        def configure_args(self):
            # FIXME: Add arguments other than --prefix
            # FIXME: If not needed delete this function
            args = []
            return args
  • The changes won’t help you to build the software. If you try to install it again, you will have the same error you had before.

Adding Dependencies

  • You need to find the dependencies of the software and add them in the form depends_on('foo'). After reviewing the documentation of mpileaks , you will find that the mpileaks software depends on mpi, adept-utils, and callpath. You add those dependencies to the package.py file using the depends_on directive as follows:

    depends_on('mpi')
    depends_on('adept-utils')
    depends_on('callpath')
  • The file will look like:

    from spack import *
    
    
    class Mpileaks(AutotoolsPackage):
        """Tool to detect and report MPI objects like MPI_Requests and MPI_Datatypes."""
    
        homepage = "https://github.com/LLNL/mpileaks"
        url      = "https://github.com/LLNL/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz"
    
        maintainers = ['altahat2003']
    
        version('1.0', sha256='2e34cc4505556d1c1f085758e26f2f8eea0972db9382f051b2dcfb1d7d9e1825')
    
        depends_on('mpi')
        depends_on('adept-utils')
        depends_on('callpath')
        def configure_args(self):
            # FIXME: Add arguments other than --prefix
            # FIXME: If not needed delete this function
            args = []
            return args
  • Adding dependencies tells Spack that it must ensure that these dependencies are installed before it can build the package.

  • If you try to install the package after adding the required dependencies, you will get a lot further. But the software won’t build successfully due to the same error.

Dependency Type

  • There are four types of dependencies:

    1. build: the dependency will be added to the PATH and PYTHONPATH at build-time.

    2. link: the dependency will be added to Spack’s compiler wrappers, automatically injecting the appropriate linker flags, including -I, -L, and RPATH/RUNPATH handling.

    3. run: the dependency will be added to the PATH and PYTHONPATH at run-time. This is true for both spack load and the module files Spack writes.

    4. test: the dependency will be added to the PATH and PYTHONPATH at build-time. The only difference between build and test is that test dependencies are only built if the user requests unit tests with spack install --test.

  • You can specify dependency type in the depends_on directive. For example, the following code makes cmake as build and run dependencies:

    depends_on('cmake', type='build')
    depends_on('py-numpy', type=('build', 'run'))
  • If the dependency type isn’t specified, Spack uses a default of ('build', 'link'). This is the common case for compiler languages. Non-compiled packages like Python modules commonly use ('build', 'run'). This means that the compiler wrappers don’t need to inject the dependency’s prefix/lib directory. But the package needs to be in PATH and PYTHONPATH during the build process and later when a user wants to run the package.

Conditional Dependencies

  • Sometimes you may have a package that only requires a dependency under certain conditions. For example, you may have a package that has optional MPI support, - MPI is a dependency only when you want to enable MPI support for the package. In that case, you could say something like:

    variant('mpi', default=False, description='Enable MPI support')
    
    depends_on('mpi', when='+mpi')
  • You added a variant named mpi, and if you enable this variant in the configuration, the package needs mpi as a dependency.

    If a dependency/feature of a package isn’t typically used, you can save time by making it conditional (since Spack won’t build the dependency unless it’s required for the Spec).

Specifying Configure Arguments

  • Since the package hasn’t been built successfully, you need to solve the problem. You explore the configure file which is placed in the source directory of the package.

  • To move to the build directory you can use the spack cd command:

    spack cd mpileaks
  • Given this is a simple package built with configure and you know that the installation directories need to be specified.

  • You can use its help to see what command-line options are available for the software.

  • To get details about configure file, type:

    ./configure --help
  • The following output shows that you can specify the paths for the two concrete dependencies with the following options:

  • --with-adept-utils=PATH

  • --with-callpath=PATH

  • Now you go back to the $SPACK_ROOT and edit the package.py using spack edit mpileaks. You add the following lines to the args:

    '--with-adept-utils={0}'.format(self.spec['adept-utils'].prefix),
    '--with-callpath={0}'.format(self.spec['callpath'].prefix)
  • Now the package.py file looks like:

    from spack import *
    
    
    class Mpileaks(AutotoolsPackage):
        """Tool to detect and report MPI objects like MPI_Requests and
            MPI_Datatypes."""
    
        homepage = "https://github.com/LLNL/mpileaks"
        url      = "https://github.com/LLNL/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz"
    
        maintainers = ['altahat2003']
    
        version('1.0', sha256='2e34cc4505556d1c1f085758e26f2f8eea0972db9382f051b2dcfb1d7d9e1825')
    
        depends_on('mpi')
        depends_on('adept-utils')
        depends_on('callpath')
    
        def configure_args(self):
            args = [
                    '--with-adept-utils={0}'.format(self.spec['adept-utils'].prefix),
                    '--with-callpath={0}'.format(self.spec['callpath'].prefix)
                                ]
            return args
  • Now, if you try to install the package, it will install successfully.

Adding Variants

  • Also, you can add variants to the package.py file. Variants represent some arguments of the package. To find what arguments you can add as variants, you look at the help of the configure file as follows:

    ./configure --help
  • The following output shows that there are some optional packages such as --with-stack-start-c and --with-gnu-ld. You will add those options as variants to the package.py file of mpileaks. You edit the file again by typing:

    spack edit mpileaks
  • Then you add the variants as follows:

    variant('startframes', values=int, default=0,
            description='Specify the number of stack frames to truncate')
    
    variant('gnuld', default=False,
            description='assume the C compiler uses GNU ld')
        depends_on('mpi')
        depends_on('adept-utils')
        depends_on('callpath')
    def configure_args(self):
            # FIXME: Add arguments other than --prefix
            # FIXME: If not needed delete this function
            args = [
                    '--with-adept-utils={0}'.format(self.spec['adept-utils'].prefix),
                    '--with-callpath={0}'.format(self.spec['callpath'].prefix)
                    ]
            startframes = int(self.spec.variants['startframes'].value)
            if startframes:
                args.extend([
                    '--with-stack-start-c={0}'.format(startframes)
                    ])
            gnuld = int(self.spec.variants['gnuld'].value)
            if gnuld:
                args.extend([
                    '--with-gnu-ld={0}'.format(start),
                    ])
            return args
  • First, you add the variants one by one. You choose a proper name for the variant. Also, specify the values of the variant and what’s the default value. You can also give a description of the variant. For example, for --with-gnu-ld, you added a variant and named it gnuld and the default value is False. values wasn’t specified because it takes only two values; True or False. But for --with-stack-start-c, the values was specified as int because it takes integer values.

  • In the configure_args function, it checks if the value of the variant isn’t 0, it will be added to the package configuration. If you show the information of the mpileaks package, you see that the added variants are available:

    spack info mpileaks

    Output:

Variants:
    Name [Default]     Allowed values          Description
    ===============    ====================    ===============================

    gnuld [off]        on, off                 assume the C compiler uses GNU
                                               ld
    startframes [0]    int(x=0) -> int or      Specify the number of stack
                       long int(x, base=10)    frames to truncate
  • Variants have been updated in the release 'v0.17.0'. Now you can make the variant conditional by adding a when=<spec> clause to the variant. This will allow variants to be conditional based on the version or other attributes of a package. The syntax to model variants that are new in new versions of a package looks like:

    variant('version_based', default=False, when='@2.0:', description="Variant that is only available in versions 2.0 and later")
  • The directive above means that the variant version_based for the package starting at version 2.0. Any command that refers to it for a version that doesn’t satisfy the constraint should fail with an error.

  • For example, the following class has a variant bar when the foo package is at version 2.0 or higher.

    class Foo(Package):
           ...
           variant('bar', default=False, when='@2.0:', description='help message')
  • The syntax to model variants that are dependent on other variants looks like:

    variant('variant_based', default=False, when='+version_based', description="Variant that depends on another variant")
  • For example, the package foo has a variant bar when the spec satisfies platform=darwin condition but not other platforms.

    variant('bar', default=True, when='platform=darwin', description='help2')