HY2FOAM

hy2Foam is an open-source two-temperature computational fluid dynamics (CFD) solver that has been developed to tackle the highly complex flow physics of the hypersonic planetary atmospheric entry. It was implemented within the OpenFOAM framework.

Installing HY2FOAM

The official documentation says that hy2foam can be installed on Ubuntu 20.04 LTS with gcc and g++ at version 7 only. Therefore, Apptainer can make the installation easy.

Creating a Definition File

The first step in installing hy2foam is creating a definition file (build recipe) that tells Apptainer how to build a container image.

As stated in the installation page, you should follow the following:

Make sure that you install it on Ubuntu 20.04 LTS, 22.04 LTS isn’t supported yet.
Make sure that the gcc and g++ version is 7. This means that if the version is greater than 7, make sure to downgrade.
Install its requirements.
hy2foam can only be install on top of OpenFOAM version v1706. You need to download this version, and its ThirdParty
Untar the .tgz files, so you can install OpenFOAM from source.
Source the bashrc of OpenFOAM.
Install OpenFAOM.
Change working directory to WM_PROJECT_USER_DIR which sat up in the bashrc file, and then clone the hyStrath GitHub repository.
hy2foam has three modules, you need to specify which module, and then install it.

Create a definition .def file (i.e hy2foam.def) with the following content:

Bootstrap: docker
From: ubuntu:20.04

%post
    # Install gcc and g++ to version 7
    apt-get update
    apt-get install -y g++-7 gcc-7
    update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-7 7
    update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-7 7

    # Install requirements
    apt-get install -y software-properties-common
    add-apt-repository ppa:rock-core/qt4
    apt-get install -y wget build-essential flex bison cmake zlib1g-dev \
                       libboost-system-dev libboost-thread-dev libopenmpi-dev openmpi-bin \
                       gnuplot libreadline-dev libncurses-dev libxt-dev \
                       qt4-dev-tools libqt4-dev libqt4-opengl-dev freeglut3-dev libqtwebkit-dev \
                       libscotch-dev libcgal-dev git bc


    # Downgrade gcc and g++ to version 7
    update-alternatives --set gcc /usr/bin/gcc-7
    update-alternatives --set g++ /usr/bin/g++-7
    update-alternatives --config gcc
    update-alternatives --config g++

    # Install OpenMPI development libraries and tools.
    apt-get install -y openmpi-bin openmpi-common libopenmpi-dev
    update-alternatives --config gcc
    update-alternatives --config g++
    update-alternatives --set gcc /usr/bin/gcc-7
    update-alternatives --set g++ /usr/bin/g++-7

    # Create necessary directory where to download OpenFOAM
    mkdir -p /opt/OpenFOAM/


    # Download and extract OpenFOAM v1706 and ThirdParty sources
    cd /opt/OpenFOAM
    wget -q https://sourceforge.net/projects/openfoam/files/v1706/OpenFOAM-v1706.tgz
    wget -q https://sourceforge.net/projects/openfoam/files/v1706/ThirdParty-v1706.tgz
    tar -xzf OpenFOAM-v1706.tgz
    tar -xzf ThirdParty-v1706.tgz

    # Update the bashrc file to setup the basic env variable appropriatly
    cd OpenFOAM-v1706/etc

    # The following 2 commands make sure that OpenFOAM and hy2foam are not installed in root
    sed -i 's|FOAM_INST_DIR=$HOME/$WM_PROJECT|FOAM_INST_DIR=/opt/$WM_PROJECT|' bashrc
    sed -i 's|WM_PROJECT_USER_DIR=$HOME/$WM_PROJECT/$USER-$WM_PROJECT_VERSION|WM_PROJECT_USER_DIR=/opt/$WM_PROJECT/user-$WM_PROJECT_VERSION|' bashrc

    # Source the bashrc file
    . /opt/OpenFOAM/OpenFOAM-v1706/etc/bashrc

    # how many processers to use to install OpenFOAM and hy2foam
    export NPROCS=16

    # Install OpenFOAM
    cd $WM_PROJECT_DIR
    ./Allwmake $NPROCS

    # Create WM_PROJECT_USER_DIR, cd to it, and then clone hy2foam git repository
    mkdir $WM_PROJECT_USER_DIR
    cd $WM_PROJECT_USER_DIR
    git clone --branch master --single-branch https://github.com/hystrath/hyStrath.git
    cd hyStrath/

    # Install selected modules (replace NUMPROCS with the desired number of processors)
    # printf is used to automatically provides the input "4" to the script install.sh using a pipe (|)
    # to specify which module to install
    printf "4\n" | bash install.sh $NPROCS

%environment
    . /opt/OpenFOAM/OpenFOAM-v1706/etc/bashrc

%runscript
    echo "Fleming environment is set up."
    exec "$@"

Build Container

After creating the definition file, you need to build it and convert it to a SIF image. For more info on building container images please refer to Building Containers.

To build the container, run the following command:

apptainer build hy2foam.sif hy2foam.def

If this completed successfully, you should find the sif image hy2foam.sif where the definition file is located.

Install OpenMPI on Discovery

To be able to use the sif image on Descovey, you need to have a compatible OpenMPI version on the host. For example, if the version of OpenMPI in the image is 4.0.3, there should be the same version installed on discovery. In this section, you will be able to find the OpenMPI version installed in the container and then install that version on Descovery. To check the OpenMPI version in the container, run the following command:

$ apptainer exec hr2foam.sif mpicc --showme:version
/usr/bin/mpicc: Open MPI 4.0.3 (Language: C)

The output shows that the version is 4.0.3.You can use Spack to install this version on Discovery through Spack which was installed using SStack. First, load SStack module, then install Spack where SStack will create a module file that you need in environment to install OpenMPI:

module load sstack
sstack install -n spack_mpi -t spack
module load spack/spack_mpi

To install the version 4.0.3 , run the following command:

spack install openmpi@4.0.3 +pmi fabrics=ucx schedulers=slurm +legacylaunchers