Continuous Integration, and Deployment (CI/CD)

This section is to brief contributors on an overview of how CI/CD process is integrated to the development process of the HPC Docs site.

GitLab CI/CD

HPC team uses GitLab as the Git repository management system thus to trigger the build and deployment process uses GitLab CI/CD. For the existing GitLab CI/CD configuration check → GitLab CI/CD configuration.

Jacamar CI

GitLab’s custom executor schedules a Slurm job on discovery using Jacamar CI. More on the Slurm job → Slurm job parameters.

You must connect to the HPC team VPN for the above URLs to work.

Jobs

CI/CD process uses different stages, and tasks-names, to refer different artifacts of the process.

  • Stage build is for testing the build process, running automated tests of different kinds to check if everything can be built without any errors.

  • Stage test is used to run Vale which brings code-like linting. Vale will run against the .adoc files that has been changed/created in the merge request and it excludes copied, deleted, and renamed .adoc files

  • Stage deploy will deploy the changes to two environments.

    • Stage deploy_dev will deploy the changes from your feature branch to the master branch/development environment → HPC Development Environment after the changes were merged.

    • Stage deploy_prod will deploy the changes from the master branch/development environment to the production environment/live website → HPC Live Website

Each job schedules and runs the job as a Slurm job on the Discovery cluster.

Merging Process

When a merge request is submitted to merge the changes from the feature branch to the remote master branch, the following processes take place one-by-one:

Stage Build

  • A job named Build is scheduled to run as a part of the CI process for the merge request. It checks whether the project is built successful or not. If everything runs as expected, without any build errors, the job passes the Build stage job.

  • The below image depicts the pipeline which begins running when the merge request gets created.

Stage Build

  • If it’s successful, you will see the progress symbol getting changed to green tick mark like below.

Stage Build Success

Stage Test

  • Once the build is successful without any errors, the next Test job gets launched which runs Vale against the changed/newly created .adoc files which are part of the merge request.

  • It checks for any errors/warnings/suggestions for the files in the merge request. If there are any errors, the job gets failed and you need to fix the errors, commit your changes again to the same remote feature branch, and your pipeline gets triggered automatically starting from the build stage again.

  • However, if there are only warnings/suggestions, the job gets passed and now, you can merge your changes to the master branch.

Only the person with the maintainer access for the project repository on GitLab can merge your request. If you have any questions, contact the HPC team at hpc-team@nmsu.edu

Stage Deploy(deploy_dev)

  • Now, merge the changes to the master branch. A new pipeline gets launched and runs the Stage Build again to check for any build errors on the master branch.

  • The below image depicts the pipeline which gets triggered automatically while merging the changes to the master branch.

Pipeline Build Master

  • After the build job is successful which can be inferred with a green tick symbol, a new job called deploy_dev will be launched automatically which will deploy the changes to the master branch.

  • If the changes are deployed to the master branch successfully, you will be provided with a confirmation like below,

Pipeline Deploy

  • Also, you can use the View app button which takes you to the → HPC Development Environment. This provides a place to test all your merged changes before merging it the production environment.

  • However, you will also notice that there is a message stating that Pipeline waiting for manual action on master. This is because, the pipeline is set up in such a way that it needs to be merged manually to the production environment.

It’s highly recommended to deploy the changes to the production environment only after a thorough testing in the development environment space.

Stage Deploy(deploy_prod)

  • To deploy the changes to the production environment manually, launch the deploy_prod job by clicking the Deployments→Environments from the sidebar which is shown below.

production environment

  • There will be two environments, development and production. To deploy the changes of the master branch/development environment to production environment, click on the play icon and it will show the deploy_prod option from the development environment like below. Select the deploy_prod option.

Deploy to production

  • It will launch the deploy_prod job and will deploy the changes to the production environment if the job is successful.

References

  • How GitLab CI/CD works, and how to integrate it → GitLab CI/CD.

  • What’s Jacamar CI and how to use it → Jacamar CI.