Why ?

For scientific reproducibility !

Image your project, stored in the ESCAPE repository and compiled into a container, being able to provide a ‘one-click’ full re-analysis at any moment, even with a new set of data. Accessible to anybody, as well as to any other ESCAPE service and at any time.

How ?

Best practices

You can check the different sessions and presentations that took place in the Workshop on Open-Source Software Lifecycles.


Building an image within a CI/CD pipeline

To make use of the CI pipeline to build a container image, you will always need to provide the recipe of the image to be built and include it in your project.

GitLab offers a series of default runners (the distant machines that runs the CI pipeline) with different softwares and technologies installed on them. To make use of a runner with the Docker engine available, you would just need to choose it and configure some parameters in the .gitlab-ci.yml file. The commands to build an image on distant runners are indeed the same that you would use to build them on your local machine.

Docker

Find below an example snippet (there are plenty of ways to do this) of how to build a Docker image.

Let’s assume we have a project structure as follows

MyProject/
  ├── myproject/
  |     └── ... (all the structure of my project)
  ├── Docker/
  |     └── Dockerfile (the Docker recipe)
  ├── README.md
  ├── LICENSE
  ├── codemeta.json
  └── .gitlab-ci.yml

The .gitlab-ci.yml file should include a stage like

build_docker:
  stage: build_docker
  image: docker:19.03.12        # Configure the image to be used within the runner
  services:
    - docker:19.03.12-dind
  before_script:
    - cat /etc/os-release       # "Alpine Linux v3.12"
    - export LAST_RELEASE=v1.9  # For example
    - echo $LAST_RELEASE
  script:
    # Move to the directory containing the Docker recipe
    - cd Docker
    # Login to the gitlab container registry
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    # Build the image and tagged it
    - docker build -t $CI_REGISTRY_IMAGE:$LAST_RELEASE .
    # Push the container to the container registry
    - docker push $CI_REGISTRY_IMAGE:$LAST_RELEASE
  only:
    - tags

Important note. $CI_REGISTRY_USER $CI_REGISTRY_PASSWORD and $CI_REGISTRY variables are gitlab environment variables that should not be modified nor edited - If working from a project in a gitlab instance with the Container Registry service available, gitlab would manage internally these variables (gitlab documentation).

Singularity

Again, let’s assume we have the following project structure

MyProject/
  ├── myproject/
  |     └── ... (all the structure of my project)
  ├── Singularity/
  |     └── Singularity (the Singularity recipe)
  ├── README.md
  ├── LICENSE
  ├── codemeta.json
  └── .gitlab-ci.yml

so that the .gitlab-ci.yml file should include a stage like

build_singularity:
  stage: build
  # Configure the image to be used within the runner
  image: singularityware/singularity:gitlab-2.6
  script:
    # Build the image!
    - singularity build your_image_name.simg Singularity/Singularity 
  only:
    - tags

Important note

  • This snippet example uses a container with Singularity v2.6.
  • This example is based on the following ‘Singularity Builder GitLab project. You can find a use case of how to include the project into another open source project here.
    Please be sure that your project is compatible with open source BSD 3-Clause licenses so that you can include the submodule in your project!

More examples

The following example is part of the eOSSR package .gitlab-ci.yml file in where:

  • Both a Singularity and a Docker images of the project are built during the CI pipeline,
  • The pipeline is configured so that the images are available in different stages of the CI pipeline,
  • The images are uploaded, together with the last release of the project, to the ESCAPE OSSR.

Important note

Check the latest version of this code snippet directly from the eOSSR documentation page.