Skip to content

Contribute to R2Devops official CI/CD Catalog

This page describes how to create or update a job in the R2Devops/hub repository. It's the guide!

Info

In order to contribute efficiently, we recommend you to know the following topics:

Tip

If you are hesitant about which job to create/update, you can get inspired by our pre-made job issues on GitLab under the label Easy first step.

Contributing workflow

Follow the 3 simple steps below to contribute efficiently in the hub. You'll see, it will all go smoothly! ๐Ÿ‘‡

๐Ÿด Step 1: Fork!

The first step is to create your own copy of r2devops/hub repository, to be able to work on it before merging your update in the real project.

  1. Go on the fork page creation: r2devops/hub.
  2. Select the group in which you want to create the fork.

๐Ÿ’ป Step 2: Work in your fork

Do not alter .gitlab-ci.yml

To leverage the R2Devops validity and security checks on your job, do not update the CI/CD configuration file in your fork (.gitlab-ci.yml file).

If you alter it, we will not be able to merge your job in r2devops/hub repository (yes, what a shame, after all your hard work...). ๐Ÿ˜•

  1. If you want to create a new job:

    1. Create a new branch for your job and work on it ๐ŸŽ‹
    2. Make sure that you have NPM installed, more about that here
    3. Install the package Cookicutter by executing this command:
      npm install -g cookiecutter
      
    4. In the hub folder, run this command and input the necessary information:
      npx cookiecutter
      

    Info

    Alternatively, you can do the last steps manually by creating a new directory dedicated to your job in jobs/ folder if you want to add a new job. You can use the job template as a starting point. If you want to modify an existing job, you don't have to create a new directory.

  2. Be sure to respect the rules we describe in this guide.

  3. Do not update the CI/CD configuration file .gitlab-ci.yml.

๐Ÿงช Step 3: Test your job

Before merging it into the hub, you should test it deeply to be sure it works as expected.

  1. Find a suitable project for your job You can pick one of yours or use a GitLab template project. To simulate the GitLab job that will run on your project, there are several way to do it :

    1. With docker You can mount the repository folder inside the container:
      # Example if your job will use node:16.3-buster as Docker image
      docker run -v /path/to/your/repo:/mnt --entrypoint "/bin/sh" -it node:16.3-buster
      
    2. With GitLab Runner If you have GitLab Runner installed, you can test your job on your project. Copy the content of the job.yml file into your .gitlab-ci.yml and include the job's stage. Then test it via:
      gitlab-runner exec docker <job_name>
      
    3. By remoting your job Move your job configuration in your project root directory and include it locally in your .gitlab-ci.yml:
      include:
          - local: 'my-work-in-progress-job.yml'
      
      OR You can remote your job from your fork of the hub. On gitlab.com, find your job.yml in your repository and click on it. Then open the raw content by clicking on Open raw button and copy the url. Now go back on your project and remote it in your .gitlab-ci.yml:
      include:
        #change it by the url of your job
        - remote : 'https://gitlab.com/<project>/-/raw/<hash>/jobs/<job_name>/<job_name>.yml'
      
      Add, commit and push your modifications and check the result of the pipeline on Git Lab, under CI/CD > Pipelines. After the merge request approval, you can remote it directly from the hub, check the quick setup section.
    4. Test the behaviour of your job ๐ŸŽฒ
    5. If you're creating a job from one of the hub's issue, there should be at least one scenario. Test each scenario separately and ensure the pipeline succeed.
    6. If you're creating a job on your own, think of each case, your job should be the most generic as possible.

๐Ÿ‘‰ You can customize it with variables

๐Ÿš€ Step 4: Merge request

1. Ensure that the last pipeline in your fork passed before going further (check it in CI/CD > Pipelines)

2. Create a new merge request in your fork (Merge Requests > New merge request) ๐Ÿ‘‡
  1. Select branches
    • As Source branch, select the branch in which you have worked in your fork (usually latest). We encourage you to create a new branch for each job!
    • As Target branch, select latest in r2devops/hub project.
    • Click on Compare branches and continue.
  2. In Title: add short description of your contribution.
  3. In Description:
    • Do not remove the default content, this is the Definition of Done (DoD)!
    • Add a description of your contribution with all information permitting us to understand what you have done and why. If your contribution is related to an existing issue, add a reference:
      ## Contribution
      Addition of a new job permitting to build golang binaries. Issue
      related: r2devops/hub#945
      
      ## Definition of Done
      [...]
      
    • Add a link to your job running and working in a publicly accessible Gitlab project.
  4. Write a comment :
    • Ping a member of the team for asking a review.
    • Share the link of the generated documentation. It could be found by browsing artifacts of the mkdocs jobs. The path is usually : /website_build/jobs/<stage>/<job_name>/index.html.
    • For each scenarios written in the issue description : Test each case with corresponding variables fulfil and ensure the pipeline succeed. Share the link in the comment and specify the scenario associated.
  5. If you want to allow commits from hub maintainers inside your fork branch, check the box Contribution (this isn't available for protected branches like latest).

3. In the newly created MR, ensure to fulfill all the steps of the job's Definition of Done, and tick the related boxes

Thanks a lot for your contribution ๐Ÿ˜€๐ŸŽ‰ !

Now, we will take a look at your work and merge it if everything is ok. ๐Ÿ‘€ Meanwhile, you can join our Discord community to tell us more about your fresh new contribution. We love talking with our contributors and users!

Guidelines (Required)

Warning

Following these guidelines is required to contribute to R2Devops/hub repository.

Each jobs must be compliant with these following rules:

  • Be compliant with the job structure.
  • Use the image option. The goal is to provide plug and play jobs working in any environments thanks to containers.
  • Use fixed tag for docker image and any external tool used inside the job. It shouldn't be latest or any tag/version that will be overwritten.
  • Use only resource with license compatible with the job license, and permitting anyone to use it.
  • Pass our Continuous Integration pipeline, which includes security check jobs (the pipeline will be run automatically inside your fork ๐ŸŽข).
  • Be compliant with our job definition.

Best practices (Optional)

Info

Following these best practices is recommended to contribute to R2Devops/hub repository.

๐Ÿค– Job definition

๐Ÿงฎ Variables

In order to be customizable, your job should use variables. This section allows to define environment variables, usable by the job script.

Tip

Set default values for your variables, to reflect the most common use-case. With this, your job will remain plug-and-play while being customizable!

Example of relevant situation to use variable:

  • File name.
  • Path to files or folders.
  • Any options/parameters used by the job.
  • Enable or disable job features.
  • Version of tools retrieved during the job.

Here is an example for newman job ๐Ÿ‘‡

newman:
    image: node:15.4.0
    stage: tests
    variables:
        NEWMAN_COLLECTION: "postman_collection.json"
        NEWMAN_GLOBALS_FILE: ""
        NEWMAN_ADDITIONAL_OPTIONS: ""
        NEWMAN_JUNIT_REPORT: "newman-report.xml"
        NEWMAN_FAIL_ON_ERROR: "true"
        NEWMAN_ITERATIONS_NUMBER: "2"
    script:
        - npm install -g newman newman-reporter-junitfull
        - if [[ ! -z ${NEWMAN_GLOBALS_FILE} ]]; then
        -   NEWMAN_ADDITIONAL_OPTIONS+=" -g ${NEWMAN_GLOBALS_FILE}"
        - fi
        - if [[ ! ${NEWMAN_FAIL_ON_ERROR} == "true" ]]; then
        -   NEWMAN_ADDITIONAL_OPTIONS+=" --suppress-exit-code"
        - fi
        - newman run ${NEWMAN_COLLECTION} -r cli,junitfull
          --reporter-junitfull-export ${NEWMAN_JUNIT_REPORT}
          -n ${NEWMAN_ITERATIONS_NUMBER} ${NEWMAN_ADDITIONAL_OPTIONS}
    artifacts:
        [...]
Name Description Default
NEWMAN_COLLECTION Name of the Postman collection postman_collection.json
NEWMAN_GLOBALS_FILE Name of the Postman globals file for variables
NEWMAN_ADDITIONAL_OPTIONS Other options you may want to use with Newman
NEWMAN_FAIL_ON_ERROR Fail job on a request/test error true
NEWMAN_ITERATIONS_NUMBER Number of Newman iterations to run (see documentation) 2

๐Ÿณ Docker image

Info

As described in our guidelines, all jobs are run inside a container instance, so they must specify the Docker image to use. Depending on your job, it can be tricky to find the perfect image.

The better place to find a docker image is the docker hub. You can start your research there with the following steps:

1. Search for an image prepared with the tool you want to run
  • This is the best situation: a ready-to-use docker image that doesn't require any additional installation.
  • Example for mkdocs job: squidfunk/mkdocs-material.
  • If you find it, check it with the general guidelines below.
2. If there isn't any image prepared for the tool you want to run, search for more general images
  • This case will require to install needed tool as first step of your job. If the installation is long or heavy, you should considers to build your own image with the tool already installed.
  • The vast majority of operating system and languages provides official images, choose the more appropriate for your job. Some examples: alpine, debian, ubuntu, python, node.

3. If you decide to build your own image: the image must be stored in a publicly reachable registry, like Docker hub or GitLab registry

General guidelines to choose the image

  • The first thing to check is if the image is official (OFFICIAL IMAGE badge on docker hub): this is the perfect image for your use case Docker official
  badge.
  • If it is not, the following points should be considered to choose an image:
    • The image must be versioned and not only with latest tag. If image isn't versioned: it's not usable for your job.
    • It should be actively maintained, with frequent updates, and should contain recent versions.
    • The image should be small, containing only required tools.
    • The image should be efficient to run the job.
    • A large usage of the image can be a good indicator, but be aware, it doesn't guarantee the quality neither the security of the image.

๐Ÿ“ฆ Artifacts

The vast majority of jobs produce a result. This result can be of different kinds:

  • Test report.
  • Build result.

In both case, the result should be exposed by the job using the artifact option. It permits passing an artifact to another job of the pipeline and expose results to users.

Info

  • In the case of a test report, you need to use when: always option under artifacts if you want to expose result even if job fails.
  • You can combine both a build result and a test report by using both artifacts:paths and artifacts:reports.

An artifact can be configured at different level of integration in Gitlab interface:

  1. Best integration: Gitlab artifacts:reports

    This is a way to integrate a report result in an user-friendly way in Gitlab's interface. We encourage all job contributors to adapt their job output to a format compatible with a Gitlab report.

    Example of artifacts:reports:junit report

    Job trivy_image that uses its output as junit report in artifacts:reports:junit section:

    trivy_image:
      [...]
      artifacts:
        reports:
          junit: "$TRIVY_OUTPUT"
    

  2. Quick integration with artifacts:expose_as

    This is a way to quickly integrate any format of report in Gitlab Merge Request interface. Technically, you don't have to shape your report output in a specific format, but we recommend to use HTML format. In this way, the report is one-click readable from any Merge Request.

    Example of artifacts:expose_as report

    Job nmap uses artifacts:expose_as to expose its HTML report:

    nmap:
      [...]
      artifacts:
        expose_as: "nmap-report"
        paths:
          - "${HTML_OUTPUT}"
        when: always
    

  3. Simple artifact without integration

    Example of artifacts

    Job that specify an artifact:

    job_name:
      [...]
      artifacts:
        paths:
          - "output"
        when: always
    

๐Ÿ”ฉ Keep your job generic

The jobs of the hub should remain as generic as possible. In order to ensure it:

  • Try to avoid using rules options, that is strongly linked to the context of the user and should be set by the user. Also, some features requiring specific workflows, as Gitlab Merge Trains, are more easily implemented by users if you don't specify rules in your job.
  • Try to avoid using before_script and after_script to let users the possibility to redefine these options while exploiting to the maximum your job.

Info

The jobs of the hub can be dynamically customized by users.

๐Ÿ“– Job documentation

As described in R2Devops/hub job structure, the documentation of a job is written inside its README.md file.

Tip

Don't hesitate to copy the documentation from another job as starting point. For example, the raw content of openapi README.md!

We recommend including at least the following sections in your documentation:

  • Objective: describe the goal of your job.
  • How to use it: a list of steps to quickly use your job.
  • Job details: describes details of the job (name, docker image, stage, etc).
  • Variables: table listing variables used by the job (name, description, default value, mandatory if needed).
  • Artifacts: describes artifact(s) defined by the job.

๐Ÿš„ Compliance with another job

Several jobs of the hub can be used together without any configuration. This is currently implemented for all jobs producing an HTML output and the job pages which deploys the HTML on a webserver.

Info

This feature is empowered by the artifacts option: jobs producing a static website output give it to the pages job through an artifact stored in a standard path: ${CI_PROJECT_DIR}/website_build.

So, if your job produce a static website output, ensure to store the result of the build in ${CI_PROJECT_DIR}/website_build, and to configure this path as artifact. You can see an example in mkdocs job.

Congratulation, you did it!

You went through all our guideline. ๐Ÿฅณ

If never something feel unclear or you're having a doubt, join us on Discord to ask us anything! We'll be more than happy to help.


Last update: October 26, 2022