GETTING-STARTED.md

Use the ci-tools to setup a new CI

• Use the ci-tools to setup a new CI
  • Quick reminder of what is a CI
    • Rules
    • Stages
    • Include
    • Extends
    • Jobs
  • Setup your Gitlab project
    • Create the release cycle branches
    • Protect your release cycle branches and tags
    • Create the GITLAB_TOKEN access token
    • Enable CI for your project
    • Enable some runners
  • Create your .gitlab-ci.yml
    • A full setup using all ci-tools templates
    • Step by step setup
      • Target when to run jobs
      • Validate commit messages
      • Generate release with semantic version scheme
      • Build and tag docker images
        • Building images
        • Tagging docker images
      • Avoid regression after stable release

Quick reminder of what is a CI

The continuous integration “is the practice of merging all developers' working copies to a shared mainline several times a day.”

This include the automatic build and test of the code for every contribution to make sure everything can be integrated correctly.

In Gitlab, the CI configuration is done in a file called .gitlab-ci.yml in the root directory of the sources of a project.

Every branches with that file could run some pipeline if configured.

We will see 5 main .gitlab-ci.yml concepts in this documentation:

• the rules permit to include or exclude jobs in a pipeline, an empty pipeline does not run.
• the stages to group jobs together and define the order of execution of the groups
• include to reuse configuration files across projects
• the extends to reuse configuration sections
• the jobs them-self to execute action in the pipeline.

Rules

Instead of defining again and again the rules to apply to jobs, we define a common set of rules to be used directly by jobs.

For example, to make a job run only on the dev branch:

include:
  # We include the definitions before using them
  - project: EOLE/Infra/ci-tools
    ref: stable
    file: /templates/Rules.yaml

run-only-on-dev:
  extends: .on-dev
  script:
    - echo "I'm running only on $DEV_BRANCH"

The rules template define variables for the default branch names:

• STABLE_BRANCH: default to stable
• TESTING_BRANCH: default to testing
• ALPHA_BRANCH: default to alpha
• DEV_BRANCH: default to dev

Stages

The stages can be whatever you want, the ci-tools templates use 3 stages by default:

• lint for .git:commitlint
• build for .docker:image:build
• release for .semantic-release:stable and .docker:image:tag

The order of stages list is important, lint should come first, then build and finally release to publish the build results.

Include

Instead of defining the same jobs in every projects, the ci-tools provides templates to be included in .gitlab-ci.yml of other projects.

include:
  # We include the definitions before using them
  - project: EOLE/Infra/ci-tools
    ref: stable
    file: /templates/Rules.yaml
  - project: EOLE/Infra/ci-tools
    ref: stable
    file: /templates/Git.yaml

stages:
  - lint

commitlint: {extends: '.git:commitlint'}

Extends

The extends keyword permits to merge different YAML jobs together, the most common case uses hidden template jobs (with names starting by a dot .).

.alpine-common:
  image: "$ALPINE_IMAGE"
  variables:
    ALPINE_IMAGE: "alpine:latest"
    FOO: "this is FOO in .alpine-common"
  before_script:
    - echo "I run before any other 'script'"

this-job-run-on-alpine:
  extends: .alpine-common
  script:
    - echo "I'm a job running on $ALPINE_IMAGE"

Jobs

Not so much to say about jobs, it's the base executing bloc of the CI.

Setup your Gitlab project

Before using the CI tools templates, you must setup your repository.

Create the release cycle branches

You must create the branches¹ required to your release cycle:

1. create the dev branch and set it as default
2. create the alpha branch if you want a staging area where to stabilise your project before release
3. create the stable branch where the releases are kept if you don't want to keep the default name (main, stable or master depending of the configuration of your Gitlab)

Protect your release cycle branches and tags

When the branches are created, you must protect theses branches to permit the access to protected variables.

The Maintainers role must be allowed to push for semantic-release to work.

Depending on your policy, you can restrict the merge to Maintainers or Developpers + Maintainers.

Finally, you must protect the release tag pattern release/ with only push allowed for Maintainers used by semantic-release. That's the reason why the release tag rules run only on protected tags.

Create the GITLAB_TOKEN access token

The semantic-release job require a GITLAB_TOKEN project variable to be able to push the generated commits and tags.

1. Create a project access token called semantic-release with the following scopes:
   • api
   • read_repository
   • write_repository
2. Create a project variable named GITLAB_TOKEN with value of the access token created previously. The value must have the following attribtutes selected
   • protected to be exposed only on protected branches and tags
   • masked to be hidden in logs

Enable CI for your project

Make sure you enabled the CI/CD for your project.

Enable some runners

Setup some personal runners or use any shared runners available in your Gitlab.

Create your .gitlab-ci.yml

The CI/CD configuration is driven by the .gitlab-ci.yml YAML file in the root of your git repository.

A full setup using all ci-tools templates

To setup a complete CI with:

• commitlint to validate your commit message formatting
• semantic-release to generate semantic version number when pushing to $STABLE_BRANCH
• build and tags of a docker image

you need mostly 4 steps:

1. create the .gitlab-ci.yml (click to view)

   # -*- coding: utf-8 -*-
   # vim: ft=yaml
   ---
   include:
     - project: EOLE/Infra/ci-tools
       ref: stable
       file: /templates/Rules.yaml
     - project: EOLE/Infra/ci-tools
       ref: stable
       file: /templates/Git.yaml
     - project: EOLE/Infra/ci-tools
       ref: stable
       file: /templates/Semantic-release.yaml
     - project: EOLE/Infra/ci-tools
       ref: stable
       file: /templates/Docker.yaml
   
   
   stages:
     - initial-checks
     - lint
     - build
     - test
     - release
   
   variables:
     # Globally defined docker image name
     IMAGE_NAME: useless
   
   
   ###############################################################################
   # `initial-checks` stage: `has-dev-branch`, `has-testing-branch`, `has-stable-branch`
   ###############################################################################
   # Make sure we have the `${TARGET_BRANCH}`
   .has-branch:
     stage: initial-checks
     extends: .not-on-stable
     variables:
       TARGET_BRANCH: $DEV_BRANCH
     # We use whatever image that has git
     image: 'bitnami/git:latest'
     script:
       - echo -e "\e[0Ksection_start:`date +%s`:has_branch[collapsed=true]\r\e[0KCheck that '${TARGET_BRANCH}' branch exists upstream"
       - 'git fetch --all'
       - 'git show-ref -q --verify refs/remotes/origin/${TARGET_BRANCH}'
       - echo -e "\e[0Ksection_end:`date +%s`:has_branch\r\e[0K"
   
   has-dev-branch:
     extends: .has-branch
   
   has-testing-branch:
     extends: .has-branch
     variables:
       TARGET_BRANCH: $TESTING_BRANCH
   
   has-stable-branch:
     extends: .has-branch
     variables:
       TARGET_BRANCH: $STABLE_BRANCH
   
   
   ###############################################################################
   # `lint` stage: `commitlint`
   ###############################################################################
   commitlint: {extends: '.git:commitlint'}
   
   
   ###############################################################################
   # `build` stage: `build-docker`
   ###############################################################################
   # The name of the built image is define globally by `$IMAGE_NAME`
   # The build is done:
   # - for contribution branches
   # - for `$DEV_BRANCH`
   # - on release tags (stable and testing) after the application
   #   versions are updated by `semantic-release`
   .build-docker-rules:
     rules:
       # The ordering is CRITICAL
       - !reference [.rules-map, not-on-schedule]
       - !reference [.rules-map, not-on-draft]
       - !reference [.rules-map, on-release-tag]
       - !reference [.rules-map, on-testing-tag]
       - !reference [.rules-map, not-on-stable]
       - !reference [.rules-map, not-on-testing]
       - !reference [.rules-map, not-on-semantic-release-commit]
       - !reference [.rules-map, on-branches]
   
   build-docker:
     extends:
       - .docker:image:build
       - .build-docker-rules
   
   
   ###############################################################################
   # `test` stage: `useless-test`
   ###############################################################################
   useless-test:
     stage: test
     # Use the previously built image, so reuse the same `rules`
     extends: .build-docker-rules
     image: "${CI_REGISTRY_IMAGE}/${IMAGE_NAME}:git-${CI_COMMIT_SHORT_SHA}"
     script:
       - echo "I successfully ran in ${IMAGE_NAME}:git-${CI_COMMIT_SHORT_SHA}"
   
   
   ###############################################################################
   # `release` stage: `semantic-release`, `testing-prerelease`,
   #                  `merge-to-dev`, `tag *`
   ###############################################################################
   # Create the release versions on `$STABLEE_BRANCH`
   new-release: {extends: '.semantic-release:stable'}
   
   # Create the prereleases versions on `$TESTING_BRANCH`
   # update `.releaserc.js` variable `betaBranch`
   testing-prerelease: {extends: '.semantic-release:testing'}
   
   # Avoid regression by merging all pre-release fixes to `$DEV_BRANCH`
   merge-to-dev: {extends: '.git:merge-to', variables: {GIT_MERGE_TARGET: $DEV_BRANCH}}
   
   ## tag contribution branches with a more stable name than `git-${CI_COMMIT_SHORT_SHA}`
   tag contrib branch:
     extends:
       - .docker:image:tag
       - .on-branches
     variables:
       # `feature/foo-bar_quux` → `feature-foo-bar-quux`
       IMAGE_TAG: $CI_COMMIT_REF_SLUG
   
   ## dev images
   tag dev:
     extends:
       - .docker:image:tag
       - .on-dev
     variables:
       IMAGE_TAG: dev
   
   ## testing images
   tag testing:
     extends:
       - .docker:image:tag
       # After `semantic-release`
       - .on-testing-tag
     variables:
       IMAGE_TAG: testing
   
   ## stable images
   # add the `X.Y.Z` tag
   tag release:
     extends: .docker:image:tag
   
   # add the `X` tag
   tag major:
     extends: .docker:image:tag
     before_script:
       - export RELEASE_PREFIX=${RELEASE_PREFIX:-release/}
       - export RELEASE=${CI_COMMIT_TAG#${RELEASE_PREFIX}}
       - export IMAGE_TAG=${RELEASE%%.*}
   
   # add the `X.Y` tag
   tag minor:
     extends: .docker:image:tag
     before_script:
       - export RELEASE_PREFIX=${RELEASE_PREFIX:-release/}
       - export RELEASE=${CI_COMMIT_TAG#${RELEASE_PREFIX}}
       - export IMAGE_TAG=${RELEASE%.${RELEASE##*.}}
   
   tag stable:
     extends: .docker:image:tag
     variables:
       IMAGE_TAG: stable
   
   tag latest:
     extends: .docker:image:tag
     variables:
       IMAGE_TAG: latest
   ...

2. configure commitlint with the .commitlintrc.yaml

3. configure semantic-release in .releaserc.js (note that the branches variable must match your $STABLE_BRANCH)

4. add a Dockerfile in the root directory of your sources

Step by step setup

We will see the steps required to setup commitlint, semantic-release and the build and tags of docker images

Target when to run jobs

The first interesting YAML file is templates/Rules.yaml which does nothing by itself except providing:

• the default branch names variables
  • $STABLE_BRANCH defaults to stable
  • $TESTING_BRANCH defaults to testing
  • $ALPHA_BRANCH defaults to alpha
  • $DEV_BRANCH defaults to dev
• several hidden template jobs to use in your own job definition with extends to select in which conditions a job should run.

To use it, just include it at the top level of your .gitlab-ci.yml:

--- .gitlab-ci.yml.orig
+++ .gitlab-ci.yml
@@ -1,4 +1,8 @@
 # -*- coding: utf-8 -*-
 # vim: ft=yaml
 ---
+include:
+  - project: EOLE/Infra/ci-tools
+    ref: stable
+    file: /templates/Rules.yaml
 ...

If the rules definition is too limited for your use case, you can combine the raw conditions to extend them like in templates/Semantic-release.yaml

You can read templates/Rules.yaml for the complete list of usable rules templates.

Validate commit messages

As described in the contributing documentation, the commit message formatting is important to generate release automatically with semantic-release.

To ensure that all the commits are correctly formatted, you just need to:

1. include templates/Rules.yaml file to define the .not-on-stable rules templates
2. include the templates/Git.yaml file
3. be sure to have the lint stage to your current stages in your .gitlab-ci.yml
4. define the commitlint job extending .git:commitlint
5. configure commitlint by creating the .commitlintrc.yaml in the root of your repository

--- .gitlab-ci.yml.orig
+++ .gitlab-ci.yml
@@ -5,4 +5,17 @@
   - project: EOLE/Infra/ci-tools
     ref: stable
     file: /templates/Rules.yaml
+  - project: EOLE/Infra/ci-tools
+    ref: stable
+    file: /templates/Git.yaml
+
+
+stages:
+  - lint
+
+
+###############################################################################
+# `lint` stage: `commitlint`
+###############################################################################
+commitlint: {extends: '.git:commitlint'}
 ...

Generate release with semantic version scheme

Before enabling the automatic release creation, you should enable commitlint.

To setup semantic-release, you need to:

1. include templates/Rules.yaml file to define the .on-stable and .on-testing rules template
2. include the templates/Semantic-release.yaml file
3. be sure to have the release stage to your current stages in your .gitlab-ci.yml
4. define the new-release job extending .semantic-release:stable
5. define the testing-prerelease job extending .semantic-release:testing
6. configure semantic-release in .releaserc.js

--- .gitlab-ci.yml.orig
+++ .gitlab-ci.yml
@@ -8,14 +8,29 @@
   - project: EOLE/Infra/ci-tools
     ref: stable
     file: /templates/Git.yaml
+  - project: EOLE/Infra/ci-tools
+    ref: stable
+    file: /templates/Semantic-release.yaml


 stages:
   - lint
+  - release


 ###############################################################################
 # `lint` stage: `commitlint`
 ###############################################################################
 commitlint: {extends: '.git:commitlint'}
+
+
+###############################################################################
+# `release` stage: `new-release`, `testing-prerelease`
+###############################################################################
+# Create the release versions on `$STABLEE_BRANCH`
+new-release: {extends: '.semantic-release:stable'}
+
+# Create the prereleases versions on `$TESTING_BRANCH`
+# update `.releaserc.js` variable `betaBranch`
+testing-prerelease: {extends: '.semantic-release:testing'}
 ...

Build and tag docker images

The templates/Docker.yaml defines 2 job templates to build and tag docker images.

Building images

The templates/Docker.yaml defines .docker:image:build job template which, by default, build the container image and push it to ${CI_REGISTRY} for all branches except $STABLE_BRANCH.

It uses kaniko which does not require to enable docker-in-docker privileged mode.

The simplest use of this template require 4 elements:

1. include templates/Rules.yaml file to define the .not-on-stable and .on-release-tag rules template
2. include the templates/Docker.yaml template
3. be sure to have the build stage to your current stages in your .gitlab-ci.yml
4. extends the .docker:image:build template to define the build job

--- .gitlab-ci.yml.orig
+++ .gitlab-ci.yml
@@ -11,12 +11,20 @@
   - project: EOLE/Infra/ci-tools
     ref: stable
     file: /templates/Semantic-release.yaml
+  - project: EOLE/Infra/ci-tools
+    ref: stable
+    file: /templates/Docker.yaml


 stages:
   - lint
+  - build
   - release

+variables:
+  # Globally defined docker image name
+  IMAGE_NAME: useless
+

 ###############################################################################
 # `lint` stage: `commitlint`
@@ -24,6 +32,33 @@
 commitlint: {extends: '.git:commitlint'}


+###############################################################################
+# `build` stage: `build-docker`
+###############################################################################
+# The name of the built image is define globally by `$IMAGE_NAME`
+# The build is done:
+# - for contribution branches
+# - for `$DEV_BRANCH`
+# - on release tags (stable and testing) after the application
+#   versions are updated by `semantic-release`
+.build-docker-rules:
+  rules:
+    # The ordering is CRITICAL
+    - !reference [.rules-map, not-on-schedule]
+    - !reference [.rules-map, not-on-draft]
+    - !reference [.rules-map, on-release-tag]
+    - !reference [.rules-map, on-testing-tag]
+    - !reference [.rules-map, not-on-stable]
+    - !reference [.rules-map, not-on-testing]
+    - !reference [.rules-map, not-on-semantic-release-commit]
+    - !reference [.rules-map, on-branch]
+
+build-docker:
+  extends:
+    - .docker:image:build
+    - .build-docker-rules
+
+
 ###############################################################################
 # `release` stage: `new-release`, `testing-prerelease`
 ###############################################################################

Tagging docker images

By default, the .docker:image:tag job template works on the release tag only to create the docker tag X.Y.Z.

In a typical release cycle, you want to create the following tags:

• dev images where developpement is integrated
• testing images where releases are stabilised
• stable images when the release is done
  • major tag with only the first digit of the semantic version, this tag will always point to the latest release of the major version
  • minor tag with only the first 2 digits of the semantic version, this tag will always point to the latest release of the minor version
  • release tag with the full semantic version
  • latest/stable point to the latest stable image

To be used, you need to:

1. include templates/Rules.yaml file to define the rules template
2. include the templates/Docker.yaml template
3. be sure to have the release stage to your current stages in your .gitlab-ci.yml
4. extends the .docker:image:tag template to create as many tagging jobs as required for your release cycle

--- .gitlab-ci.yml.orig
+++ .gitlab-ci.yml
@@ -60,7 +60,7 @@


 ###############################################################################
-# `release` stage: `semantic-release`, `testing-prerelease`
+# `release` stage: `semantic-release`, `testing-prerelease`, `tag *`
 ###############################################################################
 # Create the release versions on `$STABLEE_BRANCH`
 new-release: {extends: '.semantic-release:stable'}
@@ -68,4 +68,61 @@
 # Create the prereleases versions on `$TESTING_BRANCH`
 # update `.releaserc.js` variable `betaBranch`
 testing-prerelease: {extends: '.semantic-release:testing}
+
+## tag contribution branches with a more stable name than `git-${CI_COMMIT_SHORT_SHA}`
+tag contrib branch:
+  extends:
+    - .docker:image:tag
+    - .on-branches
+  variables:
+    # `feature/foo-bar_quux` → `feature-foo-bar-quux`
+    IMAGE_TAG: $CI_COMMIT_REF_SLUG
+
+## dev images
+tag dev:
+  extends:
+    - .docker:image:tag
+    - .on-dev
+  variables:
+    IMAGE_TAG: dev
+
+## testing images
+tag testing:
+  extends:
+    - .docker:image:tag
+    # After `semantic-release`
+    - .on-testing-tag
+  variables:
+    IMAGE_TAG: testing
+
+## stable images
+# add the `X.Y.Z` tag
+tag release:
+  extends: .docker:image:tag
+
+# add the `X` tag
+tag major:
+  extends: .docker:image:tag
+  before_script:
+    - export RELEASE_PREFIX=${RELEASE_PREFIX:-release/}
+    - export RELEASE=${CI_COMMIT_TAG#${RELEASE_PREFIX}}
+    - export IMAGE_TAG=${RELEASE%%.*}
+
+# add the `X.Y` tag
+tag minor:
+  extends: .docker:image:tag
+  before_script:
+    - export RELEASE_PREFIX=${RELEASE_PREFIX:-release/}
+    - export RELEASE=${CI_COMMIT_TAG#${RELEASE_PREFIX}}
+    - export IMAGE_TAG=${RELEASE%.${RELEASE##*.}}
+
+tag stable:
+  extends: .docker:image:tag
+  variables:
+    IMAGE_TAG: stable
+
+tag latest:
+  extends: .docker:image:tag
+  variables:
+    IMAGE_TAG: latest
 ...

Avoid regression after stable release

When the next release is prepared in the $TESTING_BRANCH, any fixes should be applied to the $DEV_BRANCH to avoid regressions.

You can either manually merge the fixes to $DEV_BRANCH or use the .git:merge-to template job to automatically merge the release tag to the $DEV_BRANCH.

To do so, you need to:

1. include templates/Rules.yaml file to define the .not-on-stable rules template
2. include the templates/Git.yaml file
3. be sure to have the release stage to your current stages in your .gitlab-ci.yml
4. define the merge-to-dev job extending .git:merge-to

--- .gitlab-ci.yaml.orig
+++ .gitlab-ci.yaml
@@ -60,7 +60,8 @@


 ###############################################################################
-# `release` stage: `semantic-release`, `testing-prerelease`, `tag *`
+# `release` stage: `semantic-release`, `testing-prerelease`,
+#                  `merge-to-dev`, `tag *`
 ###############################################################################
 # Create the release versions on `$STABLEE_BRANCH`
 new-release: {extends: '.semantic-release:stable'}
@@ -69,6 +70,9 @@
 # update `.releaserc.js` variable `betaBranch`
 testing-prerelease: {extends: '.semantic-release:testing}

+# Avoid regression by merging all pre-release fixes to `$DEV_BRANCH`
+merge-to-dev: {extends: '.git:merge-to', variables: {GIT_MERGE_TARGET: $DEV_BRANCH}}
+
 ## tag contribution branches with a more stable name than `git-${CI_COMMIT_SHORT_SHA}`
 tag contrib branch:
   extends:

1. The branch names can be configured by setting the DEV_BRANCH, ALPHA_BRANCH, TESTING_BRANCH and STABLE_BRANCH variables in your own .gitlab-ci.yml ↩