Continuous deployment setup with Gitlab CI

Continuous deployment (CD) is all the rage these days. And, if you ask me, it’s a good feeling to push to a branch, wait for some time, and eventually see your changes in production. Getting this to work within the confines of Gitlab CI can be a little tricky, so I’m going to share my preferred set-up in this post.

Pipeline structure

I’ll be talking about the following scenario:

The reason I would always include an intermediate environment (e.g. “staging”), is that I would need somewhere to run my automated acceptance tests[1] against, which try to make sure that the changes made do not brake any existing functionality. So in reality, the pipeline would look a little more like this:

TLDR: If the above scenario matches your needs, and you don’t feel like reading, you can find the a runnable example pipeline in this repository.

Looking at the part of the pipeline after the unit test stage (blue) completes, it looks the same for both staging and production. First we want to deploy to the stage, then run our acceptance tests against said stage.

As far as I’m aware, for Gitlab pipeline definitions, we have three ways to handle this duplication:

  1. YAML anchors and merge key
  2. Gitlab CI extends directive
  3. Gitlab CI child pipelines

I won’t go into detail about any of them, but I’ve found (3) to be the one that I prefer. It allows me to (mostly) separate the “deployment / acceptance test” logic into its own building block, making it easy to, for example, add new stages without having to copy and paste boilerplate.

Basic familiarity with how Gitlab CI definitions work is assumed going forward.

The parent pipeline

Okay, so let’s look at some .gitlab-ci.yml (full source available here). It starts out simple enough:

  - "Analyze/Test"
  - "Deploy Staging"
  - "Deploy Production"

  # For jobs which contain `rules`, removing this causes strange behaviour
  # when merge requests are created. In that scenario, additional detached
  # pipeline runs are started.
  # See also: https://gitlab.com/gitlab-org/gitlab/issues/34756#note_282820625

  stage: "Analyze/Test"
    - echo "Running test suite ..."

Not many surprises here, except for the workflow workaround (which will hopefully be fixed in the future). In this file you would add all stages that are independent of deployment, that is, all the checks that you wouldn’t want to re-run for each stage you’re deploying to. This would likely include your unit tests, integration tests, format checks, etc.

In addition, this is where we trigger the child pipelines, which will deploy to the individual environments:

  stage: "Deploy Staging"
    - if: '$CI_COMMIT_BRANCH == "main"'
    - when: never
    include: .gitlab-ci/deploy-environment.yml
    strategy: depend

In the rules section we define that the deployment should only be run when updating the main branch. In variables, we pass whatever information is needed for the deployment, which in this example case is only the name. Finally, the trigger starts the child pipeline, in a way that the current (parent) pipeline will depend on the result.

Careful: If strategy: depend is omitted, the child pipeline would be started, but the parent would not wait for its completion. As a result, deployment to all environments would happen concurrently.

The child pipeline

Finally, the child pipeline we’re triggering in the example above will be as simple or complex as your deployment / acceptance testing logic:

# Contents of `.gitlab-ci/deploy-environment.yml`
  - deploy
  - acceptance

  stage: deploy
    - if: '$CI_COMMIT_BRANCH == "main"'
    - when: never
    - echo "Deploying $ENVIRONMENT_NAME"

  stage: acceptance
    - echo "Deploying $ENVIRONMENT_NAME"

Because we’re triggering the child pipeline with strategy: depend, the parent will not continue to run until the child has completed successfully.

You may think that this is a lot of effort for a relatively simple pipeline. And, if we’re just looking at the example I wrote, I would agree. However, CI/CD pipelines tend to grow more complex over time, and, in my experience, a 1000+ line .gitlab-ci.yml file tends to be a bit cumbersome to understand, and especially to extend.

  1. The ISTQB defines them as:

    Formal testing with respect to user needs, requirements, and business processes conducted to determine whether a system satisfies the acceptance criteria and […] to determine whether to accept the system.