Implementing Flexible, Variable-driven GitLab Pipelines

Published in

Globant

6 min readMar 6, 2024

GitLab-Ci Introduction

GitLab CI is a continuous integration and delivery tool provided by GitLab, a web-based Git repository manager. It allows developers to automate the process of building, testing, and deploying their applications. It offers many features, and new ones are added every day. But first, let’s quickly go over the most important ones.

Pipelines: GitLab CI employs pipelines to define the stages and actions needed for building, testing, and deploying an application. Pipelines consist of multiple jobs that can be executed sequentially or in parallel.
YAML configuration: GitLab CI uses a YAML file called gitlab-ci.yml to define the pipeline configuration. This file specifies the stages, jobs, and their respective scripts or commands to be executed.
Version control integration: Since GitLab-Ci is tightly integrated with GitLab, it leverages the version control capabilities of Git. Every change made to the .gitlab-ci.yml file triggers a new pipeline to run automatically.
Code Reusability: GitLab allows the reuse of pipelines and templates. These can be private or public, and they can be in the same repo or a different one. This allows you to design your process.

However, this article is not meant to discuss in detail all the features and uses of GitLab-Ci. This article assumes you have some experience with this tool and address issues related to scaling pipelines, adding features, and working with multiple repositories and technologies.

While I was developing my first set of GitLab pipelines, I kept adding features and functionality to my projects based on conversations with developers and key stakeholders. Before long, my pipelines were 700 to 1000 lines long, and I was racing to keep feature drift from one process to the other. As the number of repos keeps increasing, so does the number of separate jobs/stages you need to maintain across your codebase.

What Do We Want To Accomplish?

Before starting work on your CI/CD pipeline implementation, you should first think about what is important for you. These things may vary and are not generally applicable. I will just give you an example based on my experience.

When I was creating pipelines for a small to medium codebase, some of the things I included were the following:

Pipelines are treated like any other software release, and a team centrally manages them.
Every new release will have a version, and new features will be made available to the teams with proper documentation.
Features can be turned on or off by individual developers in their projects by leveraging variables.
The whole process is scalable and works for 5 or 150 repositories.
Pipelines can be troubleshooted by new team members with minimal onboarding time.

Planning The Process

As with any process, it is paramount to put a lot of thought into what you think is needed in the short, medium, and long term.

Allow your pipelines room to grow, plan for them, and make sure you don’t end up with spaghetti code.
Design the logic for the pipelines and stick with it. The same goes for the naming conventions.
Whatever hardware needs you have, consider using Infrastructure as Code (IaC) to deploy said infrastructure.

Main Components We Will Use In Our Pipelines

A GitLab pipeline can be configured in many ways. There are quite a few components and blocks you may choose to use. Here are some of the ones I use in this type of configuration.

Defining Stages

This is where we list all the stages we want our pipeline to have. There are two ways to accomplish this:

stages: [code_quality,build, test, deploy, auto_tests]


stages:
 - code_quality
 - build
 - test
 - deploy
 - auto_tests

Adding Rules

We might want to create a set of rules to be used by some or all stages. Something like this:

.rules: &rules
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: always
    - if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH
      when: manual
    - if: $CI_COMMIT_TAG != null
      when: never
    - if: '$CI_PIPELINE_SOURCE == "schedule"'
      when: never

Example Stage

This is a fairly standard deployment stage:

deploy_dev:
 stage: deploy_dev
 environment:
   name: dev
   url: ''
 needs:
   - push_to_ecr
 tags:
   - shell
   - dev
 rules:
   - if: '$CI_COMMIT_BRANCH =~ /develop*/i || $CI_COMMIT_BRANCH =~ $APPROVED_BRANCH ' # does not run for hotfix 
   - if: '$CI_DEPLOY_DEV != "true"' ### if CI_TESTS variable is not enabled tests will not run ###
     when: never
 script:
   - if [ "$DEBUG" == "true" ]; then set -x  && echo "Pipeline debug mode activated"; fi

Functions

To reduce code duplication, we can use functions. These can be a script like in the example above or multiple parts of a stage (Rules, variables, and the script):

 .send_message: &send_message
 script: 
 - 'curl --request POST --header "Authorization: Bearer $bearer"  --form "text=$MYDATE $msg" https://api.teams.com/v1/messages'
stages:
 - validate
validate:
 stage: validate
 script:
   - terraform validate
   - &send_message
 variables:
   msg: "This is a sample message"

Rules and Workflows

A Workflow will control how the whole pipeline behaves. It is evaluated before the individual stages where we can have Rules. There are many ways to use these. I will just provide some basic examples.

workflow:
 rules:
   - if: $CI_PIPELINE_SOURCE == "schedule"
     when: never
   - if: $CI_PIPELINE_SOURCE == "push"
     when: never
   - when: always
job:
 variables:
   DEPLOY_VERSION: "dev"
 rules:
   - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH
     variables:
       DEPLOY_VERSION: "stable"
 script:
   - echo "Deploying $DEPLOY_VERSION version"

Putting It All Together

As illustrated by the picture below, we can arrange our GitLab-ci-main.yml, GitLab-ci-tests.yml, GitLab-ci-vars.yml, GitLab-ci-build.yml, and GitLab-ci-deploy.yml in one or multiple folders in a central repository. From there, we can reference the GitLab-ci-main.yml file, which will bring over all the code we have created to our local pipeline. Of course, you can name and arrange the templates as needed, ensuring flexibility and scalability in your final process.

Pipelines and pipeline templates schematics

As an example, the Gitlab-ci-main.yml file can include something like this:

include:
 - project: 'yourproject/devops/cicd_tools'
   # ref: main
   ref: V.0.2
   file:
     - 'build_tools/gitlab-ci/gitlab-ci-vars.yaml'
     - 'build_tools/gitlab-ci/gitlab-ci-deploy.yaml'
     - 'build_tools/gitlab-ci/gitlab-ci-build.yaml'
     - 'build_tools/gitlab-ci/gitlab-ci-test.yaml'

While the child project (the one referencing the code) can include something like this:

yainclude:
 - project: 'Yourproject/devops/cicd_tools'
   # ref: main You can use either a version or a branch as a ref.
   ref: V.0.2
   file:
     - 'build_tools/gitlab-ci/python_main_gitlab_ci.yaml'

How do I make things easy for developers and end users of the pipelines? We can leverage variables for almost all aspects of the pipeline, from turning on or off a stage to setting the environment, to defining helm options. In the end, the individual pipelines from each project will look something similar to what is below:

include:
 - project: 'Yourproject/devops/cicd_tools'
   # ref: main
   ref: V.0.2
   file:
     - 'build_tools/gitlab-ci/python_main_gitlab_ci.yaml'
Variables:  ### Variables to control pipeline stages/jobs/flow
 CI_BUILD: "true"
 CI_UNI_TESTS: "true"
 CI_AUTO_TESTS: "true"
 CI_DEPLOY_DEV: "true"
 CI_DEPLOY_PROD: "true"
 APPROVED_BRANCH: "od-141-create-api-endpoints"   # Will allow you to build/deploy feature branches
 #project Specific Vars
 PROJECT_TYPE: "backend"
 HELM_TIMEOUT: "180" #time in seconds, how long will helm wait for the services to start properly
 # HELM_VERSSION: 0.2.0 #use this in case latest version breaks your service
 HELM_ARGUMENTS: --atomic --wait --version --timeout--timeout #timeout needs to be last argument
 ### Variables passed to the k8s app/deployment
 HEALTHCKECK: "true"

Of course, you can always go to GitLab, under Build >> Pipeline editor and click on the “Full configuration” tab to see the compiled pipeline. As a good practice, you could have a document explaining what each variable does. Alternatively, you could have quick explanations in the GitLab-ci-vars.yml file.

Before we wrap up, there is just one more important thing I would like to mention. When implementing this strategy, try to adhere to a strict naming convention, as when adding tens or more functions; things can become difficult to track.

Conclusions

In this article, I’ve tried to present all the necessary pieces needed to design pipelines that are flexible, variable-driven and that can be easily scaled. We went through the necessary components (defining stage, adding rules, example stages, functions, workflow with rules) and a good way of arranging them in your repos. We also discussed the concept of how to “drive” the pipelines using variables… I am sure on the next implementation of this concept there will be more things we can improve upon, however if you get there first please don’t be a stranger and share your ideas.