Optimize pipeline implementation using GitLab CI Templates

Use GitLab CI templates to standardize, reduce implementation time and manual errors in your workflows

Published in

Globant

8 min readMar 8, 2023

If you develop automated workflows for your applications using GitLab CI, surely you have found yourself copying and pasting portions of previous pipelines to create a new one or even copying the entire pipeline to change a couple of things. This is useful in the short run but will introduce some maintenance problems.

Helpfully, GitLab CI has a feature to avoid those problems, reduce the duplicate code in our pipelines, facilitate the standardization of the workflows, and even accelerate the implementation of pipelines: GitLab CI Templating.

Assumptions

You have basic knowledge of GitLab CI pipelines, their rules, and their structure.
You have basic knowledge of Git repositories and branch strategies.

What is a GitLab CI Template?

A GitLab CI template is a YAML file consisting of stages, jobs, and anchors, such as a gitlab-ci.yml file. The difference is that the template won’t execute a pipeline as it is but will be referenced in a different gitlab-ci.yml file of a project to execute its workflow.

Being able to include the features available in a template inside an application GitLab CI pipeline means the code can be reused without the need for duplicity, can be managed easily in a single file, and can be used to standardize workflows. Also, a gitlab-ci.yml file can include many templates.

GitLab already has numerous pre-made templates used by the Auto DevOps feature, but depending on the use case scenario, those could be limited. For those, creating their own template is advised.

Note: You can learn more about Auto DevOps here.

Creating a Template

A template will be developed in a repository different from the application repositories. A good start is to take those jobs and stages that are common between application workflows and put them on the template file. Those jobs will possibly be linked to a specific technology or framework. Let’s take, for example, those NPM jobs that are common between multiple Node.js repositories:

image: node:12

stages:
  - build
  - test
  - release

npm-build:
  stage: build   
  script:
    - npm ci
  artifacts:
    paths:
      - node_modules
    
npm-test:
  stage: test
  script:
    - npm test

npm-security:
  stage: test
  script:
    - npm audit --production
   
npm-release:
  stage: release
  script:
    - npm publish
  artifacts:
    when: always
    reports:
      dotenv: build.env

We are going to create this file in our template repository as nodejs.yml. From here, you can add more jobs to the template and use it in a node.js repository.

Using the Template

To use a template, the YAML file must be included in the gitlab-ci.yml file of the project with the include property:

include:
 — project: devops/templates
   ref: main
   file:
     — nodejs.yaml

While including the template, the version reference can be added if it is located in a different repository. This could be a branch or a tag.

Note: You can find multiple ways to include a template here.

Customizing project pipelines

Both project and template YAML files are merged when a template is used. The algorithm used by GitLab CI for the merge is “closest scope wins”. This means that template properties can be overwritten in the project’s .gitlab-ci.yml file:

include:
 — project: devops/templates
   ref: main
   file:
     — nodejs.yaml

npm-build:
  #The job will run 'npm install' instead of 'npm ci'
  script:
    - npm install --production

If you run the pipeline, the merged YAML file will look like this:

image: node:12

stages:
  - build
  - test
  - release

npm-build:
  stage: build 
  script:
    - npm install - production
  artifacts:
    paths:
      - node_modules
   
npm-test:
  stage: test
  script:
    - npm test

npm-security:
  stage: test
  script:
    - npm audit --production
 
npm-release:
  stage: release
  script:
    - npm publish
  artifacts:
    when: always
    reports:
      dotenv: build.env

Hidden Configuration

GitLab CI also has a couple of features to optimize the pipeline file: hidden jobs and anchors:

Hidden Job: A hidden job is a GitLab CI job not processed by GitLab CI/CD. It can be used for disabling a job without deleting it or for reusable configuration using the extends keyword.

# This is a hidden job
.npm-test:
  stage: test 
  script:
    — npm test $TYPE
  
# This is the usage of the hidden job
test-unit:
  extends: .npm-test
  variables:
    TYPE: unit
  
test-int:
  extends: .npm-test
  variables:
    TYPE: int

Anchors: Anchors are GitLab CI configuration blocks that can be used to reuse duplicate content inside a YAML file. Anchors are also not processed by GitLab CI/CD and are used with the reference keyword or with tags.

#This is an anchor with a script and a rule set
.npm:
  script:
    — npm $COMMAND
  rules:
    — $CI_COMMIT_BRANCH == “main”
      when: manual
    — when: never

npm-build:
  stage: build 
  script:
    # Here the script anchor is used
    - !reference [.npm, script]
  variables:
    COMMAND: ci

npm-release:
  stage: release
  rules:
    # Here the rule anchor is used
    - !reference [.npm, rules]
  script:
    # Here the script anchor is used
    - !reference [.npm, script]
  variables:
    COMMAND: publish

Both features can be used in a template and are inherited in the project file. Both anchors and hidden jobs need unique names to avoid unexpected overwriting, so it is advisable to have naming conventions based on the template they are part of.

Template coupling

The hidden configuration can have different types of coupling between the project and template, depending on the use case. If you want to enforce a workflow, following an order of stages and mandatory jobs, you could use a template tightly coupled like the one we have been using in our example. In this case, if a team doesn’t want to run the npm-security job because of some business case, they need to overwrite the job with a rule, adding complexity to the file:

include:
  — project: devops/templates
    ref: main
    file:
      — nodejs.yaml
    
npm-security:
  rules:
    - when: never

It is even more confusing if they want to change the order of the stages to run the test before the build, for example:

include:
  — project: devops/templates
    ref: main
    file:
      — nodejs.yaml
  
stages:
  - test
  - build
  - release

If you would like to have more customization options, you could use a template loosely coupled and composed of hidden jobs and anchors:

image: node:12

stages:
  - build
  - test
  - release

.npm:
  script:
    - npm $COMMAND
  rules:
    - $CI_COMMIT_BRANCH == "main"
      when: manual
    - when: never

.npm-build:
  stage: build 
  script:
    - !reference [.npm, script]
  artifacts:
    paths:
      - node_modules
  variables:
    COMMAND: ci

.npm-test:
  stage: test
  script:
    - !reference [.npm, script]
  variables:
    COMMAND: test

.npm-security:
  stage: test
  script:
    - !reference [.npm, script]
  variables:
    COMMAND: audit - production

.npm-release:
  stage: release
  rules:
    - !reference [.npm, rules]
  script:
    - !reference [.npm, script]
  variables:
    COMMAND: publish
  artifacts:
    when: always
    reports:
      dotenv: build.env

In this kind of template, the project’s gitlab-ci.yml file needs to explicitly extend the jobs that will be used, ending in a larger but easy-to-read file. It also facilitates the customization of the workflow by only picking the jobs you need and reusing the anchors available on the template:

include:
  — project: devops/templates
    ref: main
    file:
      — nodejs.yaml

stages:
  - build
  - release

install:
  extends: .npm-build

build:
  stage: build
  needs:
    - install
  script:
    - docker build -t dummy:$CI_PIPELINE_IID .

release:
  extends: .npm-release

docker-release:
  stage: release
  rules:
    - !reference [.npm, rules]
  script:
    - docker push dummy:$CI_PIPELINE_IID

In this case, for example, the project needs to build a Docker container but still use some of the existing Node.js jobs, such as npm-build and npm-release. Here, the project workflow changes the names of the npm jobs and adds new ones.

Managing Templates

There are multiple ways of managing the templates based on their amount and their different versions. It’s always important to organize the way in which they are developed and used by the teams.

Template storing

The templates must be stored in a repository accessible from the GitLab instance. Although GitLab CI supports multiple origins to host the templates, using a repository or repositories within the same GitLab instance is advisable.

You could have a repository per template, a group of templates in a repository, or a single repository with all the templates. Some recommendations are:

Name your templates properly: This is more important if you use a single repository to manage multiple templates. Don’t be afraid of using folders to group up related templates. If you are using a repository per template, you can standardize the template file name.
Protect your templates: Who accesses the templates could also impact how they are stored. If different teams are responsible for different templates, having them located on different repositories is better, so the access is managed independently.

Template versioning

While using a template from a project, a reference to the template repository is needed. This feature allows the versioning of the templates, so changes can be released without affecting the projects depending on this template. These are two advisable approaches for template versioning. Remember that the versioning comes linked with the way the templates are stored:

Semantic version

This is a trunk-based strategy that uses semantic release. With this approach, the releases can be fixed, minor or major, generating a tag on the repository that will later be used in the project’s pipelines to reference the required version. This strategy is advised if the template doesn’t have many changes over time and if it is used by a few teams because every project reference needs to be updated with each new version release. Avoid if you are managing different templates inside a single repository.

The reference in the project’s pipeline (the reference could also point to the main branch, always having the latest changes but opening the pipeline to fail due to unnoticed breaking changes):

include:
  — project: devops/templates
    ref: v2.0.1 #also applies: main
    file:
      — nodejs.yaml

Stable and latest

This is the way the GitLab team versions their templates. You will have a stable version where only major version releases can introduce breaking changes, while in the latest version any change can be a breaking change. The GitLab team distinguishes those versions using different files (for example, Deploy.yml for the latest version and Deploy.gitlab-ci.yml for the stable version). However, this could also be managed using different branches to have those versions. It is also advisable to use tags while releasing a new version to stable, so if a previous version is needed, that tag can be referenced.

The reference in the project’s pipeline (the reference could also point to any tag as a static version):

include:
— project: devops/templates
  ref: stable #also applies: latest, v1.0.0
  file:
    — nodejs.yaml

Conclusions

GitLab CI templates are an effective tool for easing the CI/CD process and guaranteeing uniformity across projects. Using templates allows the automation best practices to be centralized, saving implementation time and reducing errors. Some tips for getting the most out of templates include keeping them up-to-date, managing them using a version strategy, and organizing them logically.

To make sure you’re using the most recent and effective templates, it’s crucial to keep an eye on GitLab documentation and changes. Start boosting the pipeline implementations using templates!