Photo by Haley Phelps on Unsplash

Diving into the Gitlab CI pool

Building a CI pipeline might sound bombastic, but with Gitlab CI we can do it in a few minutes. At Nielsen, we had a CI/CD process running in Jenkins and we’ve decided to migrate it over to Gitlab CI.
In this post I’ll summarise why we did it and how easily you can do it too.

Let’s start by understanding the two main keywords:
CI stands for Continuous Integration. In most companies, there are multiple teams working together and pushing new features constantly. The idea behind this concept is to enable every team to write code and push features. It will also make sure all parts continue to work together without affecting other’s functionalities.

CD stands for Continuous Delivery. This concept enables the teams to deploy their changes into the different environments, such as development, testing environment, production.

We decided to make the changes in our CI/CD process due to a few downsides we were having with Jenkins:

1. Adding a new user to Jenkins was only done by our DevOps team. This happened quite a lot and caused a bottleneck in our process.
2. Setting up CI/CD for a new repository — Creating a new CI/CD process for a repository was a time consuming task that also relied on our DevOps team. This means that instead of improving the infrastructure, they were only creating new pipelines. Eventually, our delivery time got affected by that.
3. Copy/Paste — Setting up a new process involved a lot of copy/paste.
4. Console Output — The “steps” defined in Jenkins are all streaming the output to the same file. This usually means you won’t be able to find what you’re looking for easily.

Nielsen as an organisation decided to move to GitLab as the source code management and moving to Gitlab CI solved our issues:

1. It’s easier to assign permissions for existing users in the system.
2. Rewriting the jobs in a modular way will help to make the code more readable and expandable. Eventually, making the developers more involved in the process and save DevOps time.
3. Being able to split the pipeline into smaller jobs will make the output way more readable.

Gitlab CI

In order to setup a Gitlab CI flow there are two things that need to exist:

1. Available GitLab runner: The agent that runs the CI/CD process.
2. .gitlab-ci.yml file — A file that will include the pipeline for the current repository CI/CD process.

Now Let’s dive into the .gitlab-ci.yml file:

Photo by Laurie-Anne Robert on Unsplash

Note: The file needs to be located under the root of the current repository.

Pipeline, Stages, And Jobs

The file represents a pipeline, which is a composition of stages. Stages run one after the other according to the stages defined in the file.
The stages are defined in the following manner:

stages:
  - test
  - build

Each stage is linked to a single or many jobs. A job is the implementation of a specific stage: what we are about to do in that specific stage.

To explain the link between stages and jobs let’s use the stages listed above:

test:
  stage: test
  script:
    - npm install
    - npm run test

Here we have created a simple job named test, which will run at the test stage. Within the job, we install the node modules and then run the test command. Simple right?

Let’s dig a bit deeper and assume our test stage works perfectly. Now we need to build our image and push it to a registry. Our requirement is to push the image into Gitlab registry and into Dockerhub registry. As I stated previously a stage can be linked to many jobs. That means that we can create two jobs, link them to the same stage and they will run in parallel. Saves us some time doesn’t it?

gitlabRegistryBuild:
  stage: build
  script:
    - npm run buildImage
    - ... // Code to push image to gitlab registry
dockerhubRegistryBuild:
  stage: build
  script:
    - npm run buildImage
    - ... // Code to push image to dockerhub

That’s pretty cool. If we take a close look we can see that there are some repetitions in the two jobs. Gitlab provides us with a cool tool called hidden jobs. These are jobs that won’t be processed by Gitlab in the CI/CD. Each job can extend them and use that functionality. A hidden job is declared with a dot . .

.buildImageBase:
  stage: build
  before_script:
    - npm run buildImage
gitlabRegistryBuild:
  extends: .buildImageBase
  script:
    - ... // Code to push image to gitlab registry
dockerhubRegistryBuild:
  extends: .buildImageBase
  script:
    - ... // Code to push image to dockerhub

That looks a lot cleaner. Now we have a common hidden job buildImageBase . It will be running at build stage and it will run a command to build the image. Each job extends that hidden job. Hence, at build stage, the two jobs will run the same way they would have in our first example. First, they will build the image by using our hidden job. Second, each job will push the image to the proper registry.

It’s important to note that if you use before_script in both your hidden job and your actual job, the hidden job’s before_script will get overridden:

.buildImageBase:
  stage: build
  before_script:
    - npm run buildImage
gitlabRegistryBuild:
  extends: .buildImageBase
  before_script:
    - npm run buildProdImage
  script:
    - ... // Code to push image to gitlab registry
dockerhubRegistryBuild:
  extends: .buildImageBase
  script:
    - ... // Code to push image to dockerhub

gitlabRegistryBuild job won’t run npm run buildImage command. The before_script will override the one listed in the job it extends.

Variables

Declared variables are available during our pipeline. This is a really important part as it can help us build modular and scalable code. Gitlab supplies predefined variables that we can use in addition to the variables that we add to our .gitlab-ci.yml file. Within the code, variables are referenced as $VARIABLE_NAME .

variables:
  - TESTING_COMMAND: testLocal
test:
  stage: test
  script:
    - npm install
    - npm run $TESTING_COMMAND

In the example above we have declared a variable named TESTING_COMMAND. In our test stage, we’ve written npm run $TESTING_COMMAND , this will be transpiled into npm run testLocal .

Now let’s see our full .gitlab-ci.yml :

variables:
  - TESTING_COMMAND: testLocal
test:
  stage: test
  script:
    - npm install
    - npm run $TESTING_COMMAND
.buildImageBase:
  stage: build
  before_script:
    - npm run buildImage
gitlabRegistryBuild:
  extends: .buildImageBase
  script:
    - ... // Code to push image to gitlab registry
dockerhubRegistryBuild:
  extends: .buildImageBase
  script:
    - ... // Code to push image to dockerhub
stages:
  - test
  - build

Summary

That’s it everybody, we have just built our pipeline in no time.
It was simple wasn’t it?

Gitlab provides more cool and necessary tools that can help along the way such as artifacts, dependencies, manual jobs, triggering pipelines, and many more. All you need to do now is setup the Gitlab-CI runner, create a new repository, and let the magic do it’s thing.
For further reading please follow the official gitlab CI/CD docs.

If you have any questions, feel free to reach out.
I’m here and also on linkedin.