Comprehensive Guide to GitLab CI/CD

8 min readOct 16, 2023

Introduction
The Building Blocks of GitLab CI/CD
Setting Up a GitLab CI/CD Pipeline
Running Your First Pipeline
Refining Pipeline to fit your specific needs
Conclusion

Introduction

In the fast-paced world of software development, Continuous Integration/Continuous Delivery (CI/CD) has emerged as a transformative practice that helps teams to maintain the velocity, quality, and efficiency of software releases. Among the various offerings, GitLab’s CI/CD service has gained popularity due to its comprehensive features, seamless integration with the GitLab environment, and the flexibility it provides for catering to complex workflows.

GitLab CI/CD is a powerful tool that utilizes a YAML configuration file .gitlab-ci.yml within each code repository. When you commit code, GitLab CI/CD uses this “recipe” to automatically build, test, and deploy your applications to any configured environment.

In this guide, we will take a deep dive into the world of GitLab CI/CD, exploring its fundamental building blocks, and guiding you through the process of setting up and running your first pipeline. For a practical approach, we’ve included a hands-on exercise enabling you to apply the concepts you learn, thus enriching your overall understanding.

The Building Blocks of GitLab CI/CD

These are the essential building blocks:

Pipelines
Stages
Jobs

At the heart of GitLab CI/CD are Pipelines. A pipeline is essentially a series of processes, called Jobs, that get executed in Stages and in a particular order depending on the defined .gitlab-ci.yml file.

Pipelines are automatically triggered upon each commit but can also be manually initiated if required. They encompass various stages of development, including build, test, and deploy, and commonly visualize the current status of your project.

These building blocks constitute the backbone of any GitLab CI/CD workflow, providing the foundation on which you can build, deploy, and maintain robust applications seamlessly and efficiently.

Setting Up a GitLab CI/CD Pipeline

For the purpose of this tutorial, we’re utilizing a self-hosted GitLab instance. Please note that while principles and overall workflows remain the same, certain aspects of the pipeline setup experience might differ slightly if you are using GitLab’s SaaS (Software-as-a-Service) solution.

Now, we’re going to setup a basic pipeline for a application step by step.

1. Create a .gitlab-ci.yml at your project root

{project_root}/
├── .gitlab-ci.yml
├── Dockerfile
├── app/
...
├── README.md
└── (anything else)

2. Define CI/CD stages

Typically, stages included in a CI/CD pipeline are:
1. Build: Code is compiled and ready for testing.
2. Test: Automated tests are executed on built code.
3. Deploy: After the build stage and test stages pass, the application could be deployed.
The names and number of stages can vary depending on the specific needs of your project. Others might include stages like Lint, SAST, DAST, or Release.

# in .gitlab-ci.yml

stages:
  - build
  - test
  - deploy

3. Define jobs

The script section in a job is where you define the list of commands that the GitLab Runner will execute in a vm or container. These commands represent the tasks that make up the job, such as building your code, running tests, or deploying your application. Each command runs sequentially in the same shell.

# in .gitlab-ci.yml
stages:
  - build
  - test
  - deploy

build: # this is a job
  stage: build
  script:
    - echo "Building"

unittest: # another job
  stage: test
  script:
    - echo "Run a unittest for the app"

deploy: # another job
  stage: deploy
  script:
    - echo "Deploying the app"

4. Define who (Gitlab Runner) will execute the job

Here we’re using YAML Anchors and Aliases, that help us reuse the values without rewrite it everywhere.

# in .gitlab-ci.yml
stages:
  - build
  - test
  - deploy

# In GitLab CI, a section that begins with a dot (.) is regarded as a template.
# By utilizing YAML's `anchor and alias` features, we can significantly minimize redundancy.
# `&tags_gcp_dev` means define a variable `tags_gcp_dev` and it's value
.tags_for_dev_jobs: &tags_gcp_dev
  tags:
    - gcp
    - dev

build:
  stage: build
  script:
    - echo "Building"

  # it means override (merge) this section with `tags_gcp_dev` that we defined before
  <<: *tags_gcp_dev

unittest:
  stage: test
  script:
    - echo "Run a unittest for the app"
  <<: *tags_gcp_dev

deploy:
  stage: deploy
  script:
    - echo "Deploying the app"
  <<: *tags_gcp_dev

Hint: the template will be translated by GitLab YAML parser

build:
  stage: build
  script:
    - echo "Building"
  <<: *tags_gcp_dev

# will be translated to

build:
  stage: build
  script:
    - echo "Building"
  tags:
    - gcp
    - dev

5. The last, Commit and Push

Once you’ve defined your stages and jobs, commit the .gitlab-ci.yml file to the root of your repository and push the commit to GitLab.

After the commit is pushed, GitLab will detect the .gitlab-ci.yml file automatically and initiate the pipeline. It’ll run the jobs as described in the file, outputting the status and log for each stage.

Running Your First Pipeline

Now that you’ve set up a basic GitLab CI/CD pipeline, it’s time to see it in action. Here’s how you get it running:

Commit and Push Your Changes

For now, Gitlab CI/CD will be triggered when commit to any branch at remote.

git add .gitlab-ci.yml
git commit -m 'chore: add CI/CD config file'

# Use main branch for test only.
git push origin main

2. View Your Pipeline

Go to your project in Gitlab, and get into Build> Pipelines , then you’ll see all the pipelines.

Just click one of them, you’ll see the pipeline detail.

3. Inspect Jobs and Stages:

If something failed the pipeline, just click the job to see what’s going on, then fix the problem -> commit -> push.

So, the workflow would be like a cycle:

coding
commit & push
trigger pipeline
if pipeline failed, go to 1.
done

Refining Pipeline to fit your specific needs

Generally, we would like to trigger a pipeline when:

commit to a Merge Request
merged to default (or specific) branch
git tag matched patterns for production release

Commit to Merge Request

You can configure your pipeline to run specific jobs or stages when changes are committed to a Merge Request. This can be done using the only and except keywords in your .gitlab-ci.yml file

test_on_mr:
 stage: test
 script: echo “Running tests on Merge Request…”
 only:
   — merge_requests

In this example, the test_on_mr job will only run when changes are committed to a Merge Request.

Merge to Main Branch

Similarly, you can define jobs that should only run when changes are merged into the main (or any specific) branch

deploy_on_main:
 stage: deploy
 script: echo "Deploying changes to production"
 only:
   - main

Git Tag matched patterns

deploy_on_release_tag:
 stage: deploy
 script: echo "Deploying changes to production"
 rules:
   - if: '$CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/'
   - when: never

This job will be triggered when tagging a Semantic Versioning tag (v1.2.3 ,v2023.01.01 , …) to you repo.

Semantic Versioning 2.0.0

Semantic Versioning spec and website

semver.org

A simplest pipeline:

# .gitlab-ci.yml

# pre-setting
variables:
  AUTO_DEVOPS_BUILD_IMAGE_EXTRA_ARGS: "--build-arg=VERSION=${CI_APPLICATION_TAG}"
  AUTO_DEVOPS_BUILD_IMAGE_FORWARDED_CI_VARIABLES: CI_APPLICATION_TAG
  PYTHON_IMAGE_TAG: "3.10-slim"
  POETRY_VERSION: "1.5.1"

stages:
  - build
  - unittest
  - deploy

# Workflow: Control what types of pipeline run.
# In this example, if Dockerfile does not exists in the project, the pipeline won't be triggered.
workflow:
  rules:
    - exists:
        - Dockerfile

# template: defined reusable components
# `project: cicd/ci-template` means project `ci-template` in group `cicd`
# `file`: the file in project `cicd/ci-template`
include:
  - project: cicd/ci-template
    file:
      - /Job/Build.gitlab-ci.yml
      - /Job/UnitTest.Python.gitlab-ci.yml

# before_script:
# scripts will run `before` each job, could be overrided in each job's `before_script`
before_script:
  - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY

# tag templates, define who (Gitlab Runner) will execute the job. 
.tag_gcp: &tag_gcp
  tags:
    - gcp

.tag_testing: &tag_testing
  tags:
    - testing

.tags_staging: &tags_staging
  tags:
    - staging

.tags_prod: &tags_production
  tags:
    - production

# Environment template
# Let CI/CD knows which env the pipeline is running, could be useful for environment variables (.env) injection.
.environment_unittest: &environment_unittest
  environment:
    name: Unittest

.environment_testing: &environment_testing
  environment:
    name: Testing
    url: https://testing.example.com/api/

.environment_staging: &environment_staging
  environment:
    name: Staging
    url: https://staging.example.com/api/

.environment_production: &environment_production
  environment:
    name: Production
    url: https://example.com/api/

# jobs

# build stage is defined in `cicd/ci-template` and not present in this file,
# just the scripts that how to build you application

# build:
#   stage: build
#   script:
#     - docker build -t ${CI_REGISTRY_IMAGE}:ooxx .
#     - docker push ${CI_REGISTRY_IMAGE}:ooxx
#   <<: *tag_gcp

# the unittest is defined in `cicd/ci-template`, could be override in each project 
unittest:
  variables:
    ENV_FILENAME: .env
  rules:
    - if: $CI_MERGE_REQUEST_IID

# .deploy is still a template that will be extend in different environments
.deploy:
  needs: [ "build", "unittest" ]
  script:
    - docker pull $CI_APPLICATION_IMAGE
    - docker-compose down --remove-orphans
    - docker-compose up -d

# deploy to testing or dev env when commit to a branch has an opening Merge Request
deploy-mr-review-app:
  stage: deploy
  extends:
    - .deploy
  <<: *tag_testing
  <<: *environment_testing
  variables:
    CI_APPLICATION_IMAGE: ${CI_REGISTRY_IMAGE}:mr-${CI_MERGE_REQUEST_IID}
  rules:
    - if: $CI_MERGE_REQUEST_IID

# deploy to staging env with manually trigger (click button on Gitlab UI)
deploy-review-staging:
  stage: deploy
  extends:
    - .deploy
  <<: *tags_staging
  <<: *environment_staging
  variables:
    CI_APPLICATION_IMAGE: ${CI_REGISTRY_IMAGE}:mr-${CI_MERGE_REQUEST_IID}
  rules:
    - if: $CI_MERGE_REQUEST_IID
      when: manual
    - when: never

# tag `v1.0.0` or `v2.0.1`, and deploy code to production
deploy-production:
  stage: deploy
  needs: [ "build" ]  # override .deploy 
  extends:
    - .deploy
  <<: *tags_production
  <<: *environment_production
  variables:
    CI_APPLICATION_IMAGE: ${CI_REGISTRY_IMAGE}:${CI_COMMIT_TAG}
  rules:
    - if: '$CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/'
      when: manual
    - when: never

Conclusion

You’ve just taken an in-depth overview of GitLab CI/CD, from understanding its fundamental building blocks to configuring and running your first pipeline. We’ve also dived into how to refine your pipeline based on certain conditions, like running jobs when changes are committed to a Merge Request or merged to the main branch, or even when a new tag is created.

GitLab CI/CD is a powerful tool for automating the software development workflow. It helps in maintaining higher code quality, ensuring that the code is buildable and testable at all times. It is customizable yet capable of handling complex workflows in a straightforward way.
Keep in mind that today’s guide just scratches the surface. GitLab CI/CD is flexible and expansive, integrating seamlessly with many other services and supporting many advanced features. Deepening your knowledge and experience with GitLab CI/CD will surely enhance your productivity and product’s reliability.

Don’t be afraid to try new configurations, explore the different features, and tailor your CI/CD pipeline to fit your project’s needs.

If you enjoyed my article, don’t hesitate to share it around and give it some claps. Your support is greatly appreciated and motivates me to write more! Happy Coding!

References

GitLab CI/CD
https://docs.gitlab.com/ee/ci/
GitLab .gitlab-ci.yml keywords
https://docs.gitlab.com/ee/ci/yaml/
GitLab CI/CD Predefined variables
https://docs.gitlab.com/ee/ci/variables/predefined_variables.html