Best practices for CI/CD Microservices pipelines with Gitlab-CI

About Gitlab Ci

Hello everyone, I want to share about some advantages you have when you use Gitlab CI/CD tool while focusing on Gitlab Ci/CD included steps.

GitLab CI/CD is a tool built into GitLab for software development through the continuous methodologies:

• Continuous Integration (CI)
• Continuous Delivery (CD)
• Continuous Deployment (CD)

So how it works?

Your repository will contain a “.gitlab-ci.yml” file that contains the CI/CD configuration as follows:

• The scripts you want to run.
• Other configuration files and templates you want to include.
• Dependencies and caches.
• The commands you want to run in sequence and those you want to run in parallel.
• The location to deploy your application to.
• Whether you want to run the scripts automatically or trigger any of them manually.

Here we have an example of .gitlab-ci.yml content taken from https://docs.gitlab.com/ee/ci/yaml/gitlab_ci_yaml.html:

This CI/CD pipeline contains Ruby test and build stages

CI/CD template

What should you do if many of your CI/CD steps are being used in various projects? would you duplicate it for each .gitlab-ci.yml file? How do you perform an upgrade?

As a Devops Engineer in my organization I have noticed that pipelines in different projects share many similarities and upgrading the step became a difficult mission. For that purpose Gitlab-CI introduces the “include” option.

This is the reference for the “include” option from gitlab docs -

Use include to include external YAML files in your CI/CD configuration. You can break down one long gitlab-ci.yml file into multiple files to increase readability, or reduce duplication of the same configuration in multiple places.

so i have started to dive in a look for an option to include a CI/CD step from a centralized project and then figured it out with include:remote

the include remote option allows you to include external YAML files from remote repository in your CI/CD configuration, while you can specify particular tag/branch for the content of those yaml files, but the remote repo must be public repo.

As we are using managed Gitlab I have started a new public repo that contains our microservices CI/CD steps as follows -

This project contains all our CI and CD yaml files for all our microservices.

the CD directory contains yaml files which are steps that used for deploy the microservices into Kubernetes(helm upgrade/install) , while the CI steps were for build-compile the code and build-push the docker into our registry.

So how does the .gitlab-ci.yml file of each microservice looks like? Here is my outcome:

As you can see all of the Yaml files are being referred from the remote “Solaredge Microservice Gitlab Ci Steps” public project with the “v1.2.16” remote tag.

So what are the advantages of managing the yaml files on a centralized project?

* You can reduce time and costs spent to develop custom pipelines for each microservice.

* You can manage pipelines upgrade easily as once you can manage dependencies efficiently.

* You can track your changes on a centralized way.

* You can easily generate new steps.

* Pipelines are easy to learn.

Enjoy! :)