CI/CD essentials from scratch with Gitlab.

Being part of the development team I heard my infra team talking about enabling CI/CD to the projects that are being implemented. If you are part of dev or infra team you know the pain of doing the deployments for small minute code changes. During deployment, there could be multiple stages to clean, build the project, test and finally deploy it which requires all your attention and intimate the dev team about the failures if so in between.

So the question is why do want to waste time doing so? Because anyway the whole codebase must be using version control systems(Git) these days. Once you use Git, codebase deployment requirements will become tougher and tougher.

  • One team will ask you that they want to deploy as soon as they make commit to ‘xyz’ branch.
  • One team will ask you to deploy as soon as they resolve the bug.
  • One team will ask you to generate .apk or .iso files everyday morning and send them to the testing team for testing.
  • ……(never ends 😡)

So I tried enabling CI/CD to my projects. I personally use Gitlab for version control(Because it is free. Now Github is also free for private repositories. Check it out 👉 ). I didn’t know what is CI/CD prior to this blog. I tried various possible combinations with Gitlab CI/CD. Here I am writing them from scratch. Let’s get started.

So what is CI/CD? 💁

CI/CD is a coding philosophy and set of practices that drive development teams to implement small changes and check in code to version control repositories frequently and deploy them to servers based on the configurations you have mentioned.

Before trying out various cases, make sure you create a repository in Gitlab and clone it and keep it ready.

To enable CI/CD Gitlab requires us to follow two steps. They are

  1. Add .gitlab-ci.yml to project’s root folder.
  2. Setup a Runner.

Add .gitlab-ci.yml:-

In the root folder of your Gitlab project add a file with the name .gitlab-ci.yml . Gitlab uses this file for CI/CD. All your configurations for CI/CD should be added here. Since it is under git project is also part of the version control system. As you change the configurations in this file and push it to Gitlab, It automatically runs the jobs as per your configurations using your Runner configuration.

Setup a Runner:-

In GitLab, Runners run the jobs that you define in .gitlab-ci.yml. A Runner can be a virtual machine, a VPS, a bare-metal machine, a docker container or even a cluster of containers. Here I use the default Gitlab runner. If you want to configure your own runner Please check gitlab ci/cd documentation.

Now let's try different configurations.

  1. A Basic Job:-

Let’s start in a traditional way. I mean let us create a basic job that says “Hello world” 😃

The syntax for declaring a job in .gitlab-ci.yml is as follows.

job name:
script:
- your shell script.
- more script.....
Ex:-sayhello:
script:
- echo "Hello world"

Tips:-

  • .gitlab-ci.yml should contain at least one job. If there are no jobs defined in it when you push the code to Gitlab, Gitlab runner will throw an error saying invalid yaml configuration. the job running fails.
  • Make sure you validate .gitlab-ci.yml before you commit and Gitlab. If there is any validation error you can get it during the validation process. I use Gitlab Workflow ❣️, A great plugin for git operations with visual studio code.
  • You can see all the pipelines of a project under the Pipelines section of Gitlab repository.
  • You can see the logs of the job by clicking on it. ✅ button indicates your job ran successfully. Yellow indicates your job is paused. ❌ indicates that your job failed. Blue represents that the job is running.
  • Any .gitlab-ci.yml supported keywords can't be used as a job name.

Once you commit the above code and push it to Gitlab, it should run a job with the specified name and execute the script you mentioned under script tag.

2. Multiple jobs:-

Okay now we know how to write a single job, Can we write multiple jobs in .gitlab-ci.yml?

The answer is Yes. You can write as many as jobs you want. The Only condition is that Jobs should have different names. Let’s write 2 jobs in which one says hi and the other says Bye.

#A job with sayHello name
sayHello:
script:
- echo "hi"
#A job with sayBye name
sayBye:
script:
- echo "Bye"

Tips:-

  • Make sure you don’t have multiple jobs with the same name If your config has multiple jobs with the same name, beginning jobs will be skipped and the last job will be executed.
  • The default stage for a given job is test.

The output should look like this if you click on the pipeline id which starts with #.

3. Jobs with stages:-

Generally, every project will have stages before deploying like clean, build, test and deploy. Each stage will have a set of commands to execute. How do we configure jobs in with stages in .gitlab-ci.yml ?

#stages can be declared using a keyword stages. You can add any number of stages as you require.stages:
- clean
- build
- test
- deploy
#Lets define a job with for every stage.jobCleaning:
stage: clean # This tag tells gitlab to run this job only for clean stage
script:
- echo "Cleaning the code"
jobBuilding:
stage: build # This tag tells gitlab to run this job only for build stage
script:
- echo "Building the code"
jobTesting:
stage: test # This tag tells gitlab to run this job only for test stage
script:
- echo "Testing the code"
jobDeploying:
stage: deploy # This tag tells gitlab to run this job only for deploy stage
script:
- echo "Deploying the code"

Tips:-

  • Every stage need not have a job declared.
  • Order of the job execution is based on the order of stage declaration. If a stage doesn’t have at least one job defined, Then that stage will be skipped and goes to next stage job.
  • A stage can have multiple jobs defined.
  • if you have defined a stage called “test”, and define a job without stage added in it, Then the job will run under test stage since the default stage is test.
  • If you haven’t defined the test stage and create a job without stage defined in it, Then Gitlab pipeline will throw an error like yaml invalid and will ask you to specify the stage for the job as shown below.

4. Branch-specific jobs:-

By default with the previous configurations we had, Whenever you make a push to any other branch the pipeline will get triggered on the branch. This setup is good if you have common functionality to implement. But let’s try to add a config which runs only when you push the code to dev branch.

This can be done very simply by adding only tag to a job.

stages:
- clean
- build
- test
- deploy
jobCleaning:
stage: clean clean stage
script:
- echo "Cleaning the code"
only:
-dev
jobBuilding:
stage: build
script:
- echo "Building the code"
only:
-dev
jobTesting:
stage: test
script:
- echo "Testing the code"
only:
-dev
jobDeploying:
stage: deploy
script:
- echo "Deploying the code"
only:
-dev

As discussed above snippet will run on all operation you perform on dev branch i.e for merge requests, issues, pushes.

Tips:-

  • only will accept regular expressions. If your commit matches with regular expression, pipeline gets activated and jobs will executed as per configuration.
  • allowed values to only tag are as follows.

But what if you want to run your job only for issue commits or new merge requests only?

5. Excepting a job from pipeline:-

What if your requirement is not to run your pipeline for a certain case?

We can achieve this by providing exceptto the job configuration. See the following example

stages:
- clean
- build
- test
- deploy
jobCleaning:
stage: clean clean stage
script:
- echo "Cleaning the code"
only:
- dev
except:
- merge_requests # this job will not run for all merge requests on dev branch
jobBuilding:
stage: build
script:
- echo "Building the code"
only:
- dev
jobTesting:
stage: test
script:
- echo "Testing the code"
only:
-dev
jobDeploying:
stage: deploy
script:
- echo "Deploying the code"
only:
-dev

If your job has both only, except configurations, The flow of execution as follows

  • only and except are inclusive. If both only and except are defined in a job specification, the ref is filtered by only and except.
  • only and except allow the use of regular expressions (using Ruby regexp syntax).
  • only and except allow to specify a repository path to filter jobs for forks.

6. Fail-safe jobs:-

By default If a pipeline contains multiple jobs in multiple stages, If an error occurs at any job, Next jobs will be skipped with a pipeline status as failed.

But If we still want to proceed and continue the pipeline, we can achieve this by adding a config param called allow_failure. It accepts either true or false . The default value is false. That is the reason if any error occurs in pipeline remaining jobs will be skipped.

See the below example configuration.

stages:
- clean
- build
- test
- deploy
jobCleaning:
stage: clean clean stage
allow_failure: true #this tells gitlab to not skip other jobs in the pipeline if any error occurs in the current job.
script:
- echo "Cleaning the code"
only:
- dev
except:
- merge_requests # this job will not run for all merge requests on dev branch
jobBuilding:
stage: build
script:
- echo "Building the code"
only:
- dev
jobTesting:
stage: test
script:
- echo "Testing the code"
only:
-dev
jobDeploying:
stage: deploy
script:
- echo "Deploying the code"
only:
-dev

Tips:-

  • If you want to retry the job in case of failure again for a certain number of times can be configured using retry in the job configuration.
  • If retry is set to 2, and a job succeeds in a second run (first retry), it won’t be retried again.
  • retry value has to be a positive integer, equal to or larger than 0, but lower or equal to 2 (two retries maximum, three runs in total).

7. Running Jobs at a certain job status:-

Okay, We have tried faile safe jobs and retrying them too. But What if we need to implement a job which runs if a job fails or a job runs successfully or even always despite the job status?

This can be achieved by using the when tag of the job configuration.

Allows values for the when tag are

  • on_success - execute job only when all jobs from prior stages succeed (or are considered succeeding because they are marked allow_failure). This is the default.
  • on_failure - execute job only when at least one job from prior stages fails.
  • always - execute job regardless of the status of jobs from prior stages.
  • manual - execute job manually (added in GitLab 8.10). Read about manual actions below.

Let's see an example which runs a job called jobSuccess for the successful running of the job, jobFailure which runs on job failure, runAlways which runs always.

stages:
- clean
- build
- test
- deploy
jobCleaning:
stage: clean clean stage
script:
- echo "Cleaning the code"
only:
- dev
except:
- merge_requests # this job will not run for all merge requests on dev branch
jobBuilding:
stage: build
script:
- echo "Building the code"
only:
- dev
jobTesting:
stage: test
script:
- echo "Testing the code"
only:
-dev
jobDeploying:
stage: deploy
script:
- echo "Deploying the code"
only:
-dev
jobSuccess:
script:
- echo "Running on successful running of the job."
when: on_success
stage: test
jobFailure:
script:
- echo "Running on failure of the job."
when: on_failure
stage: test
jobAlways:
script:
- echo "Running always job"
when: always
stage: test

Note:-

  • Observe that jobFailure is skipped because jobTesting is not Failed. It will run if jobTestingfails.

8. Defining variables in jobs:-

Gitlab allows you to create variables in the gitlab-ci.yml .These variables can be later used in the job configurations. A variable declaration should followkey:value format.

There are two types of variables in Gitlab CI/CD.

  • Global variables.
  • Job specific variables.

Both types of variables can be defined by using a tag called variables .

If you define variables in Job configuration, variables under it become job specific variables and will override the global variables with the same name are defined.

If you define variables as root tag, Values under it will become global variables. These can be accessed in any job under the pipeline.

Once you define the variables, They can be accessed using

# Global variables. These variables can be accessed in any job.
variables:
BRANCH_NAME: "dev"
CLEAN_COMMAND: "mvn clean"
OVERRIDE_VARIABLE: "OVERIDE BEFORE"
stages:
- clean
- build
- test
- deploy
jobCleaning:
stage: clean clean stage
variables: #Job specific variables.
SECRET_KEY: "This is secret key"
OVERRIDE_VARIABLE: "OVERIDED AFTER" #Varibale name is same as gloabl variable. SO this variable should be overriden.
retry: 1
script:
- echo "Cleaning the code"
- echo "${OVERRIDE_VARIABLE}" # should print OVERIDED AFTER.
- echo "${CLEAN_COMMAND}"
only:
- dev
except:
- merge_requests # this job will not run for all merge requests on dev branch
jobBuilding:
stage: build
script:
- echo "Building the code"
only:
- dev
jobTesting:
stage: test
script:
- echo "Testing the code"
only:
-dev
jobDeploying:
stage: deploy
script:
- echo "Deploying the code"
only:
-dev

These are some of the cases I have tried. There are more advanced topics like cache, artifacts, environments, include etc. I will write about them in another blog. If you want to read more about Gitlab CI/CD refer below document.

Here is my Gitlab repo for reference.

Hope you liked it ❤️ and is helpful to you.

Thanks for reading 🙏. Keep automating 🚀.

Join our community Slack and read our weekly Faun topics ⬇

If this post was helpful, please click the clap 👏 button below a few times to show your support for the author! ⬇

Faun

The Must-Read Publication for Aspiring Developers & DevOps Enthusiasts. Medium’s largest DevOps publication.

Saikrishna Dronavalli

Written by

L'earner' by birth. Writing at my own blog https://learnovercoffee.com

Faun

Faun

The Must-Read Publication for Aspiring Developers & DevOps Enthusiasts. Medium’s largest DevOps publication.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade