Jump start CI/CD process using gitlab runner

RakeshJayaram
Nov 22, 2020 · 6 min read
Image for post
Image for post

What is Continuous Integration?

Consider an application that has its code stored in a Git repository in GitLab. Developers push code changes every day, multiple times a day. For every push to the repository, you can create a set of scripts to build and test your application automatically, decreasing the chance of introducing errors to your app.

This practice is known as Continuous Integration; for every change submitted to an application — even to development branches — it’s built and tested automatically and continuously, ensuring the introduced changes pass all tests, guidelines, and code compliance standards you established for your app.

GitLab itself is an example of using Continuous Integration as a software development method. For every push to the project, there’s a set of scripts the code is checked against.

What is Continuous Delivery?

Continuous Delivery is a step beyond Continuous Integration. Your application is not only built and tested at every code change pushed to the codebase, but, as an additional step, it’s also deployed continuously, though the deployments are triggered manually.

This method ensures the code is checked automatically but requires human intervention to manually and strategically trigger the deployment of the changes.

What is Continuous Deployment?

Continuous Deployment is also a further step beyond Continuous Integration, similar to Continuous Delivery. The difference is that instead of deploying your application manually, you set it to be deployed automatically. It does not require human intervention at all to have your application deployed.

Introduction to GitLab CI/CD

GitLab CI/CD is a powerful tool built into GitLab that allows you to apply all the continuous methods (Continuous Integration, Delivery, and Deployment) to your software with no third-party application or integration needed.

Runners or gitlab Runners are processes running in a remote machine or instance connected to the gitlab instance.

Pipelines are set of predefined tasks triggered by an internal e.g git commit or an external event e.g slack post

Defining a CI/CD use case

The requirement is to validate and deploy the python app into a linux box and s3 buckets.

Validate

  • Run a few unit test cases on the app

Deploy

  • Copy the application into an linux machine

Step 1 : Download of prerequisite in a remote/deployment machine

The prerequisite softwares required to be installed on the remote machine/deployment server for the exercise.

git scm

Sync the code from gitlab into the remote machine on predefined event (e.g git commit)

https://git-scm.com/downloads

aws cli

Automate upload/download to aws s3 and can also perform any aws tasks via command line interface

https://awscli.amazonaws.com/AWSCLIV2.msi

pscp.exe

Sync the artifacts from gitlab into multiple linux machines

https://the.earth.li/~sgtatham/putty/latest/w64/pscp.exe

gitlab runner

Run pipelines on git runner process on a remote/deployment machine

https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-windows-amd64.exe

Installation of python , aws cli , pscp.exe(putty) is beyond scope of this web blog.

Step 2 : Installation of gitlab runner in a remote/deployment machine

Download the gitlab runner into a directory (C:\GitLab-Runner)

Rename the binary to gitlab-runner.exe

Register the Runner

Open the command prompt with administrator access

Run the command

gitlab-runner register

Enter the gitlab instance url

e.g. https://gitlab.com

Enter the registration token.

Follow the below steps for gitlab instance url and registration token.

Image for post
Image for post
gitlab runner url and registration token

Enter the required tags

The matching tags on pipeline will only be allowed to run on the runner unless specified to ignore.

e.g development , datawarehouse , marketingBU

Enter the executor as “Shell”

The gitlab runner can be configured to run a shell script , docker , docker on kubernetes etc. Detailed documentation can be found on the https://docs.gitlab.com/runner/

Update the config.toml and add the below the “executor” line (If required)

Once the gitlab runner is successfully registered, the config.toml and gitlab webUI should display the registered runner and its relevant details as shown below.

Image for post
Image for post
gitlab runner registration

Install and Start the gitlab runner

cd C:\GitLab-Runner

.\gitlab-runner.exe install — user USERNAME — password PASSWORD

.\gitlab-runner.exe start

[DEBUG] : If the gitlab runner service doesn’t start with the network credentials then follow the below steps:

click on start → services.msc → gitlab-runner →LogOn → update the username/password → click on OK

Image for post
Image for post
services account update

Step 3 : Building the pipeline definition

To define a CI/CD pipeline in a project, it is required to create a .gitlab-ci.yml in the /root directory of the repository. The pipeline contains a bunch of tasks which should be run in a specific order and details. Any command that can be run in “powershell” interface can be configured to run via the pipeline. The git runner felicitates to run those commands on the remote/deployment machine. Below are some of the tasks that can be run in an CI/CD pipeline.

  • Test a functionality by running a Selenium code

For this exercise , Below is the .gitlab-ci.yml file.

The below pipeline contains 2 stages (validate and test) and each stage contains one or more task/job to be performed via a powershell command.

Validate

  • Task/Job 1 → Run the validate.py python script on the remote/deployment machine to perform basic unit testing on the python app.

Deploy

  • Task/Job1 → Deploy the python code into multiple linux machine

For detailed information on each of the keyword refer the page : https://docs.gitlab.com/ee/ci/yaml/

Create variables in gitlab to be consumed across the pipelines;

Image for post
Image for post

Shell script;

Step 4 : Pipeline execution

Let’s commit some code and see the output of the pipeline.

The gitlab runner will first sync the code from git into the remote/deployment server under C:\GitLab-Runner\builds\<registration token>\<edit number>\*

Image for post
Image for post
pipeline
Image for post
Image for post
pipeline completion
Image for post
Image for post
pipeline tasks/jobs completion
Image for post
Image for post
pipeline output

[DEBUG] : If the pipeline is in pending state:

click on Settings → CI/CD → edit icon next to the runner

Image for post
Image for post

Documentations

Gitlab CI/CD getting started: https://docs.gitlab.com/ee/ci/quick_start/
Gitlab CI/CD configuration docs: https://docs.gitlab.com/ee/ci/yaml/

Nerd For Tech

Technical Media House

RakeshJayaram

Written by

Decisive and multi-faceted IT professional with 12 years of experience delivering data management projects with 5+ years experience in cloud and big data tech

Nerd For Tech

We are tech nerds because we believe in reinventing the world with the power of Technology. Our articles talk about some of the most disruptive ideas, technology, and innovation.

RakeshJayaram

Written by

Decisive and multi-faceted IT professional with 12 years of experience delivering data management projects with 5+ years experience in cloud and big data tech

Nerd For Tech

We are tech nerds because we believe in reinventing the world with the power of Technology. Our articles talk about some of the most disruptive ideas, technology, and innovation.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store