Migrating from Jenkins to GitLab CI : DSL side by side

Hiroki Gota
Jan 3 · 10 min read
Image for post
Image for post
Good bye Jenkins and hello GitLab CI

At Spatial Vision, we have been using Jenkins for Continuous Integration, building, testing and deploying web apps, native mobile apps, APIs and AWS resources to target environments.

Jenkins manages over 50 projects and nearly 500 jobs including active development builds and on-going maintenance tasks.

Today, we’d like to share our story on migrating from Jenkins to GitLab CI, by comparing the DSL of each platform so you can understand the differences between them and it gives you a quick introduction of GitLab CI for Jenkins users.

Our Jenkins

We have a total of 4 Jenkins instances, running on an internal Ubuntu LXC containers, and automating CI processes using a number of build machines.

The reason we run multiple Jenkins instances is that Jenkins could become slower in listing, viewing and editing of jobs when you add a lot of jobs.

We provide managed services for our clients using AWS based infrastructure and each project typically has a webapp/native mobile apps and APIs (Development/QA/UAT/Production environments).

The backed services are created and updated by AWS CloudFormation and AWS CLI.

The following are common Jenkins Jobs and they are executed based on the code change or periodic schedule.

  • Build HTML 5 web applications using Node.js and Angular
  • Build native mobile applications (Ionic, NativeScript, ReactNative) in a Mac mini build machine
  • Deploy to Google Play and iTunes using Fastlane/MS App Centre
  • Build and test APIs (Node.js, Kotlin/Java) on a Dockernized build machine
  • Build and push docker images to Docker Registry (GitLab built-in registry and Docker Hub)
  • Deploy AWS resources including S3, CloudFront, Lambda, EC2/ECS, Fargate, RDS etc.
  • Deploy Docker containers for Docker compose based APIs
  • Create/update database instances including AWS RDS (PostgreSQL and MS SQLServer)

These jobs are executed on an internal build machines (Ubuntu, Mac mini and Windows) and AWS EC2 instances in which Jenkins Agents are installed for deployment jobs.

Jenkins cons (why we looked for an alternative)

Jenkins has been around for many years and we see it as one of the best model of open source based platform. It has many plugins that are ready to perform a particular job. However we’ve found some issues over a period of time and have been looking for an alternative tools for a quite while.

  • Although Jenkins project is still active, we feel the community is shrinking and some issues never get resolved and there are many plugins that are outdated
  • Jenkins Configurations, including Global and project level, are not easily reusable as these are managed within the Jenkins UI
  • Jenkins introduced pipeline (Groovy script based ) which could be very complicated for a very simple job and you need a JVM/Groovy for a project even you might only want to build a NPM package
  • There is a configuration plugin that lets you create ‘Configuration as Code’ and it offers yaml syntax but it only supports a small set of plugins and you might need to overwrite the global configuration for a project, which might be harder to manage when you have a large number of global configuration established
  • We feel Jenkins web application user experience is quite outdated and make you click so many links to view what you’re looking for such as Console output for a particular build
  • Jenkins now offer CDF, Jenkins X, Tekton and Spinnaker, that are promoted as new generation Continuous Delivery platforms (we rather want a CI/CD platform that works well)

GitLab CI

We’ve been using Self-Managed GitLab for our code repository. GitLab provides much more than just a code repository, such as a docker registry, error tracking, wiki, issue tracking, CI etc.

GitLab CI has been around for a couple of years and it has become one of the most popular CI tools in the community. GitLab products have a great documentation and their feature development is so fast and we get new features so frequent and they are generally very useful.

We’ve tried on GitLab CI by migrating some existing Jenkins jobs including building and deploying API, webapp and native app.

We’ll walk through the migration process by comparing the DSLs used in both platforms.

Convert Jenkins DSL to GitLab CI DSL

“Build and Push Kotlin based API Docker image and deploy to a development environment”

First of all, we need a machine to run a job, which can be any platforms including Linux, Mac and Windows.

Jenkins Agent must be installed on a host machine and you specify the work space (remote directory), labels and availability. You can decide the number of executors in order to run jobs concurrently.

For GitLab, you need to install a GitLab Runner on a host machine and then register runners. You can register any number of Runners (Executors) on a host machine in order to run jobs in parallel.

GitLab Runner has a set of arch types, including SSH, Shell, Parallels, VitualBox, Docker or even Kubernetes. We like this feature as we often use a Docker container for a build and it is naturally supported:

Image for post
Image for post
GitLab Runner can use a Docker image

For building an API image, we use Shell executor to build a Docker image and push it to GitLab registry.

Jenkins only provides the Agent configuration via UI, called ‘Manage Node’. GitLab Runners can be configured via UI or using a config.toml file installed on the host of Runners.

Labels in Jenkins describes the capabilities of the Agent so when you run a job, you can specify ‘what capability is required’ instead of ‘what machine should be used’. GitLab CI calls it Tags, which seems more natural DSL.

Image for post
Image for post
Jenkins Agent vs GitLab CI Runner

We normally create a Jenkins user for a Jenkins Agent and allocate a home directory, which is the same in GitLab Runners.

Jenkins creates a directory called workspace. When you create a new job, a sub directory will be created in workspace in where the code is cloned and built.

GitLab Runner creates ‘builds’ directory and each Runner has its own sub directory inside it. Runners cache any artefacts (specified in gitlab-ci.yml) in ‘cache’ directory.

GitLab has a concept of group/sub groups to bundle related to repositories and Runners replicate the group/sub group structure inside the builds directory. e.g. builds/my-runner/my-project-group/my-api

Image for post
Image for post
Jenkins workspace vs GitLab runner builds directory

When a Runner has been installed and registered, we can then create a CI job.

In Jenkins, you need to create a ‘New Item’ to create a job AFTER you’ve created a repository somewhere (GitLab, GitHub, Bitbucket etc).

GitLab CI is immediately available for EVERY repository you create on GitLab.

Image for post
Image for post
New Item Page vs CI/CD menu

In Jenkins, you specify the build details in ‘Configure’ menu, which has a total of 6 tabs including ‘General’, ‘Source Code Management’, ‘Build Triggers’, ‘Build Environment’, ‘Build’ and ‘Post-build Actions’.

We normally use ‘Copy from’ feature to copy an existing job that is similar to a new project, then modify the content.

GitLab provides a configuration file called ‘.gitlab-ci.yml’. It needs to be placed in the root of your repository and GitLab then automatically detects the file and run a job described in the file.

Image for post
Image for post
Configure page vs .gitlab-ci.yml

In Jenkins, you will create a set of jobs (e.g. build, test and deploy) and connect them by triggering another job or use pipeline feature. For a typical API CI, we need build test and 4 deploy jobs (dev/qa/uat/prod) and glue one job to another.

GitLab provides a pipeline concept with ‘Stages’ (e.g. build, test and deploy). Each stage can have a multiple jobs and each job can have its own set of configuration including environmental variables or what branch to be executed.

Jenkins (still) supports CVS, Subversion and GIT and you can clone multiple repositories in one job. This is so generic and flexible you can fetch any number of repositories for build, test and deploy.

However, our CVS/Subversion were retired and migrated into GIT long time ago and this flexibility rather forces you to configure it every time.

You also need to somehow specify the branches to build, which does not work well when you follow Git Flow.

GitLab CI is available for each repository and you don’t need to tell any other system for how to fetch your repository.

Image for post
Image for post
Source Code Management menu vs GitLab repository

In Jenkins, you need to specify how you want to fetch your code, either periodically or poll the repository.

If you want to run a job as soon as the code has been pushed for a branch, you need to frequently poll it or use Jenkins webhook and call it on a GIT repository.

GitLab CI pipeline runs automatically as soon as a change has been pushed for a branch. It also offers 3 simple and clear DSL, ‘When’ and ‘Only/Except’ to limit jobs running within the pipeline.

When: You can set manual to only trigger a job manually or make it dependent on another job using on_success/on_failure.

Only/Except can limit what branches or tags to run a particular job. We use this to ‘deploy production’ only on a master branch type of scenarios.

Image for post
Image for post
Build Triggers vs when/only/except

Jenkins Build supports a number of scripts (shell, batch, ssh) and Java build tool such as Ant/Maven by default (you can add more builds using plugins).

We use custom shell scripts to fetch some configuration items; run test; build a docker image; push to the Docker registry.

We could run pretty much the same script in build stage of the pipeline in GitLab CI. The scripts are identical and run on the same host machine and worked great.

At this stage, we encountered a problem. We need to initialise a test database prior to running a test. In Jenkins, we have a job for it and trigger it prior to the test job.

This database initialisation was only available on Jenkins so we had to call Jenkins webhook from a GitLab CI.

In GitLab CI, the only solution we found was to use curl to call Jenkins web hook trigger in ‘before_script’ section and ‘sleep for a period of time’ to finish the database initialisation. (You could make a better script and use Jenkins CLI/API to check whether a job has been completed).

Image for post
Image for post
Build vs build stage in .gitlab-ci.yml

In Jenkins, we use Post-build Actions to run a deploy job when the build job has been successfully completed.

In GitLab, you can create triggers to run another pipeline. A trigger can be either called from .gitlab-ci.yml (using curl) or from a webhook.

Image for post
Image for post
Post-build Actions vs stages

In Jenkins, we normally create a deployment job for each environment, development/QA/UAT/Production. This is due to the fact we need our client’s approval (UAT) prior to a release to a production and we cannot fully automate for a Continuous Delivery.

We use GitFlow to achieve the deployment process with a specific branch as follows:

  • Development: Automatic deployment using develop branch
  • QA (Internal): Nightly or on demand using develop branch
  • UAT/Production: Manual using release/master branches

In Jenkins, it is not easy to make related jobs bundled and you need a very good naming conventions to make jobs somehow related each other. (Unless you use pipeline feature)

In GitLab CI, we can use stages and bundle jobs with DSLs such as environments/when/only/except so the pipeline is visualised clearly.

Image for post
Image for post

Lessons learned

We’re happy with our trial on GitLab CI and feel our CI processes are more seamless with our development activities thanks to a clear DSL with modern development practice.

We’ve since then created many CI processes using GitLab CI and there are a number of .gitlab-ci.yml templates we can reuse across projects.

The following are a few lessons learned on GitLab CI

  • As of Jan 2020, Monorepo is not fully supported and we’re simply unable to use some Monorepo tool chain, which makes Jenkins little more appealing due to its flexibility
  • Always create a project specific Runners instead of sharing by the capability of the Runner to avoid a conflict for Runner’s capabilities between projects
  • GitLab is very active and GitLab Docs is comprehensive and easy to follow. Their issue tracker is also active and many issues are solved quickly, however there are bugs shipped from time to time and it might affect on your development activities when you are fully dependent on GitLab tool chain
  • Late last year, GitLab shipped a critical bug in GitLab CI and we’re unable to build or deploy anything for a couple of days. We had to abandon all of Docker based builds for work around and wasted a lot of times
  • ‘Multi-project pipeline’ is a feature currently only available for Premium users (we’ve been using a community free edition) and there is no other reliable way to visualise and manage multi-project pipelines (which makes hard to create a sophisticated pipelines for a complex project having many components)

Code道

The Tao (Chinese: 道; pinyin: Dào; literally: “the Way” )…

Hiroki Gota

Written by

I am a Software Architect at Lapis, Melbourne Australia. I enjoy building end to end custom solutions with a Kaizen spirit in a team environment.

Code道

Code道

The Tao (Chinese: 道; pinyin: Dào; literally: “the Way” ) Code道— is a publication about all things app development. Our way — Learn through sharing

Hiroki Gota

Written by

I am a Software Architect at Lapis, Melbourne Australia. I enjoy building end to end custom solutions with a Kaizen spirit in a team environment.

Code道

Code道

The Tao (Chinese: 道; pinyin: Dào; literally: “the Way” ) Code道— is a publication about all things app development. Our way — Learn through sharing

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store