Configuring a Jenkins Pipeline using a YAML file

A few years ago our CTO wrote about building a Continuous Integration server for Ruby On Rails using Jenkins and docker. The solution has been our CI for the past years until we recently decided to make an upgrade. Why?

  • Jenkins version was way out of date and it was getting difficult to upgrade
  • Wolox has grown significantly over the past years and we’ve been experiencing scaling issues
  • Very few people knew how to fix any issues with the server
  • Configuring jobs was not an easy task and that made our project kickoff process slower
  • Making changes to the commands that each job runs was not easy and not many people had permissions to do so. Wolox has a wide range of projects, with a wide variety of languages which made this problem even bigger.

Taking into account these problems, we started digging into the newest version of Jenkins to see how we could improve our CI. We needed to build a new CI that could, at least, address the following:

  • Projects must be built using Docker. Our projects depend on one or multiple docker images to run (app, database, redis, etc)
  • Easy to configure and replicate if necessary
  • Easy to add a new project
  • Easy to change the building steps. Everyone working on the project should be able to change if they want to run npm install or yarn install.

Installing Jenkins and Docker

Installing Jenkins is straightforward. You can visit Jenkins Installation page and choose the option that best suits your needs.

Here are the steps we followed to install Jenkins in AWS:

sudo rpm — import https://pkg.jenkins.io/debian/jenkins.io.key

sudo wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins.io/redhat/jenkins.repo

sudo yum install java-1.8.0 -y

sudo yum remove java-1.7.0-openjdk -y

sudo yum install jenkins -y

sudo yum update -y

sudo yum install -y docker

Automatically adding projects from Github

Adding projects automatically from Github can be achieved using the GitHub Branch Source Plugin. It allows Jenkins to scan a GitHub organization for projects that match certain rules and add them to Jenkins automatically. The only constraint that all branches must meet in order to be added is that they contain a Jenkinsfile that explains how to build the project.

Easy to change configuration

Not so easy to change configuration

One of the biggest pains we had with our previous Jenkins was the difficulty of changing the steps necessary to build the project. If you looked at a project’s build steps, you would find something like this:

And some post build steps that cleaned up the docker:

Although these commands are not complex, changing any of them required someone with permissions to modify the job and an understanding ofwhat needed to be done.

Jenkinsfile to the rescue… or not

With the current Jenkins version, we can take advantage of the pipelines and model our build flow in a file. This file is checked into the repository and, therefore, anyone with access to it can change the build steps. Yay!

Jenkins’ pipelines even have support for:

  • Docker and multiple images can be used for a build!
  • Setting environment variables with withEnv and many other built -in functions that can be found here.

This makes a perfect case for Wolox. We can have our build configuration in a file that’s checked into the repository and can be changed by anyone with write access to it. However, a Jenkinsfile for a simple rails project would look something like this:

This file is not only difficult to read, but also difficult to change. It’s quite easy to break things if you’re not familiar with Groovy and even easier if you know nothing about how Jenkins’ pipeline works. Changing or adding a new Docker image isn’t straightforward and might lead to confusion.

Configuring Jenkins via YAML

Personally, I’ve always envied simple configuration files for CIs and this time it was our chance to build CI that could be configured using a YAML file. After some analysis we concluded that a YAML like this one would suffice:

It outlines some basic configuration for the project, environment variables that need to be present during the run, dependentservices, and our build steps.

Jenkinsfile + Shared Libraries = WoloxCI

After investigating for a while about Jenkins and the pipeline, we found that we could extend it with shared libraries. Shared libraries are written in groovy and can be imported into the pipeline and executed when necessary.

If you look carefully at this Jenkinsfile

We’ll see that the code is a chain of methods calls that receive a closure, where we execute another method passing a new closure to it.

Groovy’s flexible enough to allow this same declarative code to be created at runtime, making our dream of using a YAML to configure our job come true!

Introducing Wolox-CI

That’s how wolox-ci was born- our shared library for Jenkins!

With wolox-ci, our Jenkinsfile is now reduced to:

Now it simply checks out the code and then calls wolox-ci. The library reads yaml file like this one

and builds the Jenkinsfile to get your job running on the fly.

The nice part about having a shared library is that we can extend and fix our library in a centralized way. Once we add new code, the library is automatically updated in Jenkins which will notify all of our jobs with the update.

Since we have projects in different languages we use Docker to build the testing environment. WoloxCI assumes there is a Dockerfile to build and will run all the specified commands inside the container.

Woloxci config.yml

Config

The first part of the config.yml file specifies some basic configuration: project’s name and Dockerfile location. The Dockerfile is used to build the image where the commands will be run.

Services

This section describes which services will be exposed to the container. Out of the box, WoloxCI has support for postgresql, mssql and redis. You can also specify the docker image version you want! It is not hard to add a new service. You just need to add the corresponding file at

https://github.com/Wolox/wolox-ci/tree/development/vars

and modify how the services are parsed

https://github.com/Wolox/wolox-ci/blob/development/src/com/wolox/parser/ConfigParser.groovy#L76

Steps

The listed commands in this section will run inside the Docker container. As a result, you’ll see each of the steps on the Jenkins UI.

Environment

If you need some environment variables during your build, you can specify them here. Whatever variable you set will be available inside the Docker container when your commands listed in the steps section described above.


Wrapping up

WoloxCI is still being tested with a not-so-small sample of our projects. The possibility of changing the build steps through a YAML file makes it accessible for everyone and that is a great improvement in our CI workflow.

Docker gives us the possibility of easily changing the programming language without making any changes to our Jenkins installation and Jenkins’ Github Organization feature automatically adds new projects when a new repository with a Jenkinsfile is detected.

All of these improvements have reduced the time we spend maintaining Jenkins significantly and give us the possibility of easily scaling without any extra configuration.

Like what you read? Give Matias De Santi a round of applause.

From a quick cheer to a standing ovation, clap to show how much you enjoyed this story.