Configuring a Jenkins Pipeline using a YAML file
A few years ago our CTO wrote about building a Continuous Integration server for Ruby On Rails using Jenkins and docker. The solution has been our CI for the past years until we recently decided to make an upgrade. Why?
- Jenkins version was way out of date and it was getting difficult to upgrade
- Wolox has grown significantly over the past years and we’ve been experiencing scaling issues
- Very few people knew how to fix any issues with the server
- Configuring jobs was not an easy task and that made our project kickoff process slower
- Making changes to the commands that each job runs was not easy and not many people had permissions to do so. Wolox has a wide range of projects, with a wide variety of languages which made this problem even bigger.
Taking into account these problems, we started digging into the newest version of Jenkins to see how we could improve our CI. We needed to build a new CI that could, at least, address the following:
- Projects must be built using Docker. Our projects depend on one or multiple docker images to run (app, database, redis, etc)
- Easy to configure and replicate if necessary
- Easy to add a new project
- Easy to change the building steps. Everyone working on the project should be able to change if they want to run npm install or yarn install.
Installing Jenkins and Docker
Installing Jenkins is straightforward. You can visit Jenkins Installation page and choose the option that best suits your needs.
Here are the steps we followed to install Jenkins in AWS:
sudo rpm — import https://pkg.jenkins.io/debian/jenkins.io.key
sudo wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins.io/redhat/jenkins.repo
sudo yum install java-1.8.0 -y
sudo yum remove java-1.7.0-openjdk -y
sudo yum install jenkins -y
sudo yum update -y
sudo yum install -y docker
Automatically adding projects from Github
Adding projects automatically from Github can be achieved using the GitHub Branch Source Plugin. It allows Jenkins to scan a GitHub organization for projects that match certain rules and add them to Jenkins automatically. The only constraint that all branches must meet in order to be added is that they contain a Jenkinsfile that explains how to build the project.
Easy to change configuration
Not so easy to change configuration
One of the biggest pains we had with our previous Jenkins was the difficulty of changing the steps necessary to build the project. If you looked at a project’s build steps, you would find something like this:
And some post build steps that cleaned up the docker:
Although these commands are not complex, changing any of them required someone with permissions to modify the job and an understanding ofwhat needed to be done.
Jenkinsfile to the rescue… or not
With the current Jenkins version, we can take advantage of the pipelines and model our build flow in a file. This file is checked into the repository and, therefore, anyone with access to it can change the build steps. Yay!
Jenkins’ pipelines even have support for:
- Docker and multiple images can be used for a build!
- Setting environment variables with withEnv and many other built -in functions that can be found here.
This makes a perfect case for Wolox. We can have our build configuration in a file that’s checked into the repository and can be changed by anyone with write access to it. However, a Jenkinsfile for a simple rails project would look something like this:
This file is not only difficult to read, but also difficult to change. It’s quite easy to break things if you’re not familiar with Groovy and even easier if you know nothing about how Jenkins’ pipeline works. Changing or adding a new Docker image isn’t straightforward and might lead to confusion.
Configuring Jenkins via YAML
Personally, I’ve always envied simple configuration files for CIs and this time it was our chance to build CI that could be configured using a YAML file. After some analysis we concluded that a YAML like this one would suffice:
It outlines some basic configuration for the project, environment variables that need to be present during the run, dependentservices, and our build steps.
Jenkinsfile + Shared Libraries = WoloxCI
After investigating for a while about Jenkins and the pipeline, we found that we could extend it with shared libraries. Shared libraries are written in groovy and can be imported into the pipeline and executed when necessary.
If you look carefully at this Jenkinsfile
We’ll see that the code is a chain of methods calls that receive a closure, where we execute another method passing a new closure to it.
Groovy’s flexible enough to allow this same declarative code to be created at runtime, making our dream of using a YAML to configure our job come true!
Introducing Wolox-CI
That’s how wolox-ci was born- our shared library for Jenkins!
With wolox-ci, our Jenkinsfile is now reduced to:
Now it simply checks out the code and then calls wolox-ci. The library reads yaml file like this one
and builds the Jenkinsfile to get your job running on the fly.
The nice part about having a shared library is that we can extend and fix our library in a centralized way. Once we add new code, the library is automatically updated in Jenkins which will notify all of our jobs with the update.
Since we have projects in different languages we use Docker to build the testing environment. WoloxCI assumes there is a Dockerfile to build and will run all the specified commands inside the container.
Woloxci config.yml
Config
The first part of the config.yml file specifies some basic configuration: project’s name and Dockerfile location. The Dockerfile is used to build the image where the commands will be run.
Services
This section describes which services will be exposed to the container. Out of the box, WoloxCI has support for postgresql, mssql and redis. You can also specify the docker image version you want! It is not hard to add a new service. You just need to add the corresponding file at
https://github.com/Wolox/wolox-ci/tree/development/vars
and modify how the services are parsed
https://github.com/Wolox/wolox-ci/blob/development/src/com/wolox/parser/ConfigParser.groovy#L76
Steps
The listed commands in this section will run inside the Docker container. As a result, you’ll see each of the steps on the Jenkins UI.
Environment
If you need some environment variables during your build, you can specify them here. Whatever variable you set will be available inside the Docker container when your commands listed in the steps section described above.
Wrapping up
WoloxCI is still being tested with a not-so-small sample of our projects. The possibility of changing the build steps through a YAML file makes it accessible for everyone and that is a great improvement in our CI workflow.
Docker gives us the possibility of easily changing the programming language without making any changes to our Jenkins installation and Jenkins’ Github Organization feature automatically adds new projects when a new repository with a Jenkinsfile is detected.
All of these improvements have reduced the time we spend maintaining Jenkins significantly and give us the possibility of easily scaling without any extra configuration.
