Building Jenkins as a code

In our company, we are trying to use ‘Everything as a Code’ paradigm which is about having reproducible infrastructure, monitoring, jobs, etc. But in this article, I’ll show you how we use it for Jenkins. Yes, I mean the fully reproducible configuration for Jenkins, with infrastructure, plugins, credentials, jobs and many other things in the code. Besides, in this article you will find answers to the following questions:

• Has our Jenkins become more stable?
• Could we frequently change Jenkins’s and job configuration?
• Is updating Jenkins or plugins not a pain for us anymore?
• Have we managed all our changes in Jenkins?
• Have we managed to restore Jenkins after the accident quickly?

Let your builds always be dreamy

My name is Amet Umerov and I’m a DevOps Engineer at Preply.com. And let’s get started!

Intro

When we talk about DevOps tools, a CI/CD system is the first thing that comes to mind. We use Jenkins at Preply because we have hundreds of jobs every day, need to use many features that aren’t provided by other systems, or if provided, these features have stripped-down functionality.

We wanted to have Jenkins in code fully, with infrastructure, configuration, jobs, and plugins. Also, we had an experience with running it in Kubernetes but it didn’t fit us because of Jenkins architecture and our purposes.

That’s what I’m talking about

Building an underlying infrastructure for Jenkins

We use AWS and manage all our infrastructure using Terraform and other tools from HashiStack like Packer or Vault.

As I mentioned before, we tried to use Kubernetes for hosting Jenkins, and we’ve got issues with scaling PVCs, resources and not well-thought-out architecture for it.

Here, we use AWS resources, like EC2 instances, SSL certificates, Load Balancers, CloudFront distributions, etc. AMI is built by Packer, which integrates perfectly with Terraform and Vault.

How packer’s configuration looks like

Bootstrap file packer_bootstrap.sh contains all commands for pre-installed software in AMI. For instance, we can use Docker, docker-compose, and vaultenv or install a Datadog agent for monitoring.

Regarding the infrastructure for this AMI, we can use Terraform, CloudFormation, Pulumi or even Ansible. Here is one of the possible infrastructures on AWS for Jenkins.

Users come to Jenkins through the internal LB and GitHub webhooks through the public LB

We use Jenkins integration with GitHub, so we should have some Jenkins URLs available for GitHub through the external network. There are many possible solutions to implement it (IP whitelist, URL or token whitelist, etc) in our case, we do it in combination with Cloudfront allowed paths and token validation.

So after doing all these things, we already have a ready-made infrastructure with AMI, possibilities to monitor and access the corporate Vault for getting some credentials.

Using Docker for managing Jenkins and its plugins versions

Well, the next step is about Jenkins and its plugins. We’ve got many issues with upgrading Jenkins’s plugins, so the main point here is to pin versions for them.

And here is when Docker helps us. We can use the already pre-build Jenkins image and use it as a base image for our setup.

Here we install some additional packages for Job Builder, which we will use soon, also passing volumes for Jenkins and install plugins.

We can easily get a list of installed plugins by pasting this Groovy-code in https://our-jenkins-url/script and save it to plugins.txt file:

And finally, docker-compose configuration for running Jenkins in Docker (we also use vaultenv for passing credentials from Vault for docker-compose):

Keep in mind some Java parameters that could help us with garbage collection and resource limiting, it’s important. This article is extremely cool for tuning Jenkins.

And of course, we can run Jenkins locally with that and play with plugins or new Jenkins version. It’s really cool.

Now we have pure Jenkins with plugins, which we can run even locally and easily deploy on production. Let’s add more configuration for it.

Configuring Jenkins as a Code (JCasC) plugin for master’s configuration

There is Jenkins Configuration as Code (JCasC) plugin for storing configuration in a human-readable text format.

This plugin allows us to describe security configurations, credentials, plugin settings, nodes, views and a lot of other things.

The YAML configuration file is split into 5 blocks:

• credentials (system credentials description)
• jenkins (authorization and clouds settings, global properties, nodes, security realms, and views)
• security (global security configurations, like script approvals)
• tool (configuration for external tools, like git, allure, etc)
• unclassified (other configuration, for example, Slack integration)

We can import configuration from the existing Jenkins installation

There is support for different credential providers for managing credentials, but we can use environment variables also.

We also use Amazon EC2 Plugin for bootstrapping agents on AWS, and its configuration also could be managed with this plugin. Matrix-based authorization allows us to manage users' permissions from code.

There are many cool things that we use. If we have a flow for testing Jenkins changes locally, we can find and fix bugs before they come to production.

So we set up a reproducible Jenkins configuration, and, last but not the least, are our jobs.

Implementing Job Builder for freestyle projects

When we talk about freestyle jobs, there are a few different ways to create them in Jenkins:

• using GUI (the easiest way, just click it)
• directly using REST API
• using plugins like Job DSL or JJB wrapper

Jenkins Job Builder (JJB) allows us to configure jobs in a human-readable text format (YAML or JSON). And it’s really comfortable to manage jobs with SCM. Basically, we can set up a CI/CD process for our CI/CD tool using JJB.

Simplified structure for jobs in a git repo

Here, we can describe job definition in the file Job1.yaml, and job steps in the script (for example job1.sh).

How the simple job looks like

Here is an example of the configuration file:

It could be easily launched with the jenkins-jobs update command

Of course, our Jenkins user should have permission to create and configure jobs. We just need to run a one seed job on the master node for importing all configuration from JJB to Jenkins.

JJB is not a silver bullet because some unpopular plugins aren’t still supported. But it’s a really flexible tool for keeping jobs in the code. Besides, it can be configured with macroses.

Summary

Now that we’ve provided an overview of the ‘Everything as a Code’ paradigm and how we use it for Jenkins, we can go back to the questions we set out at the beginning of the article. Have you figured the answer? Probably, it’s been easy: the answer is “yes” to each of the 5 questions.

Since we only wanted to share our experience with you, we didn’t go deeper into the parameters configuration or Jenkins Best Practices in this article.

Subscribe to the Preply Engineering Blog for more interesting articles about engineering at Preply. Stay tuned!