‘Jenkins a la Travis’’

tl;dr;

If you are working in a company that want to keep it’s Continuous Integration Server internally, probably, like us, you are using Jenkins. It’s robust, versatile and not at all complex to use. However it lacks job management usability at scale. When you have to deal with more than 10 jobs, it starts to be a boring and error prone task. What about using the same strategy that new CI tools are using? Like Travis for instance. Wouldn’t be fantastic if we had a file in our code repository that Jenkins could recognise and execute jobs accordingly?

At Mindera we use Jenkins like that. Using an open source tool called Jenkins Job Builder (JBB) and by “eating our own dog food”, we configure our jobs with a simple YAML file, that sits next to the project code in the version control. This enables us to manage jobs configurations not only for CI but also for CD with just a YAML file.

The immediate advantages are:

• No more point and click configuration;
• Auditable and saved configurations changes;
• Job configuration;
• Few lines of configuration sets up a full Continuous Delivery pipeline;
• Config reuse with help of JJB templates and macros.

Context

At Mindera we use Jenkins. We use it because it’s open source and have an enormous versatility related with the huge number of plugins. We could use others, but for some reason it don’t fit exactly with what we want. The CI’s services in the cloud, like Travis, have “the small big problem” of being paid, but also the problem that our code/artifacts are somewhere in a place we don’t control.

So we use Jenkins. But Jenkins is not perfect. The big issue that we found when using Jenkins massively, it’s the difficulty to maintain hundreds of jobs in a clean way. Imagine the work of updating, using the Jenkins web UI, 287 Jenkins jobs to start using Java 8 instead of 6, or change the path of their SCM from Github to Bitbucket.

Well, fortunately there are several solutions for these boring tasks, for example:

• Jenkins Job Builder (JJB)
• Job DSL plugin
• Configuration Slicing Plugin
• Configuration Management tools (Like Chef, Puppet, Ansible, etc)

All these alternatives have strong advantages, but we had to pick one, and we chose JJB for its simplicity and template/macros power.

JJB

Jenkins job builder is also an open source project. It enables the configuration of Jenkins Jobs using YAML or JSON, that are then converted to XML and pushed to Jenkins.

JJB example configuration in YAML:

- job:
    name: job-name
    project-type: freestyle
    description: 'Do not edit this job through the web!'
    quiet-period: 5
    node: slave-with-java
    logrotate:
      daysToKeep: 3
      numToKeep: 20
    scm:
      - git:
          url: https://example.com/project.git
    builders:
      - shell: mvn install
    publishers:
      - email:
          ecipients: some@email.com

Old days

When we first adopt JJB we used the one-repo-for-all approach. I mean, every project has a YAML configuration file with it’s own name, and that file was stored in a big repo with all JJB Yaml files. Something like this:

├── dev
│ ├──jenkins.conf
│ ├──macros.yml
│ ├──utils.yml
│ ├──project_1.yml
│ ├──project_2.yml
│ ├──project_3.yml
│ ├──(....)
│ └──project_38.yml
├── prod
│ ├──jenkins.conf
│ ├──macros.yml
│ ├──utils.yml
│ ├──project_1.yml
│ ├──project_2.yml
│ ├──project_3.yml
│ ├──(....)
│ └──project_38.yml
└── test_dev.sh

Just by using JJB this way, make us earn lifetime. We used a shared macros file where we added all templates and macros that can be used by the projects. Despite this have worked for us, we got a big mess in version control history. Also, the developers continued to not have ownership on those jobs configuration, because those weren’t on their projects but in a common repository.

Travis

In the rare case you don’t know what is Travis (or other similar service), you should google it. Travis is a CI cloud service which is free for open source project stored in Github. Travis have a lot of fantastic features, but the one that I will talk about is the .travis.yml file. For you starting using Travis in your github project you just need to grant Travis permissions and add a simple file like this on your project root:

language: java
install: mvn install -DskipTests=true
jdk:
  - openjdk7
notifications:
  email:
    - some@email.com

Very easy right? Now everytime that some push is made to github, travis will run maven install skipping the tests, using openjdk7 and emailing success of failure notifications for some@email.com. No web UI intervention. Just a human readable file making all the configuration. Just like JJB.

Jenkins + JJB + Travis

So, why not replicate the process to Jenkins? Let’s add an configuration file to the project directory (let’s call it “jobs.yaml”) file, with the CI tasks and even the CD tasks. Now we just need Jenkins to be aware of it. How?

Our strategy is very simple: each project (let’s say we have a “Project_x” project), beyond their jobs for building, testing and deploying, has also an “jjb” job (that we call “Project_x-jjb”). This job is responsable to read the yaml file and (re)load the configuration of the jobs. Then it triggers the first job of the pipeline (in our case the “project_X-build” job.). From then on, the jobs triggers the next job in the line in order to do the project pipeline.

Because an image worth more than a thousand words…

Do we need to create the Project_x-jjb by ourselves? Of course not! That’s the magic. We have a project called “meta-jjb-root” having its own job and repo containing macros and templates for JJB but also three crucial files for all this process:

• builder.sh: responsable for updating the macros and templates in the Jenkins Master so it can be used by all the projects-jjb jobs, execute the “meta_repositories_jobscreator.py” script and run jenkins job builder against the output of ‘meta_repositories_jobscreator.py’.
• meta_repositories_jobscreator.py: Will generate a yaml file with the job configuration for each repository (project) listed in “repository_list” so we can automagically create the “projects-jjb” jobs.
• repository_list.yml: when you want that our project to use this magic, after adding the “jobs.yaml” to your project, you need to add an entry in this list with your SCM repository link.

To make this more beautiful we have some tricks:

• The “project_x-jjb” job will only execute if there is any change on the “jobs.yaml” file;
• Every other jobs for “project_x” should not execute if there is any change on the “jobs.yaml” file;
• We can also create views in the same way! Adding the file “views.yaml” and using the open source project jenkins-view-builder.

Try it yourself!

Finally, tired of theory, you probably want to see this in practice. You can test by yourself just cloning the following projects:

• cfgmgmtcamp_meta-jjb-root — Where you can find the “builder.sh”, “meta_repositories_jobcreator.py” and “repository_list.yml” files that I referenced before. There is also a Vagrantfile, so you just have to vagrant up and you have an Jenkins supporting “Jenkins a la Travis” process running in ubuntu.
• cfgmgmtcamp_jenkins_a_la_travis where there is the folder “.jenkins” where you can find the “jobs.yaml” and “views.yaml” files.

Reference links:

• ‘Jenkins à la Travis’ talk at cfgmgmtcamp.eu 2016
• jenkins-job-builder homepage
• jenkins-view-builder repository
• Jenkins homepage
• Travis homepage

João Cravo