Jenkins Pipeline Management Made Easy Using DSL

At Rupeek, the Continuous Integration and Continuous Delivery (CICD) system is heavily dependent on Jenkins. The previous model followed at Rupeek had each project having its own Jenkins declarative pipeline created through UI. While this model had its benefits — you can quickly see how you can run, test, and use to build and deploy projects — it had its drawbacks as well. These limitations were -

• Updating pipeline took a long time as DevOps had to update pipelines created for each project
• No central repository that had an overview of how a particular set of applications was built, since we were creating Jenkins pipelines from UI

To overcome these issues, we decided to revise how we manage CICD solutions. The aim was for all the teams to follow the same business process, thus standardizing CICD. Therefore, instead of making each team have their own Jenkins pipeline, DevOps as a central team took responsibility for orchestrating the pipeline development for all the applications.

Initially, we tried to move the Jenkins pipeline code that was hardcoded in the UI to a central config repository. While this gave us the flexibility to some extent on rolling out new updates across all projects, it still took quite some manual effort to update the changes.

So we took a step back, divided our problem statement into two parts, and found a solution for each of these problems. The proposed solutions were -

• Develop standard Jenkins shared libraries that can be adopted across all projects.
• Use DSL Plugins to create Pipelines and update them with minimal manual intervention.

Here is how we actualized the proposed solutions.

Rupeek CICD template using Jenkins Shared Libraries:

For every task we perform as a part of the pipeline, be it using a plugin or a simple script, we started developing a library called ‘Global Variables’ from the Jenkins declarative pipeline. Below is an example of a pipeline template using this shared library.

The above pipeline builds the application, runs unit tests, as well as code quality sonar analysis, and deploys the application to several environments. It uses Global Variables such as “build, codeQuality, deploy” provided within a Shared Library. The deploy method copies the built artifact to a remote server and starts the application.

If you’re not yet familiar with Shared Libraries, I’d recommend going through this document.

Pipeline creation/updating automation using DSL

To eliminate manual intervention in creating and updating hundreds of pipelines, we leveraged a DSL plugin — Job DSL — to automate the whole Jenkins pipeline life cycle management. Here is the workflow for automated creation/updating of a job with just a valid yaml file as an input.

Job DSL is a Domain Specific Language written in Groovy. After Installing the Job DSL plugin, you have to create a Seed job and then write a Groovy script that uses this Job DSL. You can find the full API in the DSL API Viewer. Please find the sample code snippet:

After pushing the config.yaml file to the Config repo, it will automatically trigger the seed job, which starts scanning the config repo and fetches all the yaml files. Based on the properties specified on the config files, the jobs get created automatically. DSL API allows you to create different types of jobs — Views, Folders, Queues and many more useful things — and is currently developing its capabilities. Another advantage is that you can use the full programming power of Groovy when writing seed scripts. You can make HTTP requests, parse JSON, YAML, and XML files and pretty much do anything you would do with Java.

Conclusion

Using Job DSL plugin for life cycle management of pipelines and Jenkins shared libraries for pipeline templating helped Rupeek -

• Reduce the workload more than expected.
• Reduce complexity and manual intervention.
• Manage all jobs up-to-date across the organization.
• Developers migrate their projects whenever.
• Develop templates for every new stack or business process, which could be used for all identical applications as well.

For DevOps at Rupeek, Job DSL and Shared libraries is a big win as we have drastically reduced the maintainability of CICD for all the projects.

Note: This blog was originally written by Kalyan Killari.