Automatically Generating Jenkins Jobs

by Jim Graf

A client approached us looking to better automate their disaster recovery process. Their goal was to develop a system that would allow them to recover their entire development and production environment in the event that the only surviving component of their infrastructure was their Git repository and their Amazon Web Services account. The first step to recovering their enterprise would require them to build their application artifacts and customized Ansible scripts using the Jenkins build automation tool.

Jenkins is an incredibly powerful tool for DevOps that allows us to execute clean and consistent builds and deployments that mitigate the risks of a manual execution plan. But creating the jobs within Jenkins to do this work can be labor intensive, manually updating those jobs is often error-prone, and tracking changes to those jobs is tedious at best.

Thankfully there are a couple of tools available to help us with these issues: Jenkins Job Builder and Jenkin’s Job DSL Plugin

Jenkins Job Builder

Jenkins Job Builder is a command-line utility that will create Jenkins jobs based upon YAML configurations. It can be installed by running the following script:

Then configure the installation by creating /etc/jenkins_jobs/jenkins_jobs.ini and filling in the path to the Jenkins instance and its respective credentials.

Alternatively this configuration can be created elsewhere and passed to Jenkins Job Builder at runtime using the --conf parameter. Now create a job definition YAML file:

And execute Jenkins Job Builder to create a job:

Executing this script will create a job named my_hello_world_job in the Jenkins instance. The official documentation for the Jenkins Job Builder API can be found here. The nice thing about this tool is that it can be used in conjunction with source control to manage job configurations and their changes over time and changes to jobs are implemented consistently.

One downside of Jenkins Job Builder is that there are several Jenkins plugins it doesn’t support yet and there’s no way to add support for them without modifying Jenkins Job Builder itself. Thankfully we can work around this limitation because I recently added support for Jenkin’s Job DSL plugin to Jenkins Job Builder.

Jenkin’s Job DSL Plugin

Jenkin’s Job DSL plugin provides a Groovy-based DSL for creating Jenkins jobs from within other Jenkins jobs. Its primary advantage over Jenkins Job Builder is that it provides direct access to a job’s configuration XML so that it can be manipulated directly to supplement any functionality that might be lacking in the DSL.

To create a “Hello World” job using the Job DSL plugin:

• Go to Jenkin’s Plugin Manager and install the Job DSL plugin in the Jenkins instance. Jenkins may need to be restarted.
• Create a new free-style Jenkins job

• Go to the job’s Build section and add a ‘Process Job DSLs’ build step

• Click ‘Use the provided DSL script’ and add this script:

• Save the job and run it

A job should be created called ‘hello-world’.

But this is just the beginning of the Job DSL plugin’s capabilities. Let’s say we want to do something more complicated like create a job for every repo in a Github organization. We can leverage the Jenkins Credentials plugin to store Github credentials and call the Github API to get repo information:

Combining Jenkins Job Builder and the Job DSL Plugin

We can now combine these two tools so that we can source control the Job DSL’s Groovy script, without being hindered by Jenkins Job Builder’s more restrictive API. We simply store the DSL script in a file and utilize the following Jenkins Job Builder YAML file:

This script will create a job called ‘job-generator’ that will execute the DSL script in hello-world.dsl.

End Result

You now have the means to source control your Jenkins job configurations, and generate your jobs using the power and flexibility of Groovy in the Job DSL plugin. Even if the Job DSL doesn’t support a Jenkins plugin you’re using, you can still configure it without having to modify the Job DSL itself.

About the Author

Jim Graf is a Solution Architect in Slalom Chicago’s Cross-Market Delivery Network. Jim leverages his significant experience in front-end and back-end development to better understand how DevOps can increase throughput. Find him on twitter at @jamesggraf.