Centrally Managing Jenkins Jobs With a Jenkins Template

As a company’s infrastructure grows so does the amount of jobs which are running in their Jenkins servers. Most of the time these jobs are very similar to each other with minor differences. As these jobs keep growing, the number of DSL files used to maintain these jobs also grows and It becomes a hassle to maintain these jobs individually. Take for example a typical build job used to build a docker image. The DSL file and the shell script used to create the Jenkins job is given below:

freeStyleJob('build-some-app-master-image') {
  description 'Build some-app docker images from master'
  scm {
      git {
          remote {
              url('git@github.com:acme/some_app.git')
              credentials('xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx')
              branch('master')
          }
          triggers {
              githubPush()
          }
      }
  }
  steps {
    shell(readFileFromWorkspace('build_app_master_image.sh'))
  }
}

#!/bin/bash
set -e

export AWS_DEFAULT_REGION="us-west-1"
export DOCKER_IMAGE="some-app"
export ECR_REPO="111111111111.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com"

docker build -t ${ECR_REPO}/${DOCKER_IMAGE}:latest .
docker push ${ECR_REPO}/${DOCKER_IMAGE}:latest

If the need arises to add an additional build step to this job, for example if there is a requirement to enable slack notifications for each build job failure, each DSL file for each application needs to be modified to support this change

A better way to create and maintain these files/jobs

The solution to this is to centrally maintain a single template which accepts parameters and creates multiple jobs according to the parameters this template receives.

Given below is the same build job that was shown above written in such a way to receive parameters so that it generates multiple build jobs.

An array is defined which contains the parameters for each application. This array is fed to a loop which traverses the array through a JSON parser.

When you run this DSL script on your Jenkins server it produces the build jobs that you defined in the array.

import groovy.json.JsonSlurper

def slurper = new groovy.json.JsonSlurper()

def application_list = slurper.parseText('[{"name":"app_one","organisation":"acme","app_repo":"app_one","branch":master","aws_account":123456789123","aws_region":ap-southeast-1","image_name":app_one"}, {"name":"app_two","organisation":"acme","app_repo":"app_two","branch":master","aws_account":123456789123","aws_region":ap-southeast-1","image_name":app_two"}]')

application_list.eachWithIndex { application, index ->
  freeStyleJob("build-master-image-${application.name}")
  {
    description 'Build ${application.name} docker image from master'
    scm {
      git {
        remote {          
          url('git@github.com:${application.organisation}/${application.app_repo}.git')
          credentials('xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx')
          branch('${application.branch}')
        }
        triggers {
          githubPush()
        }
      }
    }
    environmentVariables {
      envs(
        AWSACCOUNTID: "${application.aws_account}"
        AWSDEFAULTREGION: "${application.aws_region}"
        DOCKERIMAGE: "${application.image_name}"
      )
    }
    steps {
      shell(readFileFromWorkspace('build_app_master_image.sh'))
    }
  }
}

#!/bin/bash
set -e

export DOCKER_IMAGE="${DOCKERIMAGE}"
export ECR_REPO="${AWSACCOUNTID}.dkr.ecr.${AWSDEFAULTREGION}.amazonaws.com"

docker build -t ${ECR_REPO}/${DOCKER_IMAGE}:latest .
docker push ${ECR_REPO}/${DOCKER_IMAGE}:latest

This approach gives you a base to build increasingly complex templates by using the various capabilities and characteristics of the groovy language. To make it even more interesting this template can be configured to get the application configuration list from a remote api. In an environment which follows DevOps practices a pipeline can be created to automatically trigger the jobdsl build job when the application api is updated.

import groovy.json.JsonSlurper

def application_list = new JsonSlurper().parseText( new URL("https://application_list.acme.com/list").getText() )

application_list.eachWithIndex { application, index ->
 freeStyleJob("build-master-image-${application.name}")
 {
   description 'Build ${application.name} docker image from master'
   scm {
       git {
           remote {  
               url('git@github.com:${application.organisation}/${application.app_repo}.git')
               credentials('xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx')
               branch('${application.branch}')
           }
           triggers {
               githubPush()
           }
       }
   }
   environmentVariables {
     envs(
       AWSACCOUNTID: "${application.aws_account}"
       AWSDEFAULTREGION: "${application.aws_region}"
       DOCKERIMAGE: "${application.image_name}"
     )
   }
   steps {
     shell(readFileFromWorkspace('build_app_master_image.sh'))
   }
 }
}

In conclusion, as a company’s infrastructure grows, managing an increasing number of similar Jenkins jobs can become a significant challenge. To streamline this process and maintain efficiency, the solution proposed involves creating and maintaining a single template capable of generating multiple jobs based on parameters. This approach not only simplifies job management but also opens the door to automation, allowing job creation to align seamlessly with changes in application configurations through a remote API, in line with DevOps practices.