Jenkins Pipeline Archetypes

As Jenkins pipeline usage increases in a development organization, pipeline scripts tend to become similar in structure and functionality. Creating a library of pipeline archetypes reduces pipeline code duplication and increases code re-use across teams.

Concept

At Disney Streaming Services, my team maintains a Jenkins library, Libjenkins, that is available to any team in our organization. It includes not only commonly used steps but also pre-built, configurable pipelines. We call these archetypes. They contain stages and steps common to various project types, allowing pipeline scripts for those project types to be very small. They also delegate much of the build logic to the project build tool, allowing the pipelines to be lean.

Structure

One of the archetypes in Libjenkins is DockerApp. It is used for building, testing, and publishing applications packaged as Docker images. It checks out the source files, builds the application and image, runs the tests, then tags and publishes the image. The application can be written in any programming language.

DockerApp (src/archetypes/DockerApp.groovy) has the following structure:

def call(Map opts = [:]) {
   node {
      try {
         def image = opts.image ?: "..."
         // ...
         inside(registry: opts.registry, image: image) {
            stage("Checkout") {
               checkoutScm()
               if (isRelease(opts)) {
                  git.setIdentity()
                  // ... run build tool to apply git tag ...
               }
            }
            stage("Build") {
               // ... run build tool to build app and image ...
            }
            stage(name: "Test", skipWhen: opts.excludeTest) {
               // ... run build tool to run tests ...
               checkTestResults()
            }
            stage(name: "Publish", skipWhen: opts.excludePublish) {
               artifactory.withCredentials(credentialsId:
                     opts.artifactoryCredentialsId) {
                  // ... run build tool to push versioned image ...
                  if (env.BRANCH_NAME == "master") {
                     // ... run build tool to tag 'latest' ...
                     // ... run build tool to push 'latest' ...
                  }
               }
               if (isRelease(opts)) {
                  git.pushWithCredentials args: "--tags",
                     credentialsId: opts.credentialsId
               }
            }
         }
         // ... send success notification ...
      } catch (Exception e) {
         // ... send failure notification ...
      }
   }
}

Here are some things to note:

• The method call(Map opts = [:]) wraps the pipeline code, node { ... }. This makes the pipeline callable like any pipeline step (with one caveat, described below).
• The parameter opts allows the user to provide configuration options, such as excludeTest, which can be used to skip the Test stage.
• The inside step runs the given block (the stages) in a Docker container. The user can specify the Docker image and registry.
• Some of the stages have options name and skipWhen. This is possible because Libjenkins includes a custom implementation of the stage step.

Steps

Custom pipeline steps are located in the vars directory. They get loaded as global variables that can be called like methods.

The inside step (vars/inside.groovy) looks like this:

def call(Map opts = [:], Closure block) {
   // ...
   if (!opts.image) { error "missing image option" }
   def registry = opts.registry ?: env.DOCKER_REGISTRY
   def runParams = [ ... ]
   def image = docker.image("$registry/${opts.image}" as String)
   image.pull()
   image.inside(runParams, block)
}

The stage step (vars/stage.groovy) looks like this:

// custom implementation: stage(name: ...)
def call(Map opts = [:], Closure block) {
   // ...
   if (!opts.name) { error "missing name option" }
   if (!opts.skipWhen) { return steps.stage(opts.name, block) }
}
// pass through: stage("<name>")
def call(String name, Closure block) {
   // ...
   steps.stage(name, block)
}

Usage

Because it is included in the src directory, DockerApp gets exposed as a class. Using it is like using a pipeline step, with the caveat that it must be instantiated before it can be called. Configuration options (registry, image, excludeTest, excludePublish, etc.), are specified as <key>: <value> pairs.

Here’s an example pipeline script (Jenkinsfile):

@Library("libjenkins2") _
new archetypes.DockerApp().call()

Here’s another example, with configuration options:

@Library("libjenkins2@2.29.2") _
new archetypes.DockerApp().call registry: "internal",
   image: "trose-build", excludeTest: true, excludePublish: true

(You can also do new archetypes.DockerApp()(...), but I think the double parentheses look like a mistake.)

Testing

I would be remiss if I didn’t talk about testing. All of the archetypes in Libjenkins are fully tested. For unit/integration testing, we use the JenkinsPipelineUnit framework and some custom code. You can read about that in my article Testing Jenkins Shared Libraries. For functional/end-to-end testing, we use Testcontainers and a Docker container that includes Jenkinsfile Runner. I hope to write an article about that in the future.

Last Places Factory Old — Free photo on Pixabay