Testing Jenkins Shared Libraries

At Disney Streaming Services many of our teams use Jenkins for their build and deploy functions and many common features are required across teams and projects. To reduce code duplication and share learnings across teams we use shared libraries.

One of the most difficult aspects of developing Jenkins shared libraries is testing. It can be tedious and time consuming. There is no built-in support for automated testing, and the Jenkins Handbook even suggests doing manual testing via the Jenkins web interface. There is a better way.

Setting Up

Before you can start writing tests, you need to add a few things to your Jenkins library project:

• A JUnit library, Jenkins Pipeline Unit, for simulating key parts of the Jenkins pipeline engine
• A build tool, for compiling and running tests
• A build script, for specifying dependencies and other project configuration

I recommend Gradle as the build tool, because it’s fast, flexible, and very Groovy oriented. After installing Gradle, add the Gradle wrapper to your project directory:

cd <project dir>
gradle wrapper

To add the build script and jenkins-pipeline-unit to your project, put the following in a file named build.gradle:

plugins {
    id "groovy"
}

group = "<package>"   // example: com.disney.dss.dpe
version = "<version>" // example: 1.0.0

repositories {
    mavenCentral()
}

dependencies {
    compile "org.codehaus.groovy:groovy-all:2.5.0"
    testCompile "com.lesfurets:jenkins-pipeline-unit:1.1"
    testCompile "junit:junit:4.12"
}

Writing Tests

Tests that use Jenkins Pipeline Unit are written as JUnit test classes that extend com.lesfurets.jekins.unit.BasePipelineTest. This class provides properties and methods for accessing the script variable binding, loading scripts, creating mock steps, and simulating other features of Jenkins pipelines.

Let’s say your Jenkins library includes the step toAlphanumeric (vars/toAlphanumeric.groovy):

/**
 * toAlphanumeric converts text to strictly alphanumeric form.
 * Example: a_B-c.1 -> abc1
 */
def call(Map opts = [:]) {
    opts.text.toLowerCase().replaceAll("[^a-z0-9]", "")
}

To create a test for toAlphanumeric, create a class named ToAlphanumericTest (src/test/ToAlphanumericTest.groovy) that has methods to load vars/toAlphanumeric.groovy, call toAlphanumeric, and check the result. Here’s the code:

import org.junit.*
import com.lesfurets.jenkins.unit.*
import static groovy.test.GroovyAssert.*

class ToAlphanumericTest extends BasePipelineTest {
    def toAlphanumeric

    @Before
    void setUp() {
        super.setUp()
        // load toAlphanumeric
        toAlphanumeric = loadScript("vars/toAlphanumeric.groovy")
    }

    @Test
    void testCall() {
        // call toAlphanumeric and check result
        def result = toAlphanumeric(text: "a_B-c.1")
        assertEquals "result:", "abc1", result
    }
}

Running Tests

To run the tests, run the Gradle test task in the project directory:

cd <project dir>
./gradlew test

Gradle will compile the Groovy files in src/test/groovy, scan the compiled classes, then run the tests that it finds. If all goes well, you should see output like this:

BUILD SUCCESSFUL in 3s
2 actionable tasks: 2 executed

Knowing that the build was successful is good. Knowing which tests ran and the result of each is even better. To get this information, and a summary, from the test task, add the following to build.gradle:

test {
    // delete old test reports
    dependsOn cleanTest

    // don't stop if tests fail
    ignoreFailures = true

    // minimize logging
    testLogging.maxGranularity = 0

    // show stdout from tests
    onOutput { dest, event -> print event.message }

    // show test results
    def results = []
    afterTest { desc, result ->
        println "${desc.className.split("\\.")[-1]}: " +
            "${desc.name}: ${result.resultType}"
    }
    afterSuite { desc, result ->
       if (desc.className) { results << result }
    }

    // show summary
    doLast {
        println "Tests: ${results.sum { it.testCount }}" +
            ", Failures: ${results.sum { it.failedTestCount }}" +
            ", Errors: ${results.sum { it.exceptions.size() }}" + 
            ", Skipped: ${results.sum { it.skippedTestCount }}" 
    }
}

If you run the Gradle test task again and all goes well, you should see output like this:

> Task :test
ToAlphanumericTest: testCall: SUCCESS
Tests: 1, Failures: 0, Errors: 0, Skipped: 0
BUILD SUCCESSFUL in 1s
3 actionable tasks: 2 executed, 1 up-to-date

Checking Test Failures

If you run the tests and one or more of them fails, you should see output like this:

> Task :test
ToAlphanumericTest: testCall: FAILURE
1 test completed, 1 failed
There were failing tests. See the report at: file:///Users/trose/Work/JenkinsLibTesting/build/reports/tests/test/index.html
Tests: 1, Failures: 1, Errors: 0, Skipped: 0

To investigate the test failures, open the test report with your favorite web browser. From the report you can get a complete error message and a stacktrace for each failed test.

Mocking Steps

Let’s say you add the step getCommitHash (vars/getCommitHash.groovy):

/**
 * getCommitHash returns the current git commit hash.
 */
def call(Map opts = [:]) {
    sh "git rev-parse HEAD"
}

Calling getCommitHash in a test will generate a failure, because sh is only available when running in a real Jenkins pipeline. The solution is to mock sh before calling getCommitHashin the test (src/test/GetCommitHashTest.groovy):

import org.junit.*
import com.lesfurets.jenkins.unit.*
import static groovy.test.GroovyAssert.*

class GetCommitHashTest extends BasePipelineTest {
    def getCommitHash

    @Before
    void setUp() {
        super.setUp()
        // load getCommitHash
        getCommitHash = loadScript("vars/getCommitHash.groovy")
    }

    @Test
    void testCall() {
        def hash = "9ee0fbdd081d0fa9e9d40dd904309be391e0fb2b"

        // create mock sh step
        helper.registerAllowedMethod("sh", [ String ]) { hash }

        // call getCommitHash and check result
        def result = getCommitHash()
        assertEquals "result:", hash, result
    }
}

When you run the tests, you should see output like this:

> Task :test
ToAlphanumericTest: testCall: SUCCESS
GetCommitHashTest: testCall: SUCCESS
Tests: 2, Failures: 0, Errors: 0, Skipped: 0
BUILD SUCCESSFUL in 3s
3 actionable tasks: 3 executed

Pro Tip: To run a single test class, or even a single test, run Gradle with the --tests option:

./gradlew test --tests GetCommitHashTest
./gradlew test --tests GetCommitHashTest.testCall

Testing Aggregate Steps

As a Jenkins library grows, eventually some steps will call other steps, which in turn call other steps, going on for multiple levels. There are two techniques for testing these aggregate steps:

• Isolation — mocking all steps called by the step being tested
• Integration — mocking only steps that are not in the library

Let’s say you add the step getBuildTag, which calls getCommitHash (src/getBuildTag.groovy):

/**
 * getBuildTag generates a build tag using version, date, and
 * short git hash.
 */
def call(Map opts = [:]) {
    [ opts.version, new Date().format('yyyyMMddHHmmss'),
        getCommitHash()[0..6] ].join("-")
}

Using the isolation technique, GetBuildTagTest will have the same structure as GetCommitHashTest, with a mock forgetCommitHash instead of sh. Isolation makes the test simple, but not robust:

• Tests for any other steps that call getCommitHash must mock it, introducing more code for creating mocks and a greater risk of wrong mocks
• Interactions between getCommitHash and steps being tested are limited to the interface of getCommitHash, increasing the risk of missed scenarios

Integration makes testing aggregate steps both simple and robust:

• Fewer mocks are needed, reducing code and decreasing the risk of wrong mocks
• Step interactions include interfaces and behavior, decreasing the risk of missed scenarios

Using the integration technique, GetBuildTagTest (src/test/GetBuildTagTest.groovy) looks like this:

import org.junit.*
import static groovy.test.GroovyAssert.*

class GetBuildTagTest extends BaseTest {
    def getBuildTag

    @Before
    void setUp() {
        super.setUp()
        // load getBuildTag
        getBuildTag = loadScript("vars/getBuildTag.groovy")
    }

    @Test
    void testCall() {
        def hash = "9ee0fbdd081d0fa9e9d40dd904309be391e0fb2b"
        def timestamp = new Date().format("yyyyMMddHHmmss")
        def expected = "1\\.0\\.0\\-" +
            "${timestamp[0..-3]}[0-5][0-9]\\-${hash[0..6]}"

        // create mock sh step
        helper.registerAllowedMethod("sh", [ String ]) { hash }

        // call getBuildTag and check result
        def result = getBuildTag(version: "1.0.0")
        assertTrue "result: not /$expected/",
            result as String ==~ expected
    }
}

Notice that GetBuildTagTest extends BaseTest instead of BasePipelineTest. BaseTest is introduced as a way to load all the steps from the vars directory, making them callable by getBuildTag. Here’s the code (src/test/BaseTest.groovy):

import org.junit.*
import com.lesfurets.jenkins.unit.*
import static groovy.test.GroovyAssert.*

class BaseTest extends BasePipelineTest {
    @Before
    void setUp() {
        super.setUp()

        // load all steps from vars directory
        new File("vars").eachFile { file ->
            def name = file.name.replace(".groovy", "")

            // register step with no args
            // example: toAlphanumeric()
            helper.registerAllowedMethod(name, []) { ->
                loadScript(file.path)()
            }

            // register step with Map arg
            // example: toAlphanumeric(text: "a")
            helper.registerAllowedMethod(name, [ Map ]) { opts ->
                loadScript(file.path)(opts)
            }
        }
    }
}

Photo by Nicolas Thomas on Unsplash