CI Pipeline Logic: Clauses vs Composition

Continuous Integration (CI) pipelines run in a variety of contexts. For example, a pipeline might be triggered by a pull request or a nightly build, and it might run for different environments like Production versus Staging.

Different contexts may require different tasks. The pipeline for a pull request (PR) doesn’t need to publish artifacts for deployment. However, a nightly build pipeline may very well publish the artifacts it creates.

One way to look at the difference between these two pipelines is to say that the pipelines are the same — the publish step just turns off if the context is “Pull Request”.

Clauses: A Customizable Pipeline

First, start with a template pipeline that contains every task for every context:

We don’t want to run every step in every context, so we annotate each step with the context(s) it’s supposed to run in:

Here’s what happens in the context of a Nightly build:

Since it’s a nightly build, the pipeline runs the steps marked “Nightly”.
Here’s what happens in the context of a PR build:

Because the “Run Nightly tests” and “Publish” steps aren’t marked for the PR context, they’re skipped. Effectively, the PR build pipeline looks like this:

In scenarios where pipelines are simple and different contexts still have similar builds, this approach is easy to follow. However, it can get out of control when complexity grows. Here are some factors that affect the context of a CI pipeline:

• Branch — master and develop are treated differently.
• Pipeline trigger — pull request vs nightly build vs official release vs manual trigger vs …
• Matrix build variables — compiler/library versions, which partition (e.g. running tests in parallel)

If many different contexts require their own pipeline customizations, it can become very difficult to look at a pipeline and tell what’s supposed to happen in each context.

It can get much more complicated.

Composition: A Pipeline for Each Context

Another way to look at the pipeline customization problem is to give each context its own pipeline. Rather than sharing a do-all pipeline that uses filters to turn its components on and of, simply write a separate pipeline for each context. We’re trading fewer, more complicated pipelines for more numerous, simpler pipelines.

Let’s apply this tradeoff to the final diagram in the previous section:

We made the pipelines much shorter, but now there are way more of them. We need to reduce the duplication.

Higher-order Functions

This is where higher-order functions can help us. A higher-order function is a function that operates on other functions. Since we’re talking about CI pipelines, a higher-order function is a pipeline that operates on other pipelines.

In our pipeline above, we want to customize which tests run, which kind of build is used, and whether or not we publish the artifacts — three holes to fill. That sounds like a higher-order function that takes three pipelines as arguments (pseudocode):

customPipeline(test, build, publish Pipeline) Pipeline {
    do clone     // pre-defined Pipeline
    do unit-test // pre-defined Pipeline
    do test
    do build
    do publish
}

Suppose we have these sub-pipelines defined:

noTest         // (customPipeline already includes unit tests)
nightlyTest    // tests for nightly builds
releaseTest    // tests for releases
normalBuild    // all versions but v1.8
specialBuild   // v1.8
noPublish      // don't publish anything (a no-op pipeline)
nightlyPublish // publish to the nightly-builds repository
releasePublish // publish to the official-releases repository

All these sub-pipelines correspond to boxes in the diagrams you saw before. Now we can define our CI pipeline like this:

build = v1.8 ? specialBuild : normalbuild
if pullRequest {
    do customPipeline(noTest, build, noPublish)
} else if nightly {
    publish = master ? nightlyPublish : noPublish
    do customPipeline(nightlyTest, build, publish)
} else if release {
    do customPipeline(releaseTest, build, releasePublish)
}

You can immediately see that a pullRequest uses the base template with no extra tests (noTest) and no publishing (noPublish). Compare this to the equivalent pseudocode for the do-all pipeline approach:

do clone
do unit-test
if nightly {
    do nightlyTest
}
if release {
    do releaseTest
}
if v1.8 {
    do specialBuild
}
if not v1.8 {
    do normalBuild
}
if nightly & master {
    do nightlyPublish
}
if release {
    do releasePublish
}

In order to see what the pullRequest pipeline looks like, you must look through the entire template — and remember each step you find along the way. This is still a simple example, so it’s doable, but imagine a project with more pipeline steps and a greater variety of pipeline contexts.

Comparison and Conclusion

For simple cases, it makes a lot of sense to use a single pipeline definition and skip steps based on the context of the pipeline run. However, as a project’s CI use cases grow more complex, this approach stops scaling. At that point, it makes sense to apply programming techniques like higher-order functions. A CI pipeline language should support both patterns.