Stop copy/pasting your pipelines

CI/CD logic publishing

Introduction

We usually try to avoid source code duplication when developing applications. This is often resolved via package publishing, allowing the developers to share bits of useful code without directly duplicating it.

However, when it comes to CI/CD scripts, it’s not uncommon to see nearly identical blocks of configurations copy/pasted all over the various components of a given project. This creates an unnecessary maintenance overhead and makes it tedious to introduce even the smallest of adjustments across the stack.

Well, no more! This article will provide a quick overview of how you can start sharing and reusing your pipeline scripts in a similar manner to how you reuse your code libraries.

Scope

In this article, we’ll focus on Azure Pipelines CI/CD provider specifically; however, most of the other providers have similar offerings.

Additionally, we will not be discussing Tasks, as those add substantial overhead to publishing and are not designed to be used as private entities. Instead, we’ll have a look at Templates.

Overview

Templates can be approached in two ways — inheritance or composition. In this article, we’ll focus on the latter.

Composing azure templates allows us to dynamically include multiple script fragments into our main script from various external files. The best thing about it is that those scripts can also come from different repositories, allowing for maven/npm like mini-repositories to host various fragments of our script with little to no setup overhead.

To showcase this feature, we’ll be setting up a mock business case CI/CD for a micro-services based system. Here’s an overview:

• The hypothetical system contains multiple node.js based REST APIs
• There are multiple web SPAs built with node.js
• There’s a single npm library that all the projects depend on containing various utilities and domain model

To accomplish the above, the pipeline should normally do various checks and reports, before deciding if it should produce some deployable output:

• Run unit tests
• Report unit test coverage & success/failure counts
• Run SonarQube scan and publish/report the results
• Lint & build the project
• Publish the deployable

Traditional Way

Typically, we could achieve the above with a set of pipeline scripts, copied over to the projects of similar nature where needed. Here’s one way those could look like.

Replace Publish stage for library and web app to achieve the same (will omit this as we’ll cover a modular way to solve them).

As you can see, it does a lot of stuff but is quite lengthy and complex -> hard to maintain. Add duplication element to the mix, and it soon becomes a nightmare to make even the slightest of changes.

Especially since most of the steps are mirror images between even different categories of the deployable (web app, library, backend)… Let’s see how it can be improved with Templates!

Introducing Templates

Let’s split up the above file into actions and then see how we can recompose them back to achieve the same result without duplication of the entire script across multiple projects.

Build Stage

In the build stage, since all the system is node.js based, the actions are identical regardless of the deployable type. Here’s a summary:

• Setup private npm repository authentication
• Install dependencies
• Run unit tests
• Report unit test results and code coverage
• Run SonarQube analysis
• Report SonarQube results
• Build the project
• Publish build output as job artefact

Since most of the step definitions for these actions are either static or can be configured using only predefined pipeline variables, abstraction becomes as simple as just moving the scripts to a separate file.

The above script(s) introduces a few parameters for flexibility and uses them for:

• [lines 29 & 61] Conditional insertion of other template files (referencing to file by a relative path from the parent template file)
• Configuration propagation to various tasks

Also, it introduces a compile-time placeholder pattern ${{...}} . This is different from the usual $(...) runtime placeholder as it’s resolved much earlier and has access to different parameter/function scopes. This distinction allows using compile-time placeholders to build a runtime reference to a variable. e.g.: $(${{parameters.myVarName}}) which will be resolved to $(<value of the myVarName parameter>) and then expanded at runtime to contain the value under the variable. Additionally, compile-time placeholders allow the conditional insertion of scripts (as seen on lines 29 & 61) as well as iterating over objects and arrays in parameters (will be showcased in just a bit).

To recap, we just have:

• Extracted the build job into a reusable template
• Introduced few template parameters to increase flexibility
• Further extracted 2 utilities into their own templates
• Referenced them by their local relative path to compose a build template

Publish Stage

In the publish stage, things get a bit more interesting, as we’re now getting 2 branches to support depending on what kind of deployable is expected. For web app and backend services, the pipeline should build and publish docker images. However, for the shared library, an npm package is expected. We also do not want to have unrelated steps displayed if they’ll not be relevant for a given branch. We also are trying to avoid step duplication between the branches. Additionally, for our backend services, we want to tag docker image as latest only on master branch builds.

To achieve that, we’ll build each branch as a separate template and then compose them both into another template that resolves a correct branch depending on a parameter.

Here’s a simple Docker build/publishing template that optionally uses the output of the previous job to build an image and publish it to jFrog

Few things to note here:

• The template builds a Docker image repository path from git repository name, optional group parameter & branch name. Then uses build number as a tag
• It uses object iteration to dynamically generate an arbitrary number of steps from the object’s key-value pairs
• Optionally downloads an artefact from another job
• It also cleans up the built name to be docker-compliant
• Optionally, tags the image as latest as well
• Dynamically creates two new variables in the pipeline to store generated docker repository and tag names, accessible to all following steps in the job

Since docker repository & path generation might be useful in different contexts, it’s been extracted in its own, fully independent, template.

Next, let’s create a docker publishing template

Nothing to fancy with this one, except that it allows pushing multiple tags with the same invocation. This is achieved via an array iteration feature of azure templates.

With that done, we now have all the pieces required to publish our Docker-based deployable.

For npm package publishing, things can get a bit more complicated, as there’s a lot of neat features we can use to enrich our experience. Here is what we are aiming for:

• Login to jFrog private npm repository
• Bump package version if on the master branch
• Push back updated version to the repository
• Avoid circular builds
• Publish unique version if not on the master, but skip version pushback
• Tag git commit that produced the version with the link to the build that produced it

Here’s the final template

As you can see, it’s quite complex and bloated, but only because it’s aiming to support both, npm & yarn package managers as well as monorepos. It could be greatly reduced if only one of them would be chosen. Anyways, here’s what it does:

• Ensures the git credentials that were used to fetch the repository are persisted and available during the run
• Sets up required git user metadata to enable full git flows
• Authenticates the runner to jFrog npm repository (we could’ve used npmAuth template from before here, however, this particular way also enforces that the system is working ONLY with jFrog npm and not the npmjs.com)
• Optionally downloads artefacts from previous jobs
• Bumps the build version (further discussed below)
• Published the package to jFrog npm

Since most of the template’s heavy lifting is happening in version bumping step, we’ll discuss it separately. First off, it tries to detect if the head commit is already a version bump and avoids bumping it further (unless the build was started manually). Otherwise, it proceeds with one of the two paths — if the build is on the master branch, it does a full patch version bumping and pushes to git, if not, it just bumps pre version with build number as preid. Main pressure point here is the push back to git, as it needs to address various potential issues:

• If the project is a monorepo, it’s likely multiple unrelated directories could’ve been changed at the same time. Also, it’s likely that those changes triggered multiple parallel builds, thus creating a race condition. Therefore it’s crucial to pull potential parallel git changes before proceeding with any other git actions as by the time this template is called, git history might’ve been updated already.
• To avoid circular builds (a build making changes to the repository and triggering itself in an endless cycle), we must ensure that all the commits coming from the pipeline have ci skip in the header message. This suppresses further pipeline from triggering.

Having both publishing branches done, we can compose them together in a composite template with the build, and both publishings branched combined.

It has quite a few parameters; however, most of them are just propagating lower-level parameters. Key points here are:

• Appropriate publish template is conditionally selected at compile-time via isLib parameter
• extraPublishJobs & extraBuildJobs parameters are exposed to allow the consumers to add their own jobs in each respective stage. Likewise, one could add even more of these jobList parameters to spread them out in various stages and in between jobs throughout the pipeline. This allows for quite safe future expansions.

The Big Finale

So far we’ve only been using local templates, meaning they have to be on the same repository to be usable. While this is nice to help with abstracting and grouping related steps in their own standalone modules, it still does not solve the key issue that started this whole article — duplication.

— Enter Remote Templates —

Simply put, remote templates are exactly like the ones we’ve declared and used so far, except hosted on a separate (remote) repository. While the usage syntax is pretty much the same, there are few things one needs to do beforehand to get them working. Once again, here’s the final outputs for docker & npm variants consuming our root template to discuss

Both of these templates can be safely copy/pasted (I know, can’t fully avoid that just yet) into any given project and will work out of the box. It assumes that Jfrog-access variable group has JFROG_TOKEN & JFROG_USER variables declared and also that BitBucket-service-connection has read access to the mycompany/azure-templates repository hosting our templates and write access to any npm library repository.

Few things to note:

• [line 16] We’re declaring additional repository (other than implicit source repository) to be used in our build and tagging it as templates
• [line 23] ref parameter is optional, but I strongly recommend hooking it up to a tag in the repository. This way, you can modify the templates at will without breaking depending builds and just move the tag to a different commit when you’re ready. Alternatively, you could hook your consumer to a commit hash or a specific branch. It defaults to master branch otherwise.
• [line 25] We’re referring to a file on the additional repository via @templates tag after the file path. The path is relative to the repository root.

That’s it. Now any adjustments we make in the templates will be reflected in ALL the consuming pipelines immediately after we move the tag v1.0 to a new commit. We have achieved modularity and a single source of truth, happy coding!