Simple, Repeatable & Free: An Open Source Software Delivery Pipeline (Part II)

Dan Stieglitz
Jan 13 · 6 min read

In the last installment of this series (https://medium.com/stainless-ai/simple-repeatable-free-an-open-source-software-delivery-pipeline-part-i-4ae89e7a6478), we

  • Reviewed some DevOps concepts;
  • Discussed semantic versioning for our software packages;
  • Introduced a custom Gitflow and;
  • Introduced a Jenkins shared library for automating semantic versioning of our builds

In this installment we’re going to take a closer look at the jenkins-semci library and walk through using it in a simple project.

Image for post
Image for post
Photo by Brooke Cagle on Unsplash

Components

  1. Source Repository: We’ll use GitHub for managing our source code. Our solution relies on repository tagging to automate versioning, so if you have a different repository, it must allow tagging. To use our OSS plugins, it must also implement the Git protocol.
  2. Build Server: We’ll use Jenkins (https://www.jenkins.io) for automating our builds. Jenkins is a free, mature, open source build server written in Java. It provides a feature-rich plugin ecosystem and hooks for writing build scripts.
  3. The jenkins-semci plugin (https://github.com/stainlessai/jenkins-semci). This plugin provides a shared library for Jenkins to automate artifact versioning using Semantic Versioning, Git tags and branches.

Jenkins supplies a tutorial for creating multibranch pipelines, and we’re going to piggyback on that tutorial, adding automated versioning capability to the sample project provided there. The tutorial is here:

NOTE: There are some changes we need to make to the stock tutorial to demo jenkins-semci. The docker agent we use to build must have a git client installed. The agent described in the tutorial doesn’t by default. To do this, we’ll add the following Dockerfile to the project, called “Dockerfile.build”:

FROM node:6-alpineRUN apk add -U git

Next, we’ll change the agent definition in our build file to use this Dockerfile to build our agent. We’ll modify our Jenkinsfile as follows:

pipeline {
agent {
dockerfile {
filename 'Dockerfile.build'
args '-p 3000:3000 -p 5000:5000'
}
}
... FILE CONTINUES ...

If you already have a Jenkins project you want to enhance with the jenkins-semci plugin, you can skip the tutorial. Otherwise, this article assumes you’ve gone through the tutorial with the above modifications, and have Jenkins set up with the multibranch pipeline created. After running the tutorial, you have 3 branches: development, master, and production, each with a Jenkinsfile that has the stages “build,” “test,” “Deliver for development,” and “Deploy to production.”

First we have to configure the shared library in Jenkins. This will instruct Jenkins to pull the library code directly from Github when the jobs run. This is achieved on the Jenkins job configuration page, which is accessible from the “Configure” menu item of the job page in Jenkins classic, and by clicking the cog icon on the job page in Blue Ocean.

Image for post
Image for post
Configure job icon on the Job page
Image for post
Image for post
Configure job on the Blue Ocean page

Once on the configure page, scroll down to the “Pipeline Libraries” section, and click “Add”

Image for post
Image for post
Setting up the jenkins-semci shared library

We’ll name the library “jenkins-semci,” although you can name it whatever you want as long as you use the same name in the corresponding .@Library tag in the pipeline file (see below).

We’ll use the “GitHub” option under “Source Code Management,” and fill in the “Repository HTTPS URL” value. It’s a public repository, so no credentials are needed.

Image for post
Image for post
Setting up the repository URL for the shared library

We’re now going to open that pipeline file and install the jenkins-semci plugin by adding the lines below:

@Library('jenkins-semci')
import ai.stainless.jenkins.ReleaseManager
def releaseManager = new ReleaseManager(this)pipeline {
... FILE CONTINUES ...

The last thing we need to do is to ensure that our project pulls down tags when it checks out revisions to build. This is accomplished by adding an “Advanced Clone Behavior” in the project configuration under the Branch Sources panel of the configuration:

Image for post
Image for post
Image for post
Image for post
Configuring our job to fetch tags on checkout

The ReleaseManager class assumes the branch that produces release is named “master.” To change this to “main,” or in the case of the tutorial, “production,” we set the masterBranch property of the ReleaseManager to the value we want:

releaseManager.masterBranch = 'production'

Now, if we run a build on the production branch, two things will happen:

  1. The ReleaseManager will check to see if the commit hash of the build matches the commit hash of a release tag. If no match is found, an error is thrown. To resolve, tag the commit you want to release with the desired version and ensure that commit is on the master branch.
  2. The computed version will reflect the tag of the commit.

If we run a build on any other branch, the commit check is skipped and the build is named with the branch name and -SNAPSHOT in the pre-release.

To try it out in our tutorial build, we can add a “Version” stage to the build and print out the computed version. Note that we can all the releaseManager.artifactVersion() method from any stage, and even pass the string to other calls, like Docker tags, or build scripts. There are other methods as well that just return the project name appended to the version if we wanted to write a complete filename.

stages {
stage('Version') {
steps {
print releaseManager.artifactVersion()
}
}
... FILE CONTINUES ...

After running the ‘development’ branch build, we’ll see

Image for post
Image for post
Output from successful call to jenkins-semci library

The version 0.0.1 indicates we haven’t added any tags, and this is build #15. It’s not on the ‘production’ branch, so it’s labeled with prerelease development-SNAPSHOT. If we ran this on the master branch, we’d see a prerelease of master-SNAPSHOT.

Let’s build a release! First, add a valid semantic tag to the repository prefaced with v, e.g.

git checkout production
git merge development // merge in our recent changes
git tag -a v1.2.3 -m "Testing tags"

Now run the production branch build in Jenkins, and releaseManager.artifactVersion() returns a version corresponding to your new tag:

Image for post
Image for post
Newly-minted version 1.2.3

Finally, run the development branch build again. You’ll see the semantic version has changed to reflect the fact that version 1.2.3 was released while you were working on your branch, so your SNAPSHOT versions now start at version 1.3.0

Image for post
Image for post
Dynamic version updating!

The library also supports multiple projects in a single repo using an at ‘@’ prefix for discriminating between projects. More details and documentation can be found at the shared library README. Let us know what you think, or if you’re successfully using the library! PRs open for updates and changes!

The Signal

Practical notes on artificial intelligence, machine learning and computer vision from Stainless AI

Dan Stieglitz

Written by

Dan is the CEO of Stainless AI, Inc., which provides cognitive computing solutions to businesses through machine learning and artificial intelligence.

The Signal

Technical articles and musings from Stainless AI, a provider of cognitive computing solutions: machine learning, computer vision, data science, and more!

Dan Stieglitz

Written by

Dan is the CEO of Stainless AI, Inc., which provides cognitive computing solutions to businesses through machine learning and artificial intelligence.

The Signal

Technical articles and musings from Stainless AI, a provider of cognitive computing solutions: machine learning, computer vision, data science, and more!

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store