Continuous delivery with Jenkins Pipelines

A few months back, we were already using at 21Buttons some of the best practices on software development: separation of concerns, unit & integration testing, code reviews, release often… However, when a feature had to be released to production, we built the software artifacts on our local machines. This approach came with multiple downsides:

• Execution of different types of tests was not guaranteed so the artifact could be broken but generated.
• The artifact generation wasn’t isolated so for the same git commit the artifact built on one developer machine could differ from the artifact built on other developer machine.

We needed to find a solution that let us be sure that all of the best practices we were already using got reflected on the artifacts… pipelines to the rescue!

A continuous delivery (CD) pipeline is an automated expression of your process for getting software from version control right through to your users and customers. Every change to your software (committed in source control) goes through a complex process on its way to being released. This process involves building the software in a reliable and repeatable manner, as well as progressing the built software (called a “build”) through multiple stages of testing and deployment.

After analyzing different solutions for implementing software pipelines we decided to chose: Jenkins Pipelines.

We had a Jenkins instance deployed in our infrastructure so Jenkins Pipelines seemed to fit right for us. We just needed to add a Jenkinsfile inside the projects we wanted to add the pipeline and voilà the pipelines of these projects were automatically run on Jenkins.

Here you can see an example of how a pipeline of one of our backend projects looks:

The flow is the following:

• When someone pushes a change to the code repository, a Bitbucket webhook fires the execution of the pipeline.
• The “build” stage is the first stage executed checking out the code and building a Docker image.
• The “qa: static code analysis” stage runs doing static analysis on the code of the Docker image generated on the previous stage.
• The “qa: unit & integration tests” stage makes sure tests are run before deploying the artifact.
• The “approval: dev” and “deploy: dev” stages are ignored since we only deploy to dev branches that are not master.
• The “deploy: stg” stage deploys the code to the staging environment.
• The “approval: prod” stage lets you choose between deploying this artifact to production or ignoring it. If we would omit this stage from our pipelines we will be doing continuous deployment but by keeping it we are just doing continuous delivery.
• The “deploy: prod” stage deploys the code to the production environment.
• The “publish api docs: prod” stage keeps our OpenAPI documentation up-to-date.

It is worth noting that each stage is only executed if the previous stages were run successfully so, for example, you can’t deploy to staging if the static code analysis or tests fails.

Ok, fine, but how does a Jenkinsfile look? Well, Jenkinsfile can be defined on a declarative or scripted syntax. We chose to use the declarative syntax as it is simpler and gives us the flexibility required by our use cases.

A simple example of a Jenkinsfile is:

After some days working on this, Jenkins was able to automatically generate fully tested .apk artifacts for Android and .ipa artifacts for iOS as well as Docker images for our backend services.

The problem with backend was that while we have a project for Android and a project for iOS we have multiple projects for backend with really similar Jenkinsfile files. What is more, while the pipeline for Android was different from the pipeline for iOS and the pipelines from backend some of the actions of all of these pipelines were common: notify Slack, notify Bitbucket, etc.

We were lucky when we discovered Jenkins Pipelines has support for creating “Shared Libraries” which can be defined in external source control repositories and loaded into existing Jenkins Pipelines.

We can define a global “backendServicePipeline” variable which is a template of our backend pipeline:

With this template defined creating a pipeline for a new service is as easy as define the following on Jenkinsfile:

Although we have improved the way software is released on 21 Buttons the task is not finished yet as the following improvements can be made:

• The .apk and .ipa files are generated on mobile projects but these file are still uploaded manually to Google and Apple Stores. For example, it can be done automatically in Google Play with the following plugin: Google Play Android Publisher.
• Speed up pipeline execution times caching different types of data.
• Adding a rollback stage on backend pipelines (at the moment we are using Heroku rollback feature).
• Finding out better ways to keep pipelines alive before deploying to production (in the current version we have a stage with a timeout of 5 minutes).