Pipelines everywhere

Peter Vanhee
Jan 31, 2019 · 5 min read

Continuously delivering software to production systems is a key part of building great digital products and creating a culture of agility within different teams. In this post, I will share our recipes for building pipelines centered around Concourse CI.

Pipelines at Comic Relief

It does take time and perseverance to build pipelines that are trusted by different teams, starting with your engineering and product folk and all the way to your business stakeholders. Though once everyone has seen the power of continuous delivery, it is something that quickly becomes key to every project. Fixed development sprints are being gradually replaced with a fluid kanban model as production releases happen not once every two weeks but multiple times a day.

Image for post
Image for post

Whilst moving to serverless certainly reduces the need for infrastructure management and therefore greatly reduces the complexity in shipping software, there is a need to automate this and often bridge between your different technology stacks. For us, pipelines really are the glue that makes everything work.

At Comic Relief, we have a long history of continuous integration and have been using tools such as Jenkins, Travis, Circle CI, Netlify and Concourse CI. Over the last couple of years, we have settled on the following pattern:

  • Circle CI (and sometimes still Travis CI) for running tests on every pull request. These tests are great for testing things in isolation are ideally suited for front-end tests or unit tests, although we’re using Cypress to do end-to-end testing against sandbox back-end environments.

In the rest of this post, we’ll focus on the pipelines that ship code to production from the master branch onwards. We found that Concourse is ideally suited as our “master CI tool” and elegantly deals with the concept of pipelines as the complexity of our software products grows over time.

Concourse.ci pipeline concepts

A concourse pipeline can be thought of as a distributed, higher-level and continuously-running Makefile, written down in a yml format (see an example).

You start by defining your resources, which can range from the master branch of a github repository to a cron being triggered every day at 10am.

You then move on to define jobs that depend on resources, for example a deployment to staging can be written out as a job. The job plan will have a dependency on the github resource and the job can be triggered to run every time a new commit is made to the master branch.

Finally, you chain your jobs by depending on the resources that have passed through previous jobs. In our example, your fresh commit to master, once it has passed the staging job, will trigger the execution of a new job that will run feature tests against the freshly deployed staging environment.

Image for post
Image for post

Diving into a Concourse job

Jobs are the heart of your Concourse pipeline, so let’s take a look at how a job is defined.

Here is an example of a job running feature tests using Cypress against a previously deployed staging environment.

The job lists serial:true which means that only one such job will run at a given time, to avoid multiple commits to master triggering multiple jobs to be created. Instead, jobs are nicely queued up and executed serially.

The job plan has two steps. First, two resources are pulled in: a git commit from our application repository that has successfully passed the staging job and our bootstrap repository which contains a series of reusable scripts.

The second step of our job plan is a task which really can be thought of as a function mapping inputs to outputs and return either success or failure. In this case, our task is defined in a file called cypress.yml and receives as input our code repository (which has our latest commit to master) and the bootstrap repository. This task references a cypress docker-image which is used to run our task and the run script invokes a shell script cypress.sh.

That shell script in turn installs our npm packages and invokes yarn test which will run our tests using cypress.io. The exit code returned by that script will define success (exit code 0) or failure (any other number).

Finally, the last part of our job handles failures, and posts a message to a designated Slack channel mentioning that feature tests have failed. The slack-alert we use here (and which contains the private slack webhook url) is just another resource that is configured at the top of our pipeline yml file.

We could equally have handled success and posted a message to Slack, but as this job is not the final job of the pipeline, we usually only report on failures.

Rolling your own Concourse

Concourse is not as easy to set up as say Circle CI, and unfortunately no managed Concourse service exists. Luckily, the talented folks at EngineerBetter built a tool called concourse-upthat makes it really easy to set up your own Concourse CI instance on AWS.

concourse-up deploy ci-sandbox

Follow the documentation to get started with your own Concourse instance. All resources will be created and you’ll get two URLs. One points to Concourse’s UI (which is your dashboard into your pipeline), and another one to a Grafana instance that will display stats like pipeline run times.

Next, take a look at our sample bootstrap repository which defines a general pipeline and some scripts to do things like running feature tests using cypress, or use yarn and the aws cli to deploy a static site to AWS S3.

git clone https://github.com/comicrelief/sample-bootstrap cd sample-bootstrap

We can now load the pipelines that we defined in the bootstrap repository into that instance using fly, the command-line interface to Concourse.

fly -t ci-sandbox login --team-name main --concourse-url YOUR-URL fly -t ci-sandbox set-pipeline -p general -c concourse/general.yml -l private/concourse/cci_private_vars.yml

And that’s all! You should be able to inspect your pipeline via the UI. Every time you update your pipeline, you have to push it up using fly and the changes will be reflected immediately.

At Comic Relief, everything is shipped to production using pipelines. We use it for our main websites (comicrelief.com, sportrelief.com), for our donation & giftaid platforms, and for many other public (and private) services that benefit from a continuous delivery model. Having all our products accessible and visible in Concourse CI has changed the way we build and deploy software for the better.

Originally published at technology.comicrelief.com on January 31, 2019.

Comic Relief Technology

This is where we showcase the ways that we are use…

Peter Vanhee

Written by

Passionate technologist. Founder and director @marzeelabs @weareserverless. ex-head of technology @comicrelief

Comic Relief Technology

This is where we showcase the ways that we are use technology to support the core aims of our organisation and how we do it.

Peter Vanhee

Written by

Passionate technologist. Founder and director @marzeelabs @weareserverless. ex-head of technology @comicrelief

Comic Relief Technology

This is where we showcase the ways that we are use technology to support the core aims of our organisation and how we do it.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store