Continuous Release Pipeline with Travis CI

`git push -> Travis CI test+build -> automatic snapshots+releases`

6 min readMar 9, 2018

At Even Financial, we’re always looking to streamline the development process. We want to make sure any generated releases pass all our tests before they’re pushed into our GitHub repository. As our code-base grows, the time to run our tests grows, which means more waiting for our test suites to complete before we’re able to release, or even switch focus to a different task. We set out to remove this waiting component, and automate our release process. As a bonus, we’re going to be generating snapshot builds so that we can easily view changes, and new features without needing an explicit release.

In this article, we’re going to be building this pipeline for a JavaScript application. We’ll be using Travis CI for our continuous integration, GitHub for hosting our repository, and Amazon S3 to host our build artifacts.

Workflow

Our development workflow consists of starting feature branches, developing on those branches, then merging back into master. A release is a tagged version on master, and corresponds with a new (deploy-able) version of our project. Travis CI automatically builds the most recent new commits on each branch. We just need to configure Travis CI to automatically tag, and release new commits on master, and release snapshot builds on development branches. An important distinction to note, is that what we consider a release, Travis CI considers a deploy. Our travis.yml will have ‘deploy’ steps, but for the purposes of this article, they are releases. A deploy in our workflow is simply pointing our (non-versioned) URLs to a versioned s3 release bucket. So, as we develop, we’ll have automatic snapshots at https://evenfinancial.com/snapshot/<commit hash prefix>/…, and new commits on master at https://evenfinancial.com/v/<release version>/…. Then there’s https://evenfinancial.com/…, which acts as an alias for a versioned URL.

Pipeline Overview

There are 3 main cases our pipeline needs to handle:

A commit on a development branch -> a snapshot release.
A commit on master branch -> tag commit (on master) with an incremented version.
A tag on master branch -> a version release.

Base travis.yml

First, we’ll need to have Travis CI integrated with our GitHub repo, and set up to build on new commits. We’re building in an npm project, so our initial travis.yml will look like the following:

language: node_js
node_js: "8"script:
  - npm run lint
  - npm run test

This assumes that our package.json includes a test, and a lint under scripts.

Snapshot Releases

Let’s start with setting up snapshot releases on all branches, except master. Our releases consist of simply uploading our build artifacts to s3 buckets. Travis CI has a deploy provider (for our purposes a ‘release provider’) for s3. There are multiple configuration options, minimally, we need our access_key_id, secret_access_key, and bucket. We need to make sure the build artifacts are there before uploading them to s3, so we’ll also set skip_cleanup to true. Since, our npm run test does not emit artifacts, we’ll also make sure to add npm run build in our Travis CI before_deploy. (test, and build are additional entries in our package.json scripts array). We’ll add the following to our travis.yml:

before_deploy:
  - npm run builddeploy:
  - on:
      all_branches: true
      condition: $TRAVIS_BRANCH != master
    upload-dir: snapshot/${TRAVIS_COMMIT::7}
    provider: s3
    access_key_id: $AWS_ACCESS_KEY_ID
    secret_access_key: $AWS_SECRET_KEY
    bucket: "<my bucket>"
    local_dir: ./dist
    skip_cleanup: true
    wait-until-deployed: true

The wait-until-deployed option does not mark the test as complete until our files are successfully uploaded to s3. local_dir is the location of our build artifacts. We grab access_key_id, and secret_access_key from our Travic-CI custom environment variables, and the artifacts path from $TRAVIS_COMMIT, which is set by Travis CI.

This configuration sets our Travis CI to upload a snapshot to s3 on every build, since master builds should behave differently, we’ll also add a custom condition to exclude snapshot releases on master.

Version Releases

A tagged commit (we’ll get into how to tag commits later on) should trigger our Travis CI to upload our build artifacts to an s3 path based on the tag version. All our tags will be formatted as v<version number>. Travis CI already provides us with $TRAVIS_TAG, so we only need to change our upload-dir to ${TRAVIS_TAG:1} (:1 to skip the first character, which, in our case, is always v).

Travis CI has a tags condition, which causes it to ignore the default branch: master condition. We’ll use this to only do a version release when the build commit is tagged. Adding this deploy configuration, our full travis.yml should now look like this:

language: node_js
node_js: "8"script:
  - npm run lint
  - npm run testbefore_deploy:
  - npm run build-: &s3_deploy
    provider: s3
    access_key_id: $AWS_ACCESS_KEY_ID
    secret_access_key: $AWS_SECRET_KEY
    bucket: "<my bucket>"
    local_dir: ./dist
    skip_cleanup: true
    wait-until-deployed: truedeploy:
  # release a snapshot for each commit on any branch except master
  - <<: *s3_deploy
    on:
      all_branches: true
      condition: $TRAVIS_BRANCH != master
    upload-dir: snapshot/${TRAVIS_COMMIT::7}# release a version when on master and commit is tagged
  - <<: *s3_deploy
    on:
      tags: true
    upload-dir: ${TRAVIS_TAG:1}

Update Version and Tag

Travis Config
We want Travis CI to update our package.json version, and add a git tag with the new version whenever there is a new commit on master. We’ll need a custom script for this, but first let’s get the Travis CI configuration out of the way. We’ll add a deploy entry that uses the script provider:

- on:
    branch: master
    condition: -z $(git tag --points-at $TRAVIS_COMMIT)
  provider: script
  script: $TRAVIS_BUILD_DIR/bin/travis_release.sh

Travis CI tests conditions as if [[ <condition> ]]; then <deploy>; fi, so we check that the current commit does not have a tag yet using -z $(git tag — points-at $TRAVIS_COMMIT). Without this additional condition, there’s be an infinite loop of commits (our script adds a new commit, which triggers a new build). $TRAVIS_BUILD_DIR/bin/travis_release.sh is the location of our release script.

Release Script
In our release script, we’ll use npm version to update package.json, and package-lock.json, then we’ll push those changes to our GitHub repository.

#!/bin/bash
set -enpm version $GIT_TAG_VERSION.0.0 --no-git-tag-versiongit config --global user.email "build@travis-ci.com"
git config --global user.name "Travis CI"git checkout $TRAVIS_BRANCHgit add $TRAVIS_BUILD_DIR/package.json $TRAVIS_BUILD_DIR/package-lock.jsongit commit -m "Setting version to $GIT_TAG_VERSION"
git tag v$GIT_TAG_VERSION -a -m "Tagging version v$GIT_TAG_VERSION"git push origin $TRAVIS_BRANCH 2>&1
git push origin $TRAVIS_BRANCH --tags 2>&1

We use --no-git-tag-version with npm since we’re manually setting the tag and commit message. In order to determine $GIT_TAG_VERSION, we just need to get the precious tag version, and increment it. git describe —-abbrev=0 --tags gives us the most recent tag, so we can export LAST_GIT_TAG_VERSION, and the incremented GIT_TAG_VERSION in our travis.yml. Our final Travis CI config should be:

language: node_js
node_js: "8"script:
  - npm run lint
  - npm run testafter_success:
  - export LAST_GIT_TAG_VERSION=$(git describe --abbrev=0 --tags) && export LAST_GIT_TAG_VERSION=${LAST_GIT_TAG_VERSION:1}
  - export GIT_TAG_VERSION=$((LAST_GIT_TAG_VERSION+1))before_deploy:
  - npm run build-: &s3_deploy
    provider: s3
    access_key_id: $AWS_ACCESS_KEY_ID
    secret_access_key: $AWS_SECRET_KEY
    bucket: "<my bucket>"
    local_dir: ./dist
    skip_cleanup: true
    wait-until-deployed: truedeploy:
  # release a snapshot for each commit on any branch except master
  - <<: *s3_deploy
    on:
      all_branches: true
      condition: $TRAVIS_BRANCH != master
    upload-dir: snapshot/${TRAVIS_COMMIT::7}# release a version when on master and commit is tagged
  - <<: *s3_deploy
    on:
      tags: true
    upload-dir: ${TRAVIS_TAG:1}# update version, and commit tag when on master and not tagged
  - on:
      branch: master
      condition: -z $(git tag --points-at $TRAVIS_COMMIT)
    provider: script
    script: $TRAVIS_BUILD_DIR/bin/travis_release.sh

The end result is a workflow which consists of pushing feature branches to GitHub, code review, then merging said changes. All while live snapshots and releases are automatically generated without requiring any manual input.

Accessing on the Internet

Now that our build artifacts are in S3, we want it to be accessible using the non-versioned URL alias. We use Amazon API Gateway as a simple proxy in front of our Amazon S3 bucket. We set up our API to have 3 main resources:

/snapshot/{s3path+}
/v/{s3path+}
/{s3path+}

All of the resources are configured as HTTP proxies, with s3path mapped from method.request.path.s3path. /v, and /snapshot point to their corresponding https://s3.amazonaws.com/<bucket name>/<snapshot|version>/{s3path}. The remaining non-versioned resource, points to https://s3.amazonaws.com/<bucket name>/{version}/{s3path}, with version mapped from stageVariables.version.

stageVariables.version is how we control our deployments. A new deployment can be created using the Amazon API Gateway console. However, there is no guarantee that there’s a file corresponding to that version in the S3 bucket. So, at EvenFinancial, we use an AWS Lambda to verify that a version exists before deploying the API Gateway. This gives us the flexibility to initiate a deploy by simply making an HTTP request.