Set up a release pipeline

Last time I made a git repo and promised I would write a terminal loop with the golang.org/x/crypto/ssh/terminal package. Then two months past and in the meantime I read some books (including The DevOps Handbook). I’ve been thinking a lot about continuous deployment. So I decided to first create a continuous deployment pipeline for the project.

What is deployment?

We have some code that builds and runs. We want other people to be able to do the same with our code, but we don’t want them to have to install all the tools we’re using to achieve this (git, Go, etc). So we can package the application (as pkg / brew for OSX, msi / chocolatey for windows, deb for debian, apk for alpine, rpm for linux, etc) such that the end user has the same experience installing our software that they do with any other software on their machine.

What is continuous deployment?

We have a git repository with a master branch that we can push new features and bug fixes to. We want those changes to get to our users as soon as possible so we can get faster and more gradual feedback on them (rather than releasing everything at once). To achieve this we can automate the deployment of a new version for every new commit to the master branch.

So we want to automatically take some action when something is merged into master. Since we’re using GitHub I’ll use the TravisCI integration to support this automation. It’s totally free for open source projects, it’s simple and convenient to set up, and it’s driven from config in the git repo itself meaning that all changes to the automation are tracked in version control. Other options include but are not limited to CircleCI and Oracle Container Pipelines (aka Wercker).

I’m adding this .travis.yml file to the repo (at the top level):

language: go
go:
  - "1.11.x"
os:
  - linux
  - osx
script:
  - go test -race -cover ./...

deploy:
  provider: script
  script: bash scripts/deploy.sh
  on:
    branch: master

Firstly this says we’re building with Go, specifically the latest patch version of Go 1.11. Then it says to run tests on both linux and OSX, noting that we’re enabling the race detector and calculating test coverage. Then, if that is successful, the deploy script will be called.

Then we create a deploy script:

$ mkdir scripts
$ echo 'echo "TODO: write deploy script"' > scripts/deploy.sh
$ chmod +x scripts/deploy.sh

Then we push a new branch, create a PR on Github, do a bunch of clicking to set up TravisCI for the repository, see that it’s not running for the PR, amend the commit we pushed, and force push to actually trigger the first Travis build. Now that Travis is configured and working every push to a PR / master will trigger Travis to run tests (which also now runs go vet). On master only the deploy script will be run.

So I waited for the green ticks and then Squashed and merged the PR.

Aside about git bisect

You may notice “Squash and merge”. I’ve deactivated the other merge techniques for this repo. This is since:

1. All commits on master must individually pass all tests so that the git bisect technique can be properly employed.
2. Unsquashed commits don’t have to pass CI (therefore rebase is bad by #1).
3. Pure merge commits do not present a readable git log, making git bisect and general historical inquiry difficult.

What is git bisect and why is it important to maintain git hygiene to enable this technique?

Git bisect is a technique for identifying the commit that introduced a bug. First a failing test is written, then that test is played on a recent commit that is known to have the bug and an old commit that is known to not have the bug (these commits act as a baseline). Then the test is applied to the commit that is halfway between those two commits in the git history of master (hence “bisect”). This process is repeated in a binary search until the commit that introduced the bug is located. Commits should also be small and easy to review (this comes with the continuous delivery philosophy) which should make it easier to identify the bug once locating the specific buggy commit.

Git bisect can be sped up using the commit titles and descriptions in the history to prioritize testing certain kinds of commits over others. This is difficult if every commit says simply “Merged from branch X” rather than something descriptive like “add clock widget to desktop app”.

Back to TravisCI

So, I checked and apparently the deploy step runs once for each OS type specified in .travis.yml. Given that we want the deploy script to cross compile for a wider variety of platforms than Travis supports, let’s make it only run on linux like so:

deploy:
  provider: script
  script: bash scripts/deploy.sh
  on:
    branch: master
    condition: "$TRAVIS_OS_NAME == linux"

(Note: I tried "$TRAVIS_OS_NAME" == "linux" first and Travis stopped running on commits altogether)

In the Travis build output we can see that the deploy script is now skipped on OSX and it’s being run on the linux build on master only. Great!

Wait but what about the actual deploy script?

We left a big ol’ TODO in the deploy script. Isn’t the deploy script kind of the point here? What should the deploy.sh do?

Well basically it needs to deploy all the things!(TM)

We’ll start with deploying to something basic next time and then add complexity gradually (continuous delivery demands gradualness).