Commit Go Code with Joy and Confidence
Pre-commit in style with gogitix
When we commit code at LaunchDarkly, we treat ourselves to a colorful and confidence-inspiring test- and lint-running experience thanks to an open-source tool we’ve developed called gogitix. Here’s what it looks like to commit with gogitix:
How’d it come to this?
I’ve been a Go programmer for a few months now — ever since I joined the talented team at LaunchDarkly — and I have a confession to make. At least once a week, maybe more often, I commit a change to a source code branch that (a) has not been tested in any way, and (b) I am supremely confident will not break anything. The strict typing of golang has definitely allowed me to get things right far more often than first time than I ever managed while coding in Ruby or Python. Nonetheless, I think you know how this story often ends: I discover on CI that I broke a test, I have to commit a fix, and the whole process takes longer than I had ever imagined.
This is why we use pre-commit hooks
Like many companies, we set up local git pre-commit hooks on our repos to bail us out before our optimism makes fools of us. Before introducing unnecessary latency into the development process and possibly consuming a bunch of compute resources, we wisely run a few checks locally.
One of the first hooks I added upon joining LaunchDarkly was a small script to make sure I didn’t leave fmt.Print* statements in my code. This was a habit that was sending me time and again to update a PR (and wait for the tests re-run) before I could merge it. Here’s what it looks like:
#!/usr/bin/env bash
EXCLUDED=(':(exclude)vendor/' ':(exclude)*_test.go')
RED='\033[0;31m'
NC='\033[0m' # No Color
# Skip this if no non-vendor go files have changed
changed_files=$(git diff --cached --name-only --diff-filter=ACM -- '*.go' ${EXCLUDED[@]})
[ -n "$changed_files" ] || exit 0
results=$(git grep --cached -H --line-number fmt.Print -- $changed_files)
status=$?
if [[ $status == 0 ]]; then
echo -e "${RED}ERROR: Found unexpected fmt.Print* calls:\n$results$NC"
exit 1
fiI wish it went without saying that color is essential to any good pre-commit hook. A little color goes a long way to make you feel like you’re coding in style.
Another thing you might notice about this pre-commit hook is that it is not checking the filesystem but rather the git “index” through the use of the “ — cached” flag. Git has the curious property that what’s on your local filesystem is not necessarily what you’re committing. Much of the time it is, but ultimately “git commit” commits the “index” (a.k.a. staging) to the repo. So to do the correct thing and truly check what is actually going to change in the repo, you do need to look at the “index”. That may not seem particularly necessary for this rather simple check, but it is more important when we are running checks that span whole packages and require that the entire “index” compile.
An aside: Decentralizing pre-commit hooks
It’s worth mentioning that at LaunchDarkly, we have a decentralized pre-commit hook setup based on Elliott Cable’s setup for paws.js. However you set it up, it should be easy for developers to add new hooks and it’s nice if that doesn’t involve a giant, monolithic shell script. You can see how Elliot did it here.
Pre-commit Hooks: The Rules
Below are the two of rules I follow when developing pre-commit hooks.
Rule 1: If you check something in your pre-commit hook, you should also check it in your CI build.
You could probably figure this out pretty quickly on your own, but the reason for this is that you don’t want someone else to locally update a file that you just changed and then discover they can’t commit it. Just because you didn’t set up the pre-commit hooks on your local machine or you made a change using the Github UI, doesn’t mean you should be able to leave a surprise for the next engineer who updates that file.
I like to leave a little message with CI failures that have git hooks that that says, “Please make sure your git hooks are up-to-date!” I want to remind myself and my colleagues that they never needed it to come to this.
Rule 2: Make it fast!
I mentioned above that I don’t like running my tests locally. They may take too long (our full suite would take a few minutes), and I might want to checkout a fresh branch and start another another change right away. You can write a pre-commit hook that runs your entire test suite but your colleagues may not enjoy waiting for minutes. Maybe the pre-commit hook could run some relevant subset of tests. That’s more reasonable, but there’s a balance between what you want to do locally and what you want to happen on CI.
At LaunchDarkly, we’ve settled on verifying that our golang package test suites all compile, without actually running them. This job only takes about 10 seconds but it can save some really stupid errors. Finally, don’t forget that your laptop may have 2 or 4 CPU cores on it, which means that if you want fast hooks, you may want to parallelize.
Introducing Gogitix
While I loved the decentralized behavior of our git hooks that made it easy for anyone to add one, performance started to grind to a halt. To mitigate this, I only wanted checks to run only on files that had been updated locally.
Figuring out which files these are is a time consuming process. It turns out this process is also complex: Tools in the golang ecosystem operate at a number of levels. There are tools that operate on files, others that operate on individual directories, some that operate on an entire subtree, and finally tools that operate on packages. (Tools like gometalinter — which provides a way to normalize the running of linters on a go code base — have to deal with this.) In addition, we use govendor to manage our dependencies and to get lists of non-vendor packages. govendor is not particularly fast at generating these lists, so we don’t want to be calling it too many times.
While gometalinter is appealing because it can run most any linter, it’s more configuration heavy than I’d prefer and it doesn’t solve the problem of operating only on files that have changed.
Enter Gogitix
Here is how we describe golang pre-commit flow using a YAML based configuration file:
- reformat:
check: goimports -l -local github.com/launchdarkly/gogitix {{ ._files_ }}
format: goimports -w -local github.com/launchdarkly/gogitix {{ ._files_ }}
- parallel:
- run:
name: govendor list +outside
command: |
(govendor list +outside | grep -v "^ ex") || true
description: Checking for external dependencies
expect_silence: true
{{ if gt (len .packages) 0 }}
- run: staticcheck {{ ._packages_ }}
{{ end }}
{{ if gt (len .dirs) 0 }}
- run:
name: go vet
command: go tool vet -composites=false {{ ._dirs_ }}
{{ end }}
- run:
- description: Verify that govendor is consistent
command: govendor sync -n
expect_silence: true
- description: Verify that govendor status is ok
command: govendor status
{{ if gt (len .packages) 0 }}
- run:
name: gotest compile
description: Compiling and initializing tests (but not running them)
command: |
go test -run non-existent-test-name {{ ._packages_ }}
{{ end }}Don’t be surprised that this file looks a fair amount like a CircleCI configuration file. We’re very familiar with CircleCI because we use it for our CI runs.
gogitix does a few interesting things:
- It creates a temporary directory that has just the changes in your git index.
- It passes your template parameters that represent lists that have changes: files, packages, dirs and trees. (trees represent a minimal set of directories that includes all the changes to files in their subtree)
- It allows you to parallelize groups of commands.
- It has an interactive “reformat” command that will tell you if your code needs to be reformatted to meet the repo convention and prompts you to review the change.
Try it yourself. git add a change to your repo and see what gogitix does for you by running:
go get -u github.com/launchdarkly/gogitix
gogitixIf you want to use and build great tools like this every day, come join our team at LaunchDarkly. We’re hiring!
