Check Yourself Before You Wreck Yourself
How we use GitHub Checks at Compass
For the first 6.5 years of Compass’ software engineering history, the web applications teams have used a tool called Gerrit for performing code review. It was a great tool with some incredible features for performing detailed code review that are still unmatched. But as Compass’ engineering organization grew, we realized the time had come to move to a more industry-standard review tool. And so in the Spring of 2019, I began our migration from using Gerrit for code review for our frontend web applications to using GitHub Pull Requests.
Our frontend web application repository — known internally as uc-frontend— is a monorepo containing our Docker-based NodeJS applications and our NPM-based packages. When originally designing this repo in the Fall of 2017, we wanted to ensure we could benefit from the advantages that come from having all our code in one place (discoverability, maintainability, pattern enforcement) without falling into the classic traps that can arise (circular dependencies, over-coupling, monolithic deploys). We addressed this through extensive use of automation in our CI pipeline, at the time based on Jenkins. As we built out the repository, and as new apps and packages came on board, we would look out for any bad patterns and implement guards against them in our CI script to fail the verification step. Things such as not making changes to multiple separate entities at once, not linking directly between packages, and good naming schemes became standard rules, and as we thought them up we appended them to the end of our script.
As you can imagine, over the course of 18 months this script became quite hard to manage. It was difficult for new engineers to learn what the rules were since most weren’t really documented. Additionally since the script was synchronous, if they failed a rule early on it would immediately exit. If someone had 10 errors in their code, they would need to fix and push up their code to run the script 10 times before they’d get to all of them. This situation wasn’t ideal, and the transition to using GitHub was the perfect opportunity to step back and re-think how we could improve this workflow.
Enter: GitHub Checks.
Checking out Checks
Checks are a relatively new feature on the GitHub platform. They’re designed to provide contextual feedback on code during code review to prevent mistakes or discouraged patterns from entering the code base. While manual human code review is still necessary to provide insight on the structure and architecture of your code, a lot of code smells can be caught by automation rules and GitHub checks provide a pleasing UI and API to allow you to do so.
A Check is a GitHub App that implements the Checks API. The great news is there’s already a large community of GitHub app-makers, and some quick Googling turned up a framework called Probot that simplifies many aspects of app development. Probot, much like our frontend web apps, is a NodeJS-based web application, which also meant we could re-use our existing deployment infrastructure to deploy the app to one of our existing Kubernetes clusters.
With a plan in-hand, I got to work on re-writing our CI as Checks.
A Framework For Your Framework
Setting up Probot turned out to be a breeze. It’s mostly a wrapper around Express that handles much of the annoyance around authenticating with the GitHub API as an application. It also abstracts away interacting with GitHub’s webhooks, so you can write your code to look more event-driven:
You also get a pre-configured Octokit API client for interacting with GitHub’s REST or GraphQL APIs. However, what you don’t get is any sort of help with creating, updating, or finishing Checks, Check Runs, or Check Suites specifically. The Checks API requires a lot of boilerplate code, and I didn’t want each Check we wrote to have to re-implement that. So I wrote some helpers to standardize interacting with Checks via an ICheckRun TypeScript interface and a CheckRun base class.
The ICheckRun interface abstracts the basic lifecycle of a Check into three main parts: start(), run(), and finish(). The CheckRun base class functions handle interacting with the API itself.
Once abstracted in this way, new Checks simply have to extend and implement CheckRuns, and worry purely about their own business logic and not about any of the details of API options, as demonstrated in our Sample coin-flip-like Check here:
The Check bone’s connected to the… Check Suite
With our basic framework up and running, it was time to write all our Checks. We ended up with nine checks that formed a Check Suite for our frontend (internally code-named Hydra). They verify all sorts of aspects of our codebase: correctly-set NPM dependencies, app and package isolation, and catching common mistakes such as mismatched NodeJS versions between the package.json's engines field and the Dockerfile.
Our app listens for all these webhook events, and if it determines that all of the criteria are met for a Check to start (by returning truthy or falsy from the start() function), it kicks off until responding with a conclusion of “success” or “failure”. Best of all, these Checks all run asynchronously and in parallel, meaning you can see every rule you break in your PR all at once, and they all run in under 15 seconds total.
This was a massive productivity improvement to every frontend developer’s workflow, and the Checks have been very well received by team members.
We even solved the issue of not knowing what the rules are as a side-benefit of writing unit tests:
A chicken in every pot, and a Check on every PR
The Checks and the idea of creating a Check Suite ended up being so successful that we got great interest across the engineering organization from people wanting to write their own Checks for their own repositories. After briefly looking through the code of the Probot app, I realized that nothing about this app had code that limited it to just our codebase. Only the Checks themselves had specific logic, and even then it was enforcing patterns and didn’t tie themselves to any one repo.
So after a small bit of refactoring to organize existing Checks a bit better, we added a yaml configuration that would allow individual GitHub repositories to opt-in to specific Checks. Probot already has support for reading from the .github/ directory, so we added a config named .github/pull-request-bot.yaml and placed configuration options there, including the list of Checks to enforce for the repo.
Now, anyone could write any Check they wished, deploy the application, and opt-in to enforcing it on their repository.
It’s still early days in this brave new Check world at Compass, but I expect this to be a great way to make sure we can scale our engineering operation while maintaining high quality standards.
Interested in what we do here? We’re hiring!
