Improving code review on GitHub

A product design process story highlighting several iterative projects following the initial release of the Reviews feature on GitHub in 2016.

In September 2016, GitHub improved code review on pull requests with the Reviews feature. It was an important step forward for the core collaboration workflow on GitHub, and it was just the beginning.

Iterating based on feedback

Following the initial ship of Reviews, our team worked closely with the Support team and Enterprise Sales team to monitor feedback from both dot com customers and GitHub Enterprise (“GHE”) customers. We also listened to developers on Twitter and other social outlets where developers were publicly sharing feedback.

  • Contributors often needed to catch the attention of repository maintainers if discussion in a pull request had gone stale.
  • Maintainers needed a way to require review prior to a pull request being merged to ensure that code wasn’t merged carelessly or without sufficient review.
  • Maintainers wanted to designate certain people as owners for specific files or directories corresponding to areas of responsibility to help distribute the review load across their organization’s teams.

1. Review requests

We decided to begin our iterations with a formalized way for a pull request author to request review from repository maintainers, as it was one of the most common themes we heard from feedback.

Initiating the review cycle

Review requests provided pull request authors a way to signal when their code was ready for review. While Reviews allowed for a formalized review process, often pull request authors aren’t ready for review the moment they opened a pull request.

Reaching the right reviewers

Pull request authors also wanted a way to be able to request review from specific people. Often a project’s maintainers and collaborators list is very large, so making sure a pull request gets reviewed by the right person or people could be difficult.

Refocusing attention in stale threads

When a pull request is initially created repository maintainers often take a quick first pass for triaging with labels and “at-a-glance” review. If the pull request can be easily merged, maintainers will often take action immediately. However, if the pull request requires further discussion, review, or additional changes sometimes the thread may go stale.

Design process

We framed the feature in one basic feature requirement:

  • A way to notify requested reviewers of the review request
  • A way to filter pull requests in the pulls index by review status
  • Display review status in the pull request merge box
  • Should requested reviews be required for merging by default?
  • Should review requests be exclusive? (i.e. review only allowed from the requested reviewer)
  • How many people can be requested for review?
  • Can you request from a whole team, or just individuals?
  • How do I know who to ask for review?
  • What users are shown in the reviewers select menu? Any contributor who is part of an org?
  • Should we provide a way to indicate why your review was requested?
  • filtering pull requests by review status (cut for scope)
  • requiring reviews for merging (cut for scope)
  • web notifications for requested reviewers (cut for product concerns and scope)

Listening to feedback

We started listening for feedback as developers began to use Review Requests. Again, we heard some common themes that helped us measure our decisions and prioritize further iteration.

  1. General UI improvements (e.g. things that were confusing to understand, bugs, etc)
  • After a review is given, I want new commits pushed to dismiss my review and re-request.
  • I want to request review from an entire team.
  • I want to use search filters to see pull requests in my repo index by review status.
  • API support and webhooks
(Note: Twitter — here, and elsewhere — used as an example of feedback received from GitHub Support and other channels. Obviously, GitHub does not rely solely on Twitter for product feedback. 😉)

2. Filtering pull requests by review state

The next code review improvement we decided to tackle was the problem of how to easily get to pull requests where your review has been requested.

Design process

The basic workflow story for filtering was:

3. Team review requests

Organization maintainers on GitHub use the Teams feature to help distribute maintainer responsibilities and focus communication with team @ mentions. Adding Team support to Review Requests seemed like a natural next step.

Design process

Design work began with a simple workflow story:

4. Code owners

With Review Requests, code review on GitHub now had an explicit way to initiate the review cycle, but it was still a manual process ripe for automation.

Who should be reviewing my changes?

In some cases (such as private repositories with company collaborators) workflow considerations like who should review changes to certain parts of a codebase are solved by organizational structure and communication happening at the company itself outside of the GitHub platform. Those structures are then simply carried into and mirrored in the GitHub workflow.

Reducing the triage burden for maintainers

For repository maintainers with a large number of incoming pull requests, often a significant amount of time is spent triaging them with labels, comments, reviews, and review requests to other more maintainers who would be more appropriate reviewers.

Design process

The design process was framed around three key workflow stories:

  • What icon, label, or other styling is most appropriate to distinguish code owners from non-owner reviewers?

Listening to feedback

We received lots of helpful feedback from developers who were using Code Owners in ways we had not considered, or just not prioritized in the initial release.

  • Can review be required from multiple teams?
  • Can I suggest review from Code Owners instead of request or require?
  • Could we have a more elaborate hierarchy of owners with higher level owners able to unblock review required by lower level owners?
  • Many developers were familiar with the pattern used in the Chromium project where Owners are defined in a per directory basis, and wanted to use GitHub Code Owners in the same way (also similar to how .gitignore works).

5. Including Protected Branches support for increased code security

Many organizations use protected branches to ensure code passes required status checks prior to merging to the default branch — a critical protection configured on a per-repository basis to ensure code the production branch is always secure and reliable.

Requiring reviews on protected branches

Once we shipped requesting review from maintainers, we added the ability to require reviews prior to merge, similar to how status checks could be required for mergability.

Restricting review dismissals

To further empower maintainers who needed greater control over their production codebase, we also added a protected branches setting to limit or remove the ability to dismiss a review, a workflow privilege given to maintainers by default.

Dismissing stale reviews when new code is pushed

We also heard from some developers who wanted the ability to re-require reviews when new code was pushed to a pull request after review was given, an extra layer of protection that was vital to their workflow.

Final reflections

Throughout 2016 and 2017 I had the privilege of working with Workflow Team Cactus 🌵on improving code review on GitHub.

Workflow insights

Through this work, I learned a lot about how people use GitHub to collaborate on code together, and how teams do code review. The key takeaway being that there are multiple ways these things happen, and trying to be prescriptive is not as helpful as allowing flexible options that support an individual teams workflow.

  • contributor, maintainer, or admin?
  • personal or organization repo?
  • open source, personal, or work project?

Critical reflections

On the Code Owners features — in hindsight I wish I had the foresight to spend less time exploring and pushing the design comps for the visual UI product direction. I was so excited about the idea of helping making code owners super simple to use that I overlooked the simplest path forward (a plain text file versioned by Git) and it took department leadership to come to this conclusion themselves.

People involved in this work

Many humans were involved in the creation of this work. I’m incredibly grateful to the following people for their involvement in these projects:

URLs

- GitHub features
- Code Review on GitHub features overview
- Pull request reviews Help doc

GitHub blog

- More code review tools
- Review requests
- Filtering pull request reviews and review requests
- Protected branches review dismissal restrictions
- Team review requests
- Code Owners

News coverage

- GitHub introduces code owners for code review
- Own it: GitHub introduces Code Owners
- GitHub launches a Trello competitor, pull request reviews, redesigned profile pages

Gentle Giant. I design software, make art, and enjoy strategic systems thinking. Design at Apollo GraphQL. Previously Netlify, GitHub. Spread love! 🐻❤️⚡️🤙

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store