The end of visual regressions

How visual diffing with Happo allows you to move faster with more confidence.

Update (July 13, 2016): Diffux-CI has now been renamed to Happo

Are you ever worried that your CSS changes will break the styling of an element? Are you scared that your code refactoring will result in visual bugs? Do you find it difficult to understand how a code change affects the appearance of your components? If you answer yes to any of the questions above, you should consider introducing a visual diffing tool.

At Brigade, we are finding visual bugs before they are merged with the help of Happo. This command-line tool targets JavaScript-powered websites (such as those built with React) and is easy to integrate with most continuous integration systems (or “CI” for short). It renders each component in isolation, takes a screenshot, and diffs that image against a previous screenshot for that component.

Diffux CI in action on the Brigade codebase.

Happo is a continuation of Diffux, a tool that Joe Lencioni and I first introduced it at a lightning talk @ Apartment List and later also showcased at Camp Sass 2014. Like many other tools out there, Diffux was all about diffing an entire web page. That’s all fine, but if you really want an automated tool for visual diffing, you need to isolate diffs. With Diffux, you have to manually figure out which component changed within a page, but with Happo you get this information back directly.

How we use Happo

We use Jenkins as our CI environment. On each commit, we kick off a Happo check. This check will run a script that performs the following operations:

• Check out the commit prior to the commit we’re testing
• Precompile the JavaScript bundle
• Run happo

These three steps make sure that we have a set of baseline snapshots to check against. The second step could have included precompiling CSS too, but we already bundle our CSS with the JS code with help from Webpack and sass-loader.

• Check out the commit we’re testing
• Precompile the JavaScript bundle
• Run happo

After these steps, we know if we have diffs or not. If we do, we run a few more steps:

• Upload diffs to a publicly accessible URL
• Post a link to the diffs on the commit being tested

We use Gerrit for code review, so the last step includes posting a comment back to the change set with a link to the diffs. Here’s what that looks like in the script:

At Brigade, we post a message back to the commit if diff(s) are found.

The diffs that are uploaded contain the baseline screenshot, a diff image representation, and the new screenshot. Here’s an example of what that can look like:

An example of a Diffux CI diff. From right to left: baseline screenshot, diff, new screenshot

Not all visual diffs are bugs. In fact, many are expected as a result of applying a commit. When such commits are pushed, the diff has an added benefit of showing reviewers how the code change relates to the visual appearance of the application. That’s why we don’t automatically -1 a commit if we find a diff. It’s up to the author (or reviewer) of the commit to look at the diff and determine whether or not it’s a regression.

Example script

An example script is provided in the Happo repo that you can use as a starting point if you want to integrate with your own project:

Features

Happo uses Firefox under the hood. The screenshots you get are from a real browser that real people use. Goodbye PhantomJS.

Components can be checked responsively by rendering them in different window sizes. No more manual diffing on smaller screens. [docs]

You can make diffs available on a public URL by uploading them to an S3 account. All you have to do is provide your S3 auth keys. [docs]

To install Happo, simply run gem install happo. The Ruby gem comes bundled with a handful of useful command-line tools. [docs]

With some simple tricks, Happo can run in a headless environment. CI systems love this. [docs]

Since introducing Happo at Brigade, we have been moving faster than ever. We refactor CSS with confidence, we understand how our code changes affect visual styling, and we don’t have to manually test the appearance of our components. It has become a vital part of our workflow and we believe that it has a bright future. We’re eager to see how it can help you. Please file issues if you find bugs or want to discuss enhancements. And keep them pull requests rolling!