Embracing spurious diffs

If you’ve ever tried implementing visual regression testing for your application, you know that spurious diffs can be a real pain. These are minor differences in the UI screenshots causing an example to show up as having a diff despite not having changed.

Spurious diff for a screenshot taken by Chrome. Two pixels in the lower-right corner are different.

There are many things you can (and should) do to reduce the level of spuriousness. Disabling animations is the first one — you don’t want your screenshots to depend on timing. Then comes making sure to wait for asynchronous events — you need to wait for images to download before you take the screenshot. A tricky and subtle one is disabling scrollbars — these can cause major pain to anyone trying to take consistent screenshots.

But despite applying all these techniques, you’re still going to end up with spurious diffs (yep, you are). Chrome, for instance, won’t consistently render the same rounded corners when there is an image background. Firefox has the same issue, but only when the image is an external one, and much less frequent. And this is all assuming your own code is completely free from spurious things, which is rare.

The silver lining in all this is that even though there’s known spuriousness, the spuriousness is pretty consistent. At happo.io, we create a hash for each screenshot based on the image file content and then use that when saving the file. While tracking down yet another spurious Chrome issue, I noticed that the same spurious diff kept coming back when looking at these hashes. For instance, this couple of screenshot URLs was a common diff:

https://happo-io.s3-us-west-1.amazonaws.com/8e454bb9c14d8dc539b3a7f094db9475
https://happo-io.s3-us-west-1.amazonaws.com/20bcb51d46e307b8d7698cd0b47663fc

The consistency of the spuriousness made me think of a simple solution to this. By storing pairs of “equal” URLs, I could prevent the same spuriousness from ever showing up as diffs again. This is what the “Mark as equal” feature at happo.io does.

Screen recording of marking a diff as equal.

Any diff that’s been previously marked will continue to be so for future happo runs. Over time, this means that your diffs will have a higher probability of showing a true regression. More confidence, less frustration. Win-win!

Give happo.io a try! We have room for a few more beta customers. Sign up at https://happo.io