Making visual regression useful

The idea of having an automated way to understand, preview and review the visual changes you and your team have made on your website, over a fixed period and between builds is a concept that many jumped at the chance to experiment with.

When we looked at this idea our QAs, Developers and Managers started to salivate in unison. The excitement was real. Testing is a laborious, mind numbing and expensive task for most developers but an (for the most part) unavoidable one.

Several years after researching this area I’m still being asked by the small group that were aware of the work, how can they could also apply this concept to other projects, how to get it working and what is involved. I found myself telling them it wouldn’t be worth the effort, until recently I realised that I was doing it wrong and many others too.

Testing all the things!

The problem is we wanted to test everything and there for we were testing nothing. We already took a lot of screenshots for every device and resolution and we had a dashboard to make the diffs easy to view. It looked like this…

Pretty sweet looking but utterly useless. While seeing the thumbnail of the whole page go red or yellow is fun and all it’s providing no meaningful or useful data to anyone. If the header changed from 65px to 70px then the whole page would need to move, so comparing these two images would result in everything appearing to change. Throw the unpredictability of content or the database changing over time then you’re in a bit of a pickle. Most of your tests won’t be useful and It’s also just too easy to follow the white rabbit and start trying to make your environment more stable to try fix this problem.

I’ve seen the approach of snapping the whole page used too often. BBC, Facebook and Google have publicly shown that they’ve attempted or have used this approach before. In these cases it feels like this is as a tool to trim the fat of the already large stack of testing requirements and understanding what has changed rather than immediately understanding the outcome.

Components to the rescue, but not really.

The other approach is to look to components. Write your codebase so that you can reliably load a component in isolation. Test for regressions with stable (or mock) data and the problem is solved. Global changes no longer paint your visual regression report in a blinding shade of red and your QA team or your fellow developers can be happy as your component will always show regressions and thus your meaning is restored.

There is a bit of a big caveat to this however. This is great for some but it still leaves others faced with rewrites or expensive restructures to allow this to be a possibility. I feel like those that I’ve heard “looking into” visual regression reached this point and realised a rewrite or restructure was too far away of a goal to achieve and thus it falls back into the technical backlog in hope that it happens in the next few years.

I know this because I faced the same problems. The initial investment of getting the process working alone can be a big leap for some and to understand the need for a rewrite or restructure of any kind is a bitter pill the swallow.

While React and similar are making truer isolation easier it’s still not a silver bullet. It’s likely this is the reason why we’re not seeing so many companies making full use of visual regression.

Segmentation to the rescue.

As ImageMagick (The backbone of most visual regression tests) is a tool simply for processing images its unlikely this part of the process can be improved. We can’t make this part that much smarter without putting something on top.

Therefore the solution I believe is to stop perceiving the visual regression test as a test for the whole page but instead a test for the components within it. If you can’t isolate your code. Isolate your screenshots.

• No rewrites
• No significant technical overhead for implementation
• Meaningful test results
• No impact from external components or positioning

Everyone wins.

Visual regression is a very effective tool and is here to stay, especially with this approach and I would also like to give huge applause to garris shipon of LinkedIn for the development of BackstopJS. You’ve done the development world a favour. If I was to recommend to anyone of a solid approach for visual regression testing, this pattern would be it.

If your using a codebase where components can already be rendered in isolation an approach where adding a process such as Diffux CI to your pipeline.

TL;DR — Don’t just screenshot your entire pages. It’s not always realistic to rewrite your codebase to start testing with visual regression. Isolating your screenshots and not your code is probably a better option for you. You may also find BackstopJS useful or perhaps write something similar to suit you.