AyeSpy, a new way to test for visual regressions

by Matt Lowry

Bill Gates famously said, “I will always choose a lazy person to do a difficult job because a lazy person will find an easy way to do it.”

At The Times, there is an incredible amount of business value placed on the aesthetics of the site. There have also been past incidents where CSS bugs have caused rollbacks.

With this in mind, traditional `automated` functional testing with selenium is ineffective to find these defects — in addition to being slow and high maintenance. To add to the problem, The Times release far too often to make manual verification possible.

This is where visual regression tools shine through. Their sole purpose is to give confidence that the applications under test are visually correct.

So what is visual regression?

There are 3 main parts to understanding how visual regression works.

1. Baseline:

A set of images that define how the application should look, based on previous recordings.

2. Latest:

A set of images detailing how the application currently looks.

3. The comparison:

Once we have both the baseline and the latest, we are able to perform a comparison between how the application is supposed to look and how it looks now. If there are differences, the build will fail, and you will need to approve the changes to update the baseline images once more.

We have used a number of visual regression tools within the Times Tooling team at News UK and each proved to have limitations.

A core testing principle that we believe at ECS Digital is you need to be testing as close to production/end users as possible.

Headless browsers such as phantomJS may give you a small performance increase when executing tests, but these browsers are far from how your end users will be interacting with the application under test.

Our first visual regression tool only supported headless browsers. We had several instances where it allowed bugs through, but this only occurred on Firefox and not PhantomJS. This loophole was the reason we decided to move on.

The second tool we tried was what we believed to be the industry open source favourite. After battling with it for well over a week we could not get it running stable or under 30 minutes, which as a developer is an unacceptable feedback loop.

As you can imagine, these inefficiencies didn’t sit well with the Times Tooling team and we decided to address the problem head-on and create our own “hand-rolled” visual regression tool.

Based on our previous painful visual regression experience, we were determined to build a tool that was:

• Super performant
• Lightweight, and
• Made it easy to interpret results

A proof of concept was put together before fully refining the capabilities of the tool. We then waited for priority to allow before creating ‘AyeSpy’ in one sprint.

Four months down the line and AyeSpy has been successfully implemented, gaining approval from our clients and users on GitHub. Whilst the Times Tooling Team engineered AyeSpy, The Sun and data teams within News UK have since adopted it and it’s not hard to see why — AyeSpy takes less than 90 seconds to run 44 webpage comparisons. Other benefits include:

• Only requires a .json config file to run
• Maintenance is low
• Able to explicitly wait for elements before screenshot is taken
• Can integrate the Dom before screenshot
• Drop cookies into browser
• Remove dynamic elements from the DOM
• Tests are farmed out to a containerised selenium grid for distributed testing and consistent state

When deciding to use visual regression, we have found in our experience that the tool works best on reasonably static sites that do not require long user journey to be completed before the screenshot. For example, clicking through a checkout journey would introduce a high level of risk and take away value from the tool. Ideally, you want to load the page, remove all dynamic elements, and then snapshot.

Where you can find the tool?

ECS Digital love to find value for our clients and give it back to the wider community, which is why we make these tools available on open source platforms such as GitHub and NPM.

I will also be hosting a hands-on session and demonstration of AyeSpy at an upcoming DevOps Playground on the 29th of November. Come along to learn more about what the AyeSpy has to offer!

******

Originally posted on ECS Digital’s website