Getting started with Automated Visual Testing
When working in teams with mixed skills one of the biggest technical challenges I’ve faced is managing the changes to how things render visually across a codebase. Often you change something and it causes unintended problems somewhere else. This is where Automated Visual Testing has become invaluable.
What is Automated Visual Testing
Automated Visual Testing (or Visual Regression Testing) is designed to automatically check that your user interface looks the way it’s supposed. It does this by checking and comparing what pixels are being rendered much like how you can see the differences in images in tools like Bitbucket.
Replacing bad tests
Before setting up Automated Visual Tests we used Snapshot Tests to take snapshots of our component output to check the changes were necessary. This was a terrible way to confirm the output was correct — it relies on making sure people can visualise the outcome of combinations of CSS. The outcome we are typically looking for with tests like these is to ensure that what the user can see is what we are expecting. We don’t need to know how the visual rendering is achieved.
By switching to Automated Visual Testing we were able to replace checking visual output by looking for CSS selectors, rules or HTML elements with a test that checks the pixel output of the components.
Team safety
When working with teams who are not highly experienced with CSS, or visual design, these tests provide an additional level of safety by visually demonstrating the differences, particularly those that might not be obvious to some members of the team.
Some people are better at noticing small visual changes than others, for example, changing the margin by a few pixels in a component might be obvious to one person and not another. Relying on a person’s “eye” to pick up these changes often results in mistakes. If instead, you make the most of a visual test the computer can let you know exactly what pixels you have changed and flag any unintentional adjustments before they reach production.
Theming and Variations
When developing a component library one component can often do many things. For example, you might be able to toggle on and off different elements, like timestamps, or images. Or you might be applying different styles or themes depending on where it’s used, or what product it’s for. In these scenarios, a visual reference for each version or theme helps to maintain consistency and reduces the need to manually check across multiple products or themes.
Setting it up
There are several tools you can use for Automated Visual Testing including:
- Percy (https://percy.io/visual-testing)
- Chromatic (https://www.chromaticqa.com/)
- Applitools (https://applitools.com/storybook)
Alternatively, you can setup a custom implementation. At SWM Perth we use jest, puppeteer, canvas, and resemble.js. We extracted the code and put up a visual regression testing repo on Github you can check out. The code does work, we just haven’t set up the build process in the Github repo yet.
Our custom implementation works primarily with Jest and Puppeteer as the core elements (and is designed to work with CSS-in-JS libraries). In essence, Jest does a server side render and creates a static result. We then set up localhost with a random port for puppeteer. This renders the static result and takes a snapshot of the component. From here we take the snapshot and use resemble.js and canvas to create a diff of the file before returning the results.
See the visual regression code on Github.
The benefits of this custom approach are that we can set up VR tests for any of our components, you don’t need a Styleguide or page for this to work which means it is very flexible. The downside is that there is no Client Side JavaScript because we are relying on the server side render. Admittedly this is problematic in a couple of scenarios, video players are a good example, but for the majority of our components, it has not been an issue.
Considerations
Automated Visual Testing is fantastic, but there are some things to keep in mind.
As far as tests go, they do tend to take longer than more traditional, lightweight approaches like unit tests. For this reason, you want to consider the value of each test you add in comparison to the time it takes to run.
Figure C shows the times for our VR tests, we have 925 tests and it takes approximately 0.65s per test.
The second major issue that people often face is that 1:1 pixel matching is not ideal. Human eyes cannot perceive minute changes as effectively as a computer algorithm. So while something might look identical to us, to the algorithm it can look completely different. This commonly occurs with anti-aliasing or with CSS like gradients, box-shadows etc. While I don’t personally believe this is a reason to not include automated visual testing as part of your testing process, if you hit these problems regularly, tools like Applitools (mentioned above) have made use of Machine Learning to provide more useful results and ignore less useful changes.
In my opinion, automated Visual Testing is extremely valuable as part of a development process, especially when working in teams on large codebases that change regularly, or has many themes/variations to account for. I cannot count the times these tests have saved us from landing bugs on production. If you have the capacity/opportunity to implement this in your project I highly recommend you give it a go.
Enjoy testing
Mandy 💜
