Visual Regression Testing
what, why and how
“When we talk about visual regression testing, we are concerned with preventing unintentional changes to our application’s visuals. If we change the styles of a Button component, how can we be sure that we haven’t messed up the styles of our ButtonGroup?”
Regression testing, in general, is validating that the changes we introduce to our source code do not have an unexpected impact on our system. With visual regression tests we are aiming for the same goal, but we are more concerned about the interface that is being presented to our final user rather than how functional our system is.
At the end of the day, what is important is what our user is seeing and what it is interacting with. Essentially, visual regression tests allows us to understand the changes that will be perceived by our users through screenshots and content comparison between those images.
Motivation
We might have many tests written in our codebase already, which should cover many scenarios and give us some confidence about our codebase. So what are the motivations behind introducing another type of testing to our projects? Here we are going to explore some of the interesting motivations behind implementing visual regression tests.
Visual Bugs
Many times in our projects, bugs occur due to how the interface is presented. We might have a fully functional component in our system and yet the user could be impaired to interact with it somehow.
For instance, think of a Dialog with a text that is longer than the predicted on the initial layout. Then, unpredictably, it starts to cover one of the Dialog's action button, making it impossible for the user to click it. The button click might be working as expected, but our user will never be able to do so, due to a visual bug introduced to the system.
In the image below, we can see a similar problem. We have a form section in which the dropdown arrow button is not possible to be interacted with since the text is covering it. Another example of a visual bug.
Functional vs Design
“While it’s pretty clear that you need to test your application for functional regression, a lot of people make the mistake of assuming that this covers the visual element as well. There is a clear distinction between the two, and compared to functional regression testing that checks if any new code has impeded the functionality of a page, visual testing is specifically about deviations in appearance.”
Most certainly, functional testing in general (unit testing, snapshot testing and else) are essential to guarantee that our system parts are functionally working as expected. However, they are not able to give us the guarantee we need about the interface that is presented to our users.
Therefore, they are not able to detect problems that are specifically related to it. Those problems might be just small inconveniences and standardization problems but also obtrusive and impeditive issues for our users (as shown in Visual Bugs). These tests can not catch information on the user experience.
Take for instance the example below, where we have a screenshot of a modal. It could have easily passed our functional tests, since in terms of functionality it is working as expected; everything we wanted was rendered and the expected actions binded to the buttons are working properly.
However, when we are talking about the user experience, the final result we have here is not good. This could be introduced to the system when making CSS changes that applies to many system parts or maybe some library version update. With visual regression tests it becomes much easier for us to capture those problems before our user's do.
Visual Compatibility
In web development we have a huge range of browsers, mobile devices and screen sizes to work with. A single page in our system might be rendered by Chrome or Firefox; accessed by small, medium or big desktop screens; navigated by different operational systems in mobile devices with also a lot of variable screen sizes.
We are talking about a huge amount of possible combinations to be tested. Therefore, it becomes impossible to manually tests the interface of all those variations; and that is just for one screen, imagine for a whole system.
Especially when we are talking about responsive systems, visual regression tests become more important to identify compatibility problems when rendering pages. Moreover, some of those cross-platform problems that might be harder to perceive could be identified with those tests.
Visual Changelog
With visual regression tests we can easily have a visual changelog of changes in our system. That might especially be interesting when working with components libraries or design systems; a resource that is complementary to functional changelogs.
Cooperation
It might be an interesting tool for cooperation between developers, designers, project managers and else. By detecting in an automatized manner interface flaws and acquiring visual "evidencies" of the problems, it makes it easier to share feedbacks and facilitates cooperation within teams to identify action points that might be necessary due to the problems detected.
Automation
Considering all the motivations explained before, let's dive into how we can have an automated process of implementing visual regression tests in our projects and some good practices related to it.
The Steps
- We write our testing scenarios; we define what we are going to capture in screenshots and where to capture them from our application. Also, those scenarios should handle any necessary user interaction.
- We run the testing tool in our application in order to capture all the screenshots, based on the previously written scenarios.
- The tool runs an algorithm to compare the recently captured screenshots with the older ones; the reference screenshots. It then generates a report containing all the differences found in the comparison.
- A revisor analyses the report and identifies wether the changes introduced, if any, were expected. If any changes are necessary, once they are made, we return to Step 2.
- After all the necessary changes have being applied and the final result is as expected, the reference images must be updated for future tests run.
The image below explains visually the usual visual regression testing workflow, as explained above.
Good Practices
- Automatized tests should be able to capture dynamic content and replicate user interaction;
Take for instance a scenario in which we have a dialog in our project. We need a tool that is going to allow us to trigger the action that will open that dialog for us to run our test scenarios; otherwise we are just testing a button.
- You should not rely on tools error configuration ratios when comparing images. The most important is wether the human eye can perceive the difference that was introduced to the system;
We might have scenarios in which tools might encounter a very big amount of difference between the images but that the final result for the final user is the same; like shown in the image below. At the same time, you might have a test pass due to a small error ratio configuration even when a critical but very small layout change was introduced to system.
- Prefer validation of interface pages rather than components individually.
That makes it easier to have a higher test coverage and smaller tests. Also it makes it possible to test your system components together as a whole rather than isolated, in which some problems might not be presented.
Tools
Choosing the right tool for the job is not always easy. For visual regression testing, there certainly is not a better tool for everyone at every moment. So before jumping into any tool, it is important to have in mind:
- What am I trying to achieve when implementing visual regression tests in my project?
- Does it make sense to my current scope? How necessary is it?
- How much effort am I willing to put into it?
- Can I afford a paid solution? Should I, considering the answers to the questions above?
In this reference you can find a very complete list of tools and resources about visual regression testing. Here I am just going to overview some of the popular tools that I had the chance to study.
Paid Tools
The paid tools for visual regression tests are very interesting solutions. You are able to upload all the screenshots to their CDN and have many plug-and-play integrations ready to make your life much easier. Also, as paid tools, you can always count with excellent docs and support.
However, all this come at a high costs. They do have free tiers, which are good for testing them or for very small projects. The additional charges are for the amount of snapshots taken and when we are talking about scale and continuous delivery that count grows very quickly; and so does the price.
Percy is one of the most famous tools for visual regression testing and it is very flexible and easy to use. It comes with a very wide range of integrations and has a really nice and collaborative dashboard; where changes can be reviewed by the team and approved/declined.
Chromatic was created by the maintainers of Storybook for Storybook. It is out of the box integrated with Storybook and extremely easy to add to your project if you are already using it. As Percy, it has a very nice and collaborative dashboard, very similar in many ways.
Opensource Tools
With opensource tools you are not going to have interesting features such as the dashboards, reviews and CDN hosting that the paid tools provide. However, you can probably find a project that fits your needs and is going to allow you to achieve your objectives regarding visual regression testing.
BackstopJS is a very popular visual regression opensource project, well maintained and very complete and flexible to be used in many different scopes of JS web projects. It renders screens with Chrome Headless and handles interactions with Puppeteer scripts. It generates a nice HTML report with the differences found and also provides us its Docker Image to help us run the tests in CI.
Loki is a very easy to use tool for projects that use Storybook and also runs with Chrome Headless. If you are already using Storybook in your project it is just plug-and-play. It has a very complete documentation and also has an easy CI integration.
This one is a module that adds visual regression testing to Cypress. Cypress is a very powerful testing framework and if you are already using it in your project you can easily add visual regression testing with this tool, integrating it with your existing tests.
Visual Regression Test are a very powerful way of validating regressions on projects and we have many tools at our disposal to help us accomplish it. Having them in a project do envolve quite some effort so even though the results might be very interesting, it is important to identify if that effort makes sense for your actual scope.
Hope it helps! 😉
