Reducing UI bugs down to almost zero @ Funding Circle
UI bugs can be a nightmare for frontend devs, playing the whack a mole game of fixing one bug, only to find out a week later that we had introduced two more! Recently we added a lesser known type of testing for our UI that not only reduced the number of bugs we found to almost zero, but it also sped up our development process and increased our confidence in deployments.
When we started the frontend rebuild of Funding Circles’ US Application we thought long and hard about what we wanted to achieve with this ‘refresh’.
Using our experiences with previous projects we wanted to create a solid foundation that would allow us to build features to a high standard, in an efficient manner and in a way that was exciting and pain free to work on as an engineer.
At Funding Circle we work in multidisciplinary teams (FE/BE/Full Stack engineers), meaning not everyone working on the project would have the same experience working with the frontend app and its ecosystem. As a result of this, one of our core priorities was to make the app developer friendly, with enough guardrails in place to give any developer enough confidence that they weren’t doing something wrong or going against coding conventions/idioms.
A significant part of getting this right would come down to starting with a robust testing strategy. From our past experiences, a big area for bugs in the FE have been as a result of unintentional UI changes. These often came from changing the styles in one place that you didn’t know are used elsewhere. To counter this, we adopted visual regression testing from the start.
What is Visual Regression Testing
“Regression — a return to a previous and less advanced or worse state, condition, or way of behaving.” — Cambridge Dictionary
Visual regression testing is an automated process that helps us build and modify user interfaces whilst heavily reducing the risk of unintentional changes.
Take this example, you modify some styles for an existing feature. Unbeknownst to you the CSS class you changed is also used in a feature you didn’t even know existed! Many weeks later someone finally notices. Now not only do you need to fix the bug, you also need to reimplement your original changes, nightmare!
So how does visual regression testing work to prevent this scenario?
- You take images of your applications user interfaces on your main/master branch — this creates a ‘baseline’, these get stored for later.
- When you make changes, you’ll again take images of the user interfaces for this branch.
- You then compare the images you took for your new branch and compare them, pixel by pixel, against your baseline images.
- If they are the same then we are all good; If they are different then you need to decide if the differences are intentional. i.e. Did you mean to move that button 10px down? or if the changes were unintentional? i.e. you didn’t mean to move the button down 10px on every page!
- When you are happy with your changes, you then merge our branch with main/master and the images generated from this branch now become your ‘baseline’
Here is an example of how this might work when accepting a change.
Why is it useful
Not everyone knows everything
Whether you are a seasoned veteran of an application or have just joined the team, there are always going to be parts of a code base you just didn’t know existed.
If you don’t know something exists, how do you know to make sure it isn’t going to be affected by your changes? Sure, you can spend time digging through the codebase, looking for functions calls or variable usages but this is not fail safe and most importantly takes time!
This is one of the reasons why we write unit tests, to fail fast and fail early. Like unit tests, visual regression helps us do this with our user interfaces.
Imagine a world where we can just make a change with very little forethought, run a command and can see if something unintended happened with that tightly coupled UI logic you didn’t even now existed — lovely.
Tests the edge cases
I think we can all agree that manual testing (for the most part) can feel like a drag, but equally, being frontend engineers we appreciate the importance of providing great and consistent experiences to all users!
Does your team support multiple browsers/browser versions? What about different devices and viewport sizes? Various hover/focus states? Different combinations of features and/or support different languages and locales/times zones?
How do you know that change you just did didn’t break an edge case? You either spend hours testing all the various scenarios or you pick a representative sample to run through — all of this takes times.
With visual regression testing we can take images of our interfaces in countless different states and browsers. For example, we can track what a component might look like in both a left to right language (e.g. English) or a right to left language (e.g. Arabic) with minimal overhead.
When something breaks we know about it.
We all miss things from time to time
When manually testing our UIs, there are going to be some changes that are more obvious than others. For example, a prominent call to action disappearing off the page is going to be more obvious than a subtle color change.
What if we accidentally changed the color of some text to be slightly lighter? or increased some spacing by a couple of pixels? Would you spot that? I know I wouldn’t.
If we can’t easily detect these changes, as time goes on we will introduce more and more subtle changes. These changes, perhaps insignificant on their own, take us further and further away from our old, pixel perfect intended designs.
Even worse, because these changes happen slowly over time we are likely to become acclimated to them and therefore miss that this is even happening.
I call this “design drift”.
Visual regression helps us identify those unintentional “design drifts” before we deploy our code, down to a movement of just a few pixels.
The above image shows a series of subtle changes:
- (1) the original
- (2) the spacing between navigation items has increased
- (3) the font size gets slightly smaller
- (4) the font color gets lighter
- (5) we accidentally change the border radius of the button.
The changes here are subtle, and over a long enough time period are hard to notice, let’s contrast the full design drift.
Quite a difference right? Visual regression testing would help us eliminate this.
Tools
If you’re thinking about implementing visual regression in your project there are plenty of both free and paid solutions available.
Free solutions will give you the basics but will often lack many of the features that paid solutions offer, they’ll also require more effort to get up and running. Paid solutions will also usually offer impressive user interfaces that help you better spot differences. It’s also worth noting that many of the paid solutions actually offer free tiers for smaller projects or enough to get started on larger projects.
Here are a few to get you started:
Free Options
Jest-image-snapshots or cypress-image-snapshot
Excellent free choices, especially if you are already using Jest or Cypress in your testing setup.
Easy to setup, works with Storybook and supports iOS & android emulators
One of the oldest tools on the block but packed with functionality. Lots of customisability and reporting options.
Paid Options
Maintainers of the Storybook package, has good change tracking and collaboration tooling.
Storybook focused, offers decent cross browser testing.
Well rounded visual testing tool that claims to use AI to compare images rather than just pixel difference.
Acquired by BrowserStack a few years ago, Percy gives you excellent cross browser testing capabilities
There are a tonne of tools available depending on your tech stack, requirements and budget, you can find a more extensive list here, along with a load of other resources.
How does Funding Circle use visual regression testing
When we started building our new application, we heavily focused on adoption of Storybook for an increased developer workflow, documentation and visibility of our user interfaces (some of our UIs are tricky to see in our production/testing environments).
We decided to go with Chromatic for our visual regression tool of choice, they are the maintainers of Storybook and it made sense given our priorities.
We built our new UI into 3 groups of components:
- Base Components — The basic building blocks e.g. button, typography
- Components — Combinations of our base components brought together to apply “business logic”
- Pages — A mix of our base and non-base components, sometimes containing page specific components
All of these components and their variations are represented in Storybook and Chromatic is configured to take screenshots in a variety of viewport sizes in both Chrome & Firefox.
These tests run on our master branch and are required to pass before merging, if any of the UI components are different we pair review them to decide and agree on the next steps.
As a result of our use of visual regression testing we also found ourselves writing significantly fewer unit tests. When writing new UI components we no longer have to write the “component X should be on the screen” tests as we get that for free via Storybook/chromatic. Unit tests are now reserved for functionality tests such as “does Y happen when I click component X”.
Conclusion
In the 6 months+ we’ve been using visual regression testing we haven’t had a single unexpected UI defect found in production.
Bugs are cheaper to fix earlier on in the process, finding them before merging to master rather than after a week in production gives us so much more time to focus on the important things, and we are already reaping the benefits.
We’ve just started our journey with visual testing at Funding Circle. In the future we’d love to explore more ways it can give us confidence in the accessibility of our applications and also look into what benefits interactive stories in Storybook might bring us.
2022 is going to be a big year for Funding Circle which means we need even more engineers to support the growth of our products! Check out what opportunities we have available on our careers site — careers.fundingcircle.com
