Visual testing web UIs with Mugshot

Imagine you’ve just finished building a really nice dropdown component. It’s covered by unit tests, it has some really snazzy styles going for it, and you’ve manually tested it in Chrome to make sure everything looks good. To go the extra mile, you open Firefox and you check it there as well.

This sounds like a reasonable approach. You commit to it for every future version of the dropdown component. After all, it takes a couple of minutes to open a web page in two browsers and glance over the result. But, as you’re only human, you might miss things. You might not notice a border getting thicker, or you might overlook the corners of a button getting rounder. You might miss a color change because of a poorly calibrated monitor. You might even believe that it was always like that.

By now you’re probably thinking that these things don’t matter. If they were so small that you missed them, surely the user won’t notice them! I’ll leave the argument of subjectivity out of this, but what will happen when you move past maintaining a single component to maintaining a full app? What will happen when your team grows from one developer to a dozen?

Checking the correctness of your app and scaling that as the app and the team grow sounds reasonable — you just write more tests and teach more people how to write them. But scaling a manual and error prone process seems bound to fail. What if we could automate it, the same way we can automate checking app logic?

First, let’s break down the process a developer would go through when it comes to checking the “visual correctness” of an app:

1. Open the browser.
2. Navigate to a certain URL.
3. Interact with the app (log in, fill in some forms, open some dropdowns etc.).
4. Inspect the visual aspect of the page and compare it to something we remember as being correct.
5. If something seems off we might have to checkout an early revision and compare side by side because we can’t easily remember the previous state.
6. If there are indeed unexpected changes we find and fix the code that caused them.
7. If the changes are expected then we remember them as the new reference.

I’ll split this process into two parts and talk about them separately: automating browser interaction and automating visual testing.

Automating browser interaction

Whether you’re already familiar with this step, or this is your first time writing such a test, I invite you to walk through a simple example of using Mocha, Chai and WebdriverIO to check that an image appears on a page.

I don’t want to focus on tool choices here, use whatever is convenient for you. As we’ll see later, Mugshot can work with anything so don’t get stuck in the example, it’s just meant to highlight the basic flow of a browser test.

We start with creating a WebdriverIO instance by connecting to a local Chrome browser. This can be a Chrome node connected to a Selenium grid, a geckodriver server for Firefox, or anything else that understands the webdriver spec. Setting this up is outside the scope of this article, but you can find many great resources on the web, such as this and this.

After getting the instance, we use it to navigate the browser to an URL and then ask it if a certain image, identified through a CSS selector, is present on the page.

A simple browser test using Mocha, Chai and WebdriverIO

While this test will guard us against accidental removal of the logo from the page, it doesn’t tell us if the logo changes.

Automating visual testing

There are many projects out there that aim to solve this. Most of them are frameworks that come with their own test runner, their own assertion language and their own way of doing things. This means that you will need to learn a new way of writing tests and figure out how to update your build pipeline to run the new tool.

Mugshot is not a framework. It’s a visual regression testing library made to fit right into your existing tests. It doesn’t come with a test runner, it doesn’t manage its own grid of browser instances and it doesn’t force a new assertion library onto you. What it provides is a simple API call:

Adding visual testing capabilities with Mugshot

Because each framework has its own API we wrap the browser instance into an adapter that translates that API to a common interface that Mugshot understands. After that it’s business as normal — we are in charge of controlling the browser and pointing it to the page we want to test. When we’re done with that, we call Mugshot to take a screenshot of the logo and make sure there are no changes from last time. Mugshot will do the following:

1. Look for an existing screenshot called logo.png located in tests/screenshots.
2. Ask the browser to screenshot the element identified by the CSS selector.
3. Compare that screenshot to the baseline file from step 1.
4. If any differences are found, a logo.actual.png and logo.diff.png are written in tests/screenshots and a failing result is returned.
5. If no differences are found a passing result is returned.

Here’s what happens if I make some changes to the logo:

Mugshot failing result

The differences have been highlighted in the diff image and a new file has been created with the actual screenshot. If we want to update the baselines (the “correct” state) we just replace logo.png with the new logo.actual.png.

And that’s it. We’ve just added visual regression testing to our existing tests with one simple call. We didn’t need to make any modifications, apart from choosing an adapter that bridges between Mugshot and whatever tool we’re using to control the browser. Such adapters are easy to write and if you want to implement your own, or add a new one for a different tool, Mugshot exports a suite of contract tests that you can use to validate your implementation.

Golden masters

Visual tests are great for checking things like styling and cross browser consistency. But they also work really well as golden master tests, especially if you’re trying to refactor a codebase you’re not familiar with.

A golden master is a snapshot of your entire app (or the part that you want to refactor) without including details about how the app is structured. This might sound similar to Jest snapshots, which you can use to snapshot the DOM of your app. The downside of using DOM snapshots is that they will most likely change while you refactor the app — div containers might move around, markup could be slimmed down and component trees might be restructured. This makes them flaky and you might end up treating them as noise.

A visual golden master on the other hand is much more versatile since you can restructure your app without affecting its visual result. Take a screenshot of the page you want to refactor and use it to guarantee that you’re not breaking anything along the way. This will allow you to focus on understanding the underlying design and writing more granular tests that will lead the design in the desired direction.

A note about React

While testing a React UI won’t be any different, the React ecosystem offers us some nice tools that we can use to automate testing even further. Using a component playground like react-cosmos or storybook can allow us to easily iterate over every component fixture/story and create a test using Mugshot.

Thank you for reading and if you want to learn more about the API and Mugshot’s features please head over to the GitHub repo.

NiGhTTraX/mugshot