Soluto by asurion
Published in

Soluto by asurion

Why jest snapshots can be harmful — practical examples

What are jest snapshots?

Jest is a popular framework for testing JavaScript, and Jest snapshot is a method for creating an HTML entity map for regression tests. Ideally, a developer can create a jest snapshot for each output\component and create a test that easily compares it in each run. If any change is detected, the test fails.

Snapshot tests are very easy to use in a jest environment. Basically, it looks something like this:

import { render } from '@testing-library/react';
import { TestComponent } from '../some/path/to/TestComponent';
const { container } = render(<TestComponent />);
expect(container).toMatchSnapshot();

Put that code in a jest test file, and a snapshot file will be created on the first run. Commit this file to git, and for every run, the component will be rendered by jest, and the output presented as an HTML file. In the future, if any changes are made, the test will fail and the developer will be able to see the differences between the old HTML and the new one, and fix the problem or update the snapshot.

Jest Snapshots can be a great tool, but they have their flaws. Since we make a lot of changes to the UI, especially when using styled-components, there are many changes that may result in false negatives, for example, class name changes — in styled components, those class names are generated automatically, and change with every small change to the UI.

If using the component library, even minor changes in the library can cause multiple class name changes. This clamor of false-negatives might cause developers to get used to just hit “u” to update the snapshot. Additionally, the sheer volume of snapshots and the snapshot HTML code can overwhelm even the most fastidious developer.

While even jest snapshots have their use-case, for UI changes, snapshots can cause more harm than good. To show that, I will now give several real-life, practical examples that demonstrate the problem of relying on snapshots for unit testing.

For my first example, here is a jest test of a react component called toast:

it('Toast with action button should match the snapshot', () => {
const { container } = render(
<Toast
message="Default message sample"
actionButtonText="dismiss button"
buttonAriaLabel="mock-aria-label"
isOpen={true}
/>
);
expect(container).toMatchSnapshot();
});

The test is rendering the component and then creates a snapshot of it. This is what the snapshot HTML code looks like:

exports[`Toast Component Snapshot Tests Toast with action button should match the snapshot 1`] = `
<div>
<div class=”ToastContainer-mx-asurion-ui-react__sc-17dd48v-0 cmtsKH”>
<p class=”ToastMessage-mx-asurion-ui-react__l91648–0 cZbSWW”>
Default message sample
</p>
<button aria-label=”” class=”ToastActionButtonElement-mx-asurion-ui-react__sc-15jpslk-0 dKfYUl”>
dismiss button
</button>
</div>
</div>
`;

This test reaches 100% coverage.

Cool, right?

Well… no. It’s not.

The problem is that developers seldom look at the output. Let’s try that right now: take a hard look at the jest snapshot output. Can you spot the issue?

The issue is that the aria-label isn’t there, not really. I found it when I deleted the snapshot and wrote real unit tests that asserted the output based on the component attribute. It’s harder to achieve high coverage because it requires activating each component attribute manually and asserting the output. It’s tedious work, but when I’ve tested the aria-label with actual assessment:

it(‘Toast with buttonAriaLabel property should match the aria-label text’, () => {
const { getByLabelText } = render(
<Toast
message=”Default message sample”
actionButtonText=”dismiss button”
buttonAriaLabel=”mock-aria-label”
isOpen={true}
/>
);
expect(getByLabelText(‘mock-aria-label’)).toBeTruthy();
});

That test failed. Then I noticed the bug — no aria-label.

Yes, if I had taken a long hard look at the snapshot, I should have probably seen it, but why would I? The snapshot file is long, with a lot of HTML code, and the coverage report doesn’t reveal any issues.

Let’s look at another real-life example. Here’s a test for a component named Icon. That tests the output of a snapshot.

it(‘should render the title that is passed’, () => {
const { asFragment } = render(
<Icon src=”Dashboard” title=”This is the title” />
);
expect(asFragment()).toMatchSnapshot();
});

And this is the snapshot:

exports[`Icon component Functional tests title should render the title that is passed 1`] = `<DocumentFragment>
<span class=”Icon__SvgSpan-asurion-ui-react__sc-c22xzd-0 fPXbhN”>
<svg
aria-label=”This is the title”
viewBox=”0 0 24 24"
xmlns=”http://www.w3.org/2000/svg"
>
<path
clip-rule=”evenodd”
d=”M6.8092 8.1067C6.45084 7.74919 5.87052 7.74945 5.51248 8.10749L2.26872 11.3512C1.91043 11.7095 1.91043 12.2905 2.26872 12.6488L5.51248 15.8925C5.87078 16.2508 6.45169 16.2508 6.80999 15.8925L10.0537 12.6488C10.412 12.2905 10.412 11.7095 10.0537 11.3512L6.8092 8.1067ZM13.9463 12.6488L17.19 15.8925C17.5483 16.2508 18.1292 16.2508 18.4875 15.8925L21.7313 12.6488C22.0896 12.2905 22.0896 11.7095 21.7313 11.3512L18.4875 8.10749C18.1292 7.74919 17.5483 7.74919 17.19 8.10749L13.9463 11.3512C13.588 11.7095 13.588 12.2905 13.9463 12.6488ZM8.10749 6.80998L11.3512 10.0537C11.7095 10.412 12.2905 10.412 12.6488 10.0537L15.8925 6.80998C16.2508 6.45169 16.2508 5.87078 15.8925 5.51248L12.6488 2.26872C12.2905 1.91043 11.7095 1.91043 11.3512 2.26872L8.10749 5.51248C7.74919 5.87078 7.74919 6.45169 8.10749 6.80998ZM15.8925 17.19L12.6488 13.9463C12.2905 13.588 11.7095 13.588 11.3512 13.9463L8.10749 17.19C7.74919 17.5483 7.74919 18.1292 8.10749 18.4875L11.3512 21.7313C11.7095 22.0896 12.2905 22.0896 12.6488 21.7313L15.8925 18.4875C16.2508 18.1292 16.2508 17.5483 15.8925 17.19ZM12 4.21498L10.0537 6.16123L12 8.10749L13.9463 6.16123L12 4.21498ZM4.21498 12L6.16123 13.9463L8.10749 12L6.16123 10.0537L4.21498 12ZM10.0537 17.8388L12 19.785L13.9463 17.8388L12 15.8925L10.0537 17.8388ZM19.785 12L17.8388 10.0537L15.8925 12L17.8388 13.9463L19.785 12Z”
fill=”black”
fill-rule=”evenodd”
/>
</svg>
</span>
</DocumentFragment>
`;

Can you spot the bug?

The bug is that the title is missing. But since there are a lot of snapshots in these component tests, the snapshot itself is hidden in a mega file of thousands of lines of the snapshot file. Good luck finding your snapshot in there.

When we switched from snapshot test to real test, we spotted the bug:

it(‘should render the title that is passed’, () => {
const { getByLabelText } = render(
<Icon src=”Dashboard” title=”This is the title” />
);
const svg = getByLabelText(‘This is the title’);
expect(svg).toBeTruthy();
});

While you can blame the developer that created the snapshot or modified the snapshots without fully understanding what the changes mean, I think you should leave the poor guy alone — this process is destined to fail. You can’t rely on the developers to tediously check HTML code hidden in a file. You can’t rely on the developers to understand when the class name change is because of a major UI change, or if it’s just a minor update of a dependency of a component library. And if developers get used to one-click updating the snapshots, they’ll do it even if something broke.

Unit testing is a dreary process, and this is why we have a coverage report: to make sure that every component is tested — every fork and every line. When you render everything, you will achieve high coverage, but that requires you to rely on the developers to scan every HTML line of the snapshot, and in real life — NOBODY DOES THAT. Bugs will occur, without you knowing it.

So there you have it. The examples I discussed illustrate the problem: while it can be useful, jest snapshots can be harmful and might lead to misguided confidence in your code due to their high coverage. It’s either using it and strongly monitoring the developer’s behavior and the changelog to verify that every snapshot update wasn’t breaking anything, or just avoiding it and let the developers verify the real output that they want to get.

Many thanks to Giora Guttsait, Daniel Amram, and Neil Giarratana for the inspiration & help on this article.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store