Why I stopped using snapshot testing with Jest

These days a lot of people is using Jest’s snapshot testing on a daily basis. I will show you reasons why I stopped.

We build spectacular apps, check auriosoftware.com

Fragility

We changed a little bit of code and it blew off half of our tests. The problem is that these snapshots are highly coupled with our implementation and as we all know, coupling is one of the biggest pain in software development.

How coupled you asked?

Imagine we have a component and we created a new div container because of a new design requirement. We run the tests and here it is, our friendly output from our failing snapshot test.

We voluntary highly coupled the component’s DOM with our tests. Every time we do even a little change to our code, our tests will fail!

That new div container is there only because of some minor cosmetic change. It did not even slightly touch component’s functionality and it definitely did not change the component’s contract, but still, our test failed.

Some of you might say…

“But this test helps you because you can notice that something is wrong”

Let me introduce you to my working process.

1. I create a new feature
2. I will check “manually” that the UI is properly rendered and all functionality is really working.
3. I will check whether my tests are passing
4. Before every commit I create, I will run git diff and look whether a code is clean and properly tested.

And look at my console’s output, does not it look familiar to you?

This output has the same value as the snapshot test, and yet, it is not that annoying.

I can see that I added only one div element. I will create a commit and push it to the source code repository. Then, I am creating a pull request and guess what.

My colleague will go through the same working process as I did.

Our bugs in the system do not exist because of our luck of snapshot tests, but most likely because of our lack of discipline and professionalism.

Irrelevance (after time)

Let's say you have quite a big project with a lot of these snapshot tests. You created a new feature, you are great, you are happy… until you run the tests.

Well, most of them failed.

It is frustrating, but you check all of these failed tests whether something is wrong and update them. This becomes part of your working process.

A couple of weeks later.

You have become professional snapshot updater, congratulations. You have gained so much experience because you have been doing this whole eternity. Your brain knows every shortcut for updating a snapshot.

Actually, you are so experienced that you most probably don’t even look at these failed snapshots. You will update them automatically because your brain is convinced that nothing is wrong.

These tests became irrelevant because you are completely ignoring them.

No TDD!

This is the biggest obstacle for me because I can’t imagine to code without writing a test first. If you are thinking about creating a snapshot first, well, good luck with that.

Descriptiveness

Seriously, who would not love this syntax?

test('render Markdown in preview mode', () => {
    const wrapper = shallow(
        <MarkdownEditor value="*Hello* Jest!" />
    );
    expect(wrapper).toMatchSnapshot();
    wrapper.find('[name="toggle-preview"]').simulate('click');
    expect(wrapper).toMatchSnapshot();
});

Good tests are supposed to be good documentation. If you don’t know how the component works, you should just look at your tests.

But…

What is the mysterious snapshot? Where it comes from? What actually happens when i click on the toggle preview?

Tests should have 3 parts: Arrange, Act and Assert.

1. Arrange — instantiate the component
2. Act — click on the toggle preview
3. Assert — match snapshot (?)

Do you really know what should happen after we clicked on that?

I am missing something in the Assert part. It is not explicit enough. It reminds me of this scene from South Park.

https://www.youtube.com/watch?v=tO5sxLapAts

Needless to say, there is quite a big chance that you would create an invalid snapshot because you are depending on the correctness of your implementation.

False safety

When I joined a team which has been working on a project for a while, my first question was

“Do you have a tests?”

And they replied (with the proud look in their eyes)

“Hell Yeah”

I was really glad because I had worked with people who did not even try to write tests. So I jumped to the code and saw something…interesting.

describe('<UserForm />', () => {
    it('should render', () => {
        const renderer = new ShallowRenderer();
        renderer.render(<UserForm />);
        const result = renderer.getRenderOutput();
        expect(result).toMatchSnapshot();
    });
});

I call it ‘it should render ’ disease. I saw it everywhere, and why? Because it was so easy to write these tests and let me tell you, managers were pumped how fast the implementation went (yes, it was a greenfield project only a couple of months old).

Why is this a problem?

Because the tests were green!

They were confident enough to deploy all new features with these “tests” because at the end of the day, tests were passing and after all, tests were part of the acceptance criteria of a complete ticket (story), therefore mandatory.

It was fun, it was fast, but it was missing test cases which would test the component’s functionality (contract), therefore it was useless.

I could not imagine changing the existing code because I would not know whether my change broke some functionality, but I definitely knew that I am going to break some snapshot tests.

Summary

I am not saying that snapshot testing is a bad thing…

Just kidding, I think it is pretty bad.

It is not a great idea to couple your implementation details with these tests. At least in my world. I can not imagine a world where all members of a team are being professional enough and being careful and brave to pain attention to these failed snapshot tests.

I would look around for a different solution, maybe something like Chromatic, because it is not coupled with DOM.