Snapshot Testing. Testing the UI and Beyond (Part 1)

This posts-series aims to provide a complete overview of snapshot testing and its use cases, helping the developers understand what problems it can solve effectively and how to make it part of their software verification tools.

Part 1 is focused on mobile UI development while part 2 goes beyond UI testing and explores other rather interesting non-UI related applications. Part 3 is an iOS hands-on guide that illustrates how this can be done in practice.

What is snapshot testing?

Snapshot testing is just a form of unit testing. In unit tests, you have a function, and given some input, you explicitly assert against the expected output.

The equivalent snapshot test would look like the following and as you can see the main difference is that there is no explicit expected output.

But then, how does this work? Well, you can run a snapshot test in two modes.

  • When running the test in record mode, a snapshot of the output is produced and saved as an artifact on the disk (just a simple text file).
  • When running the test in verify mode a snapshot of the output is produced, it then looks on the disk for a file of the previously saved reference snapshot, it compares the two and if they differ it fails with a nice error message.

When the test fails, either the change is unexpected, or the reference snapshot needs to be updated to the new version. Finally, the snapshot artifact should be committed alongside code changes and reviewed as part of your code review process¹.

Classic assertion-based tests are perfect for testing clearly defined behavior that is expected to remain relatively stable. Snapshot tests are great for testing less clearly defined behavior that may change often. UI components often change in small and trivial ways. The copy is changed, whitespace is added, a border color is modified².

Snapshot testing is by no means a replacement of unit testing, but rather an additional test mechanism³.

Testing the UI

Ιn the mobile community the most popular form of snapshot testing is screenshot testing.

Screenshot testing

Screenshot testing is a visual regression testing method where the snapshot that you produce is an image of a part of the UI (could be a whole screen or just a single view).

Comparing screenshots

When you run the test in record mode, a reference image is saved on the disk.

Record mode

Then every time you run it in verify mode, a newly generated image is produced and compared to the previously saved reference image.

Verify mode

If they differ, then the test fails and the failure diff would look like something like that.

Failure diff

It is the same concept as before when snapshotting the output of a function, but instead of text as the produced artifact you now have an image, and instead of applying text comparison you now apply pixel by pixel comparison. If you can serialize your output into a format, like text or image, that can be saved to disk and you have a way to compare and produce a diff, then you can also apply snapshot testing.

Benefits

By adding screenshot testing into our mobile project, you have already gained quite a lot.

  • Trivial to write, almost effortless to update
  • A fast way to verify what users see and also check how your view looks while developing (No need to fire up the simulator and navigate to a specific screen just to see how the view you are developing looks like. You just take a look at the produced image snapshot)
  • They provide far more coverage than a unit test normally allows (test multiple properties at once)
  • Plug them in the CI and detect UI regression errors

Device configurations

These are all nice benefits but you should not stop here. To maximize what screenshot testing has to offer, you have to make sure that you test against all possible device configurations.

Language

You can quickly test how your view looks like in other languages and produce a screenshot for each language.

Screenshot for the Greek language

Or instead of supporting multiple locales and multiple screenshots for them, you could just use one single pseudo-locale⁴ to simulate strings with increased length and see how larger texts stress your UI.

Screenshot for pseudo-locale

Device size

You can just as easily test how your view looks like on other devices and check, for example, if something does not fit in smaller devices.

Theme

If your app supports a dark theme, no problem... This can be tested as well.

Screenshot for dark theme

Dynamic font size

Finally, you can also see how your view supports dynamic font size.

Screenshot for larger dynamic font size

Device configuration matrix

If you were to test each of the above configurations in a different snapshot test, you would inevitably end up with lots of generated reference screenshots. Instead, you can group different configurations and reduce the number of configurations that need to be tested. A configuration matrix helps you define your grouping strategy.

Configuration matrix

In this example, we grouped everything in two configurations.

  • The large configuration tries to be as close as possible to what the majority of the users will see (using the most common device size, theme, and font size) and possibly also matches the designs⁵.
  • The small configuration tries to check if everything still looks ok in small devices, using a dark theme and an XXXLarge font size.

Exploring the domain

More importantly, screenshot testing also allows you to quickly see how the same view looks like in all different cases of your domain.

Let’s take a look at the view we have been using as an example so far. Its business purpose is to inform the user about when the market is open during the day.

Just by mocking your data, you can quickly have a look at what the view looks like when the market is closed

Different domain case (the market is closed)

or when there are more than one open intervals.

Different domain case (more than one open interval)

With screenshot tests requiring almost no effort to set up, exploring all the different UI domain cases has never been quicker and easier.

The biggest benefits

Testing all device and domain configurations

When you combine all the above, you will end up testing multiple configurations (localisations — theme — size — dynamic font size) all at once and be done with them early in the development phase.

In that way, you would have quickly tested cases that otherwise you would have never bothered to manually recreate and test. And most importantly if the CI has your back, you also avoid UI regression errors for all cases.

All device and domain configurations

The perfect pull request

As a very nice side-effect, screenshot testing will change your workflow when developing the UI and how the PR would look like in that case.

The perfect pull request
  • The view can and should be developed independently. You do not need to wait for the container of the view to be implemented first.
  • Thus the PR becomes quite concise, as it only contains the code for the view along with screenshots of how it will like in all possible UI cases.
  • More importantly, screenshots add a missing visual dimension to the code review, thus reducing cognitive effort for the reviewer. The code reviewer no longer has to try to imagine how the UI will look like, he can actually see it.

Other interesting use cases

In the scope of the Composable Architecture, the PointFree team presented in episode 86 their solution on e2e functional tests using SwiftUI and snapshot testing. It is absolutely mind-blowing 🤯 and a must-watch.

This talk (Testing and Declarative UI’s) by Nataliya Patsovska is about combining Xcode Previews and snapshot tests. Pretty interesting as well.

People have been using screenshot testing to build and test their app’s UI design system (see Playbook for iOS and Storybook for Web)

Testing animations is also possible. We have been using it to test Lottie animations by taking snapshots of the animated view. Moreover, Stagehand, a composable type-safe animation library from Cashapp, uses snapshot testing that can also generate animated PNG files to test its functionality. Similar concepts are described in this talk.

Finally, some people use as a snapshot the difference produced between two snapshots. In this way they can isolate what changed before and after you interacted with it and test just that. Check it out here.

Tools

For iOS projects, one should check the excellent SnapshotTesting library provided by the amazing PointFree team.

Every iOS developer should also check the variety of snapshot strategies that this library supports. You will find strategies for things like CGPath, UIBezierPath, WKWebView, URLRequest and so much more. Really inspiring work by the PointFree 🥇.

For the sake of completeness, I would also like to mention the iOSSnapshotTestCase library, which is the very first library available for iOS.

For Android projects, the options are the Shot library by Karumi or the most recent Paparazzi by CashApp.

You can also check out part 3 which is an iOS hands-on guide that illustrates how all the above are translated into code.

To go beyond UI testing and find out lots of examples of how snapshot testing could be used in a non-UI context, you can continue reading part 2.

You can follow me on Twitter and LinkedIn.

References — Further reading

[1] Jest Snapshot Testing

[2] https://benmccormick.org/2016/09/19/testing-with-jest-snapshots-first-impressions/

[3] https://www.thoughtworks.com/radar/techniques/snapshot-testing-only

[4] https://netflixtechblog.com/pseudo-localization-netflix-12fff76fbcbe

[5] As a next step, you could even produce a side-by-side comparison of the generated screenshots against the designs (exported directly from e.g. Figma). And using this to automatically augment the context of a pull requests for the code reviewer would be very nice indeed.

[6] https://www.stephencelis.com/2017/09/snapshot-testing-in-swift

[7] https://www.vadimbulavin.com/snapshot-testing-swiftui-views/

[8] https://osinski.dev/posts/snapshot-testing-self-sizing-table-view-cells/

[9] https://troz.net/post/2020/swiftui_snapshots/

Special thanks to Vagelis Koutkias for introducing screenshot testing to the team a while ago (lots of the screenshots are actually part of that very first PR), and mentioning [5] while reviewing.

Special thanks to Nikos Linakis for letting me use screenshots and terms from his initial article on Screenshot testing and always coming up with the best terminology (“the perfect pull request” is all his).

Special thanks to Natalia Chalkidou and Stelios Poulakakis for their ideas about how one device configuration should match the designs.

XM Global

Coding is fun!

Sotiropoulos Georgios

Written by

Over 10 years of experience as a mobile software engineer with a current focus on Swift, iOS and all kinds of software verification methodologies.

XM Global

XM Global

Coding is fun!

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store