How to add Snapshot Tests in Xcode 14 + Cloud
Learn how to automate the testing of mobile apps.
Here’s part 3 of a step-by-step guide to building an iOS app.
I wrote about collecting requirements, use cases, sketches, and creating design files in Part 1. Part 2 discusses creating an app prototype. In this post, I’ll discuss adding snapshot tests.
What is snapshot testing?
Snapshot testing is a testing technique that takes a snapshot of the current state of the system and compares it to a previously known valid state, called a baseline. If the two states are different, the test fails.
Snapshot testing provides many benefits, including the ability to catch bugs early and prevent them from being introduced. Snapshot tests can also be run automatically without any input from developers or testers. Because of this, they are ideal for regression testing, which is often used to ensure that changes have not broken existing functionality.
In addition to their benefits, snapshot testing also has some disadvantages. They are tightly coupled to a system’s output, which makes them vulnerable to failure. It is possible for snapshot tests to fail even if only insignificant changes are made to the output. In order to verify that everything is still working properly, developers must manually update the snapshots by re-recording the baseline state.
A primary snapshot testing strategy relies on UIView or UIViewController rendered view snapshots that are saved as image files and then pixel-by-pixel compared.
Additionally, other testing strategies are available, such as testing view hierarchies, texts, etc.
You can find a full list of snapshot strategies provided by the SnapshotTesting framework that I’ll be using in the project here.
Having covered some theory, let’s get our hands dirty and add some snapshot tests.
Adding snapshot tests
Snapshot tests check whether UI states are rendered correctly across a range of screen sizes. The MVVM architecture allows us to achieve that nicely. We construct ViewModels representing different states and with SnapshotTesting framework we can record them as images for comparison.
For a prototype app, I will add snapshot tests all at once. However, adding them as you develop features one by one makes more sense since snapshot tests are fast and allow spotting problems without launching the app. In future features, snapshot tests will be present right away.
Snapshot testing starts with adding the SnapshotTesting framework to the project.
We should invest some time in helper functions afterwards. As we will be testing across multiple screen sizes and appearances, we should add a function which does that. Precision <= 0.99 is required for passing tests on Xcode Cloud — images rendered on Intel and M1 differ slightly.
Now we are ready to add snapshot tests.
For the cards list I’ll cover:
- Display an empty list when not loaded.
- Display spinner when loading.
- Display placeholder when empty and loaded.
- Display a list with one section when loaded.
- Display list with two sections when loaded.
- Display a short list when the search bar is in the header.
- Display a long list when the search bar is in the header.
- Display a short list when searching.
- Display a long list when searching.
For card details I’ll cover:
- Display a short title and barcode with a background image.
- Display a long title and barcode with the background image.
- Display a short title and barcode with colored background, without an image.
- Display long title and barcode with colored background, without image.
- Display a short title and short QR code with colored background, without an image.
- Display short title and long QR code with colored background, without image.
As you may have noticed, it covers all possible view states, short & long titles, one, multi-section, varying-size data sets.
After running these tests I’ve encountered a few issues related to view rendering using snapshot testing — cell size caching logic was incorrect because the device size was changing throughout the tests, and frame-based title views were rendered incorrectly.
Upon fixing these test-related issues, real issues began to appear, such as:
- Missing placeholder on an empty list.
- Visible section header when one section is displayed.
- Non-hiding search bar in the list when the list is short.
- Incorrect position of long card title in card details.
Without snapshot tests those issues would be noticed just by running the app. Now we caught them while testing, re-recorded the baseline and we can safely go further with the development of the app.
Identifying issues right away leads to a better user experience while using the final product. In today’s spoiled market, delivering a 5-star user experience is essential.
Running snapshot tests in Xcode Cloud
This part should be straightforward, but it’s not. Turns out there are a couple of issues:
- To run snapshot tests baseline images should be stored in the ci_scripts folder as testing happens in some sort of isolated container.
- Another issue is that in the release workflow, where the Apple Store archive is created, the ci_scripts folder is purged, and snapshot tests do not work at all. So for now I won’t be adding testing to the release workflow. If anybody figured out how to solve this I’d be happy to hear.
To address the first issue we need to extend our helper function which would save & validate baseline images to the ci_scripts folder.
In a Development workflow, we simply need to add a Test action with one destination. This is because we aren’t using UI testing and there is no advantage to running snapshot tests in multiple containers.
That’s basically it. After pushing changes to the master branch, the build will be tested in Xcode Cloud using snapshot tests. After merging the master to the release branch, a testflight build will be created and distributed by Xcode Cloud for testing.
You can check the full app source code with tests here.
