Snapshot testing in XCTest

This Note originally published on my Personal Blog here. Read original note so that you won’t miss any content.

Introduction

“It’s not a bug, it’s a feature”. Almost every member of the team has already learned this phrase. And yes, curiously enough, it can be said about some amount of defects. But not about badly displayed layout. This is especially important for mobile testing, given the time-cost of delivery to the store.

We have many similar tools for testing the layout in iOS applications. But the most part of them forked or took an idea from only one. So, in this XCNote I would like to talk about that — iOSSnapshotTestCase.

This tool was originally created as an open source project under the leadership of Facebook. Over time, having got into the archive, the initiative passed to Uber and the project is still successfully developing. Most pleasantly, that iOSSnapshotTestCase is inherited from XCTestCase, which obviously means, that we don’t need to make any changes into our automation system at all.

Also I would like to notice we can use iOSSnapshotTestCase as in XCUITest as in XCTest. Awesome, let’s try.

Precondition

At the beginning we should:

• add an additional pod in Podfile:

• and set the environment variables in our test scheme:

Common Usage

The functional of saving reference snapshots is regulated by the variable recordMode, which is located in the setUp():

recordMode = true  // saving reference snapshots
recordMode = false // verifying reference snapshots with actual

To make more than one check per test we should set “identifier” as an argument of FBSnapshotVerifyView and FBSnapshotVerifyLayer methods.

Usage in XCUITest

Usage in XCTest

File Name Options

These are capabilities that allow to add distinctive features to the names of the created snapshots, depending on the required filtering. Wow 😳. If explains it on fingers and a little bit easier:

Imagine that we have a mobile application opened on the screen with the text. We see difference between fonts on iOS 9 and iOS 12. We understand that these font differences are not a bug of the app, but when verifying snapshots, the test has been failing.

To avoid this problem, we can set the agnostic option, which will add the iOS version to the name of out test snapshot. Thus, we will need to create reference snapshots for each version of iOS, and during verification, the test will look for the snapshot it needs.

Currently out of the box we have 4 types of file name options:

1. FBSnapshotTestCaseFileNameIncludeOption.device

The file name should be agnostic on the device name, as returned by UIDevice.currentDevice.model.

2. FBSnapshotTestCaseFileNameIncludeOption.OS

The file name should be agnostic on the OS version, as returned by UIDevice.currentDevice.systemVersion.

3. FBSnapshotTestCaseFileNameIncludeOption.screenScale

The file name should be agnostic on the screen scale, as returned by UIScreen.main.scale.

4. FBSnapshotTestCaseFileNameIncludeOption.screenSize

The file name should be agnostic on the screen size of the current keyWindow, as returned by UIApplication.sharedApplication.keyWindow.bounds.size.

We can use the file name options in the same way as the recordMode, by assigning them in the setUp():

Tolerance

I mean, as far as we allow current snapshots to be different from the reference ones. It is calculated on a scale from 0 to 1, where tolerance = 0.05 is a difference in snapshots equal to 5%. By default: 0%. The tolerance is set as an argument of FBSnapshotVerifyView and FBSnapshotVerifyLayer methods. The library provide us with two realisations of this feature:

• overallTolerance — how many pixels may not match
• perPixelTolerance — how each pixel may not match

Tear off the Status Bar

If we need to verify the snapshot of full screen via XCUITest, we may face the problem that we will have conflicts in the status bar (battery, time, etc.). It is possible to solve this problem, although in a slightly ridiculous way — by cutting off a few pixels that used for status bar, while taking a snapshot.

The main problem is that since X series of iPhone we have nonstandard size of status bar:

We can implement this crutch using UIImage extension:

Thereafter, for the correct operation of this cutting on all iPhone models, we will need to learn how to determine it from our tests. To do it, we need to extend the UIDevice:

And then will use it like this:

Painting

Sometimes we may need to verify the snapshot of the full screen with the exception of a specific element.

Can anyone draw? And paint?) It is time to paint over, because we can solve this problem with the help of “colorings”. I am not kidding 😉. For this we need:

1. to extend the UIImage once again by creating the fill(element: XCUIElement) method, which will paint over the chosen element on the snapshot:

2. to create a new method verifyScreen(without element: XCUIElement) near the previously created verification methods:

Conclusion

To sum it up, I would say that testing with snapshots on mobiles is at least possible and at most mighty speed up our automation testing.

If you had any questions or clarifications after reading the article about the snapshot testing, I’ll be happy to answer them. And you can find the completed project here:

https://github.com/alter-al/sample_of_ios_snapshot_testing_2048

So, good luck, have fun, see you later (: