Snapshot testing in XCUITest
“It’s not a bug, it’s a feature”. Almost every member of the team has already learned this phrase. And yes, curiously enough, it can be said about some amount of defects. But not about badly displayed layout. This is especially important for mobile testing, given the time-cost of delivery to the store.
We are very limited in choosing ready-made tools for testing the layout in iOS applications. But in general, everything is not so bad, because we have at least one gorgeous tool for these purposes, which I would like to talk in this XCNote — iOSSnapshotTestCase.
This tool was originally created as an open source project under the leadership of Facebook. Over time, having got into the archive, the initiative passed to Uber and the project is still successfully developing. Most pleasantly, that iOSSnapshotTestCase is inherited from XCTestCase, which obviously means, that we don’t need to make any changes into our automation system at all.
At the beginning we should:
- add an additional pod in Podfile:
- and set the environment variables in our test scheme:
1. At first we should import FBSnapshotTestCase and inherit our test class from the FBSnapshotTestCase
2. The functional of saving reference snapshots is regulated by the variable recordMode, which is located in the setUp():
recordMode = true // saving reference snapshots
recordMode = false // verifying reference snapshots with actual
3. In the most cases, we will need two verification options:
verifyView() // full screen verification
verifyElement(_ element: XCUIElement) // verification of a single element
4. At a result, it is enough to call the verification method in our test:
These are capabilities that allow to add distinctive features to the names of the created snapshots, depending on the required filtering. Wow 😳. If explains it on fingers and a little bit easier:
Imagine that we have a mobile application opened on the screen with the text. We see difference between fonts on iOS 9 and iOS 12. We understand that these font differences are not a bug of the app, but when verifying snapshots, the test has been failing.
To avoid this problem, we can set the agnostic option, which will add the iOS version to the name of out test snapshot. Thus, we will need to create reference snapshots for each version of iOS, and during verification, the test will look for the snapshot it needs.
Currently out of the box we have 3 types of agnostic options:
The file name should be agnostic on the device name, as returned by UIDevice.currentDevice.model.
The file name should be agnostic on the OS version, as returned by UIDevice.currentDevice.systemVersion.
The file name should be agnostic on the screen size of the current keyWindow, as returned by UIApplication.sharedApplication.keyWindow.bounds.size.
We can use the agnostic options in the same way as the recordMode, by assigning them in the setUp():
This property allows to set the percentage error in the verification of snapshots. It is calculated on a scale from 0 to 1, where tolerance = 0.01 is a difference in snapshots equal to 1%. The tolerance is set as an argument of FBSnapshotVerifyView class:
Tear off the Status Bar
If we will need to verify the snapshot of full screen, we may face the problem that we will have conflicts in the status bar (battery, time, etc.). It is possible to solve this problem, although in a slightly ridiculous way — by cutting off a few pixels that used for status bar, while taking a snapshot.
The main problem is that since X series of iPhone we have nonstandard size of status bar:
We can implement this crutch using UIImage extension:
Thereafter, for the correct operation of this cutting on all iPhone models, we will need to learn how to determine it from our tests. To do it, we need to extend the UIDevice:
And then will use it like this:
Sometimes we may need to verify the snapshot of the full screen with the exception of a specific element.
Can anyone draw? And paint?) It is time to paint over, because we can solve this problem with the help of “colorings”. I am not kidding 😉. For this we need:
1. to extend the UIImage once again by creating the fill(element: XCUIElement) method, which will paint over the chosen element on the snapshot:
2. to create a new method verifyScreen(without element: XCUIElement) near the previously created verification methods:
To sum it up, I would say that testing with snapshots on mobiles is at least possible and at most mighty speed up our automation testing.
If you had any questions or clarifications after reading the article about the snapshot testing, I’ll be happy to answer them. And you can find the completed project here:
So, good luck, have fun, see you later (: