iOS Snapshot testing with Perceptual hash
Snapshot testing is used to test the rendered output of a UI element. The UI element can be a view controller which is a whole screen or just a view component. This idea is introduced by Facebook with the FBSnapshotTestCase project and it is continued by Uber as iOSSnapshotTestCase.
Basic idea of the snapshot testing is to store a reference to disk as an image and compare the reference image with the on the fly renderings while testing. This is working well since a lot of information about a reference is stored already in the image file.
I see an improvement point here. Let’s say we have a component and the view model of the component have 8 different property to configure the view. It can be assumed that a fixed value or a nil value is assigned to a property. This ends up with 256 different variations. The number of variations is likely to be increased if projects supports dark mode or RTL languages. If any onf the mis supported, the total number of test cases become 512. If we assume that each image file is 10KB, the total size required to test for a simple view is around 5MB.
Apparently, this is not a big deal, otherwise there should already be an alternative. As i said, i see this as an improvement point and i propose that this redundant space requirement can be removed by using perceptual hashing.
Perceptual hash is similar to cryptographic hashes in terms of the idea of creating a fingerprint from a data. While cryptographic hash functions have avalanche effect which means the output of the hash function is being effected drastically even just for one bit of changes. However, output of perceptual hash functions are kept same or changed slightly with the small changes in the input data. This property of perceptual hashes make the reverse media search easier like Google Image Search or Shazam.
There are a couple of image hash functions available such as average hash, blockhash, perceptual hash, etc.. Many of them are compared in this article here and perceptual hash seems performing better. I will not go into the details of this algorithm but i recommend you to check how those algorithms work.
The output size of perceptual hashes are just 64bits. The outputs of them are not randomly distributed by design but it is possible to say two images are most probably similar images if their hash values are same. In the use cases of these algorithms, a threshold value is determined since having same values as a requirement for similarity can be too strict for just a search. The threshold value is simply calculated by Hamming distance of two hashes which is basically the number of different bits in two hashes.
Usage of a perceptual hash in testing is a new use case and same hash result should be expected from output of the tests. This is possible if it is guaranteed that the same simulator will be used for tests. When this is guaranteed, even a cryptographic hash may be possible to use. However, this condition is not feasible and different device simulators simulate different screens with different scale values. Basically, the output of two different simulators can produce scaled version of each other and since the data is different the hashes would also be different.
It is obvious that, even if scaled images represent different data, they are similar images. This can conclude that the perceptual hashes of them should be close to each other. Using a very small threshold can make the use of perceptual hashes possible in testing and it ends up with 64bits of space requirements for each snapshot which ends up with 4KB instead of 5MB in the previous example above.
It is good to note that even if the above images produce same perceptual hash values, the distance between two values which is called as threshold above should be small enough. This shows the perceptual hash method does not have platform dependency while cryptographic hashes have.
Perceptual hash algorithms are focused on perceptions, not the data integrity that’s why the differences caused by scaling is tolerated. Not only scaling but very small changes can be tolerated as well. For example, in above images the space between title label and bookmark icon is 8px. If the space is increased up to 12px, the algorithm would tolerate and it would end up with the same hash value. Increasing the space to 12px would create 1 bit difference which means that tests can catch this change but will miss the cases between 8px and 12px.
One of the calculation steps of perceptual hash is to convert the image into grayscale. This is basically done by taking the average of the RGB values of a pixel. In above design, the user name is rendered with purple colour whose RGB value is (0.5,0,0.5). The grayscale pixels of these purple pixels are calculated as 1/3 = 0.33. If the colour is changed to red with (1,0,0) values, the grayscale pixels will still have the same value 1/3. This shows that the algorithm also tolerates some specific colour changes whose grayscale values are same.
Changing the colour to a different one whose grayscale value is not 0.33 effects the perceptual hash value. In the image below, the colour is set to (0.7,0.6,0.3) whose grayscale value is 0.8. This will result a hash value 2 bit away from original hash.
In the lights of these inspections above, it is not easy to say that perceptual hash can be used for any project’s snapshot testing but also it is not easy to say that it is useless. This method can be used if the project is not so sensitive against small changes, so i recommend you to evaluate your project’s requirement before using it.
Implementations of perceptual hashes are very straight forward, so it can be implemented simply in a project and tweaked a bit if needed or there are available libraries that can be used through CocoaPods like CocoaImageHashing which provides a couple of different hash implementations.
I hope you enjoyed reading and thanks for your time.
