Snapshot Testing Motivation in Android

What is snapshot testing ?

Snapshot testing is a type of testing which helps us ensure our UI does not change unintentionally. Inflate a view, take a screenshot of it, and compare it to an already stored image taken as reference. If they differ, the test fails, passes otherwise. The main motivation behind is that it can catch visual regression bugs.

Potential view bugs:

1. How edge cases are displayed.
2. Styles, themes, spacing etc.
3. View alignment with different screen size.
4. The UI elements display in the desired color.
5. Unexpected changes by a library version change, Material design components.
6. Very verbose languages, non-latin alphabets etc.
7. Huge system font size (lots of users using a font size different from the default one).

Snapshot tests are more widely used than UI tests by a lot of big companies: Airbnb: ≈ 30.000 snapshot tests
Uber: thousand of snapshot tests vs. a handful of UI tests
Spotify: ≈ 1.600 snapshot tests vs. ≈ 500 UI tests
Shopify: ≈ 2.300 snapshot tests vs. ≈ 20 E2E tests

Paparazzi

An Android library to render your application screens without a physical device or emulator. Paparazzi runs completely on the JVM without the need of a device or emulator.
Also generates an HTML report at sample/build/reports/paparazzi/ showing all test runs and snapshots.
Saves snapshots to a predefined source-controlled location (defaults to src/test/snapshots).
Runs tests and verifies against previously-recorded snapshots. Failures generate diffs at sample/out/failures.

How snapshot testing works ?

All these visual properties are pretty easy to verify with Screenshot testing. A snapshot testing consists of 2 steps.

1. When we are done building a piece of UI, it is a whole screen or just a button, we write our snapshot test and record. Screenshot is saved in our source code and it defines the baseline for our UI.
2. Snapshot test takes a new screenshot and compares the two images. If they are not identical, our snapshot test fails. Most likely we will touch some code at a later point which might or might not change the UI from our original screenshot.

the process goes as follows:

1. Execute ./gradlew recordPaparazziDebugto generate the corresponding snapshot files and open the corresponding PR containing them.
2. Reviewing the snapshots to confirm that it reflects the state of the view we are expecting. Once approved and merged this snapshot is taken as source of truth for future verifications of the tests.
3. Verify ./gradlew verifyPaparazziDebug the snapshots periodically. Ideally you have a CI with a job configured to "verify" them. This means, it executes the snapshot tests with the most up-to-date code, and takes a screenshot. After that, it compares this screenshot with the last one uploaded to the CI or the one being pushed in the PR. If they both match the test passes.

Diff between both status view.

Tests are fail because in this case status icon is not the same. Some new logical changes may unintentionally effects UI. In this step, find out the unintentionally UI change. We run either on CI/CD or manually allows us to find potential bugs in advance.

Android system comes with special settings to improve the UX for users. One of those settings is the font size to enhance readability. It enables users to increase the system font size to use their phones without holding them too close to their eyes.

You can find font size options under Settings > Display > Font sizes

Using large or largest font sizes, what makes UI elements with text taller and larger. This might result into overlapping of UI components. Without any visual impairment, we’re likely testing our UI-related code with the default device or emulator font size. Thus, any text-related issue is not be noticed.

Conclusion
Quickly find out visual bugs to ensuring that code changes do not break your app, screenshot testing can help you with all that. For take advantage of snapshot testing, automate all process with CI/CD, more visual asset when opening new PR and help your team members for attentive code review process.

Resources
https://github.com/cashapp/paparazzi
https://github.com/sergio-sastre/Android-screenshot-testing-playground
https://speakerdeck.com/gio_sastre/an-introduction-effective-snapshot-testing-on-android