How we build verifiable Android apps

Separation of logic: make the views dumb

The Shopify Point of Sale for Android (POS) app needs to function as expected. Our merchants use it on a daily basis to run their business; it powers their livelihood by allowing them to take payments for their products and services. This means that the app needs to be well tested before we ship updates to users, so we’ve invested heavily in making it testable.

The primary way that we’ve built for testability is by moving logic out of components that are difficult to test. Views are built to simply take some state and display themselves based on it. This allows us to avoid writing instrumented tests that use Espresso actions as a catalyst for test cases, which we have found to be relatively error prone. Instead, we use screenshot tests on the views to record baselines for what the view should look like in each state. Our CI then generates new screenshots for each build and compares them to the baselines to see if any visual regressions were made. Screenshot testing has increased reliability and reduced the overall execution duration of our test suite.

ViewModels and Models: unit tests

Because our ViewModels and Models are strictly business logic that is decoupled from any view or lifecycle concerns, writing unit tests against them is a breeze. These are exclusively JVM unit tests that are very quick to execute and highly reliable. It’s no coincidence that the majority of both our critical business logic, and automated tests to verify said logic, live in this area.

The Model objects are generally the easiest to write tests against. Set up or mock its state, call a function on it, and verify the output or new state. ViewModels are slightly trickier in that there is a bit of indirection in their output. A ViewModel’s input is through a set of ViewAction functions: events that come from the view, for example onAddCustomerButtonPressed().

The output of a ViewModel is generally a new value published to one of its LiveData instances. Therefore a ViewModel unit test will start by observing one of the LiveData, then calling one of the ViewAction functions. The output is verified by asserting that our observer was notified of the expected change to the LiveData.

Network requests: API integration tests

Much of the business logic for Shopify POS lives server-side, and the app interacts with it via a set of GraphQL APIs. These server-side controllers and API have their own thorough tests, however Shopify believes in shipping fast. Though the necessary precautions are taken, it’s still possible for regressions in the GraphQL APIs to make it to production. It’s imperative that we catch these regressions before they affect our users, so we’ve built a set of API integration tests that periodically exercise the API requests that the app sends to the backend.

The API integration tests are JVM unit tests that use the app’s networking stack to execute an actual HTTP request against Shopify’s production servers and verify that the response is what we expect. As with the ViewModel and Model tests, these tests tend to be very reliable as they are run directly on the JVM. However because they perform a full network round-trip they are expensive in terms of execution time. For that reason we must weigh the potential value of each API integration test with its cost. In general this means that the more critical operations, such as a request to process a credit card payment, will be thoroughly exercised by integration tests.

Cost vs. benefit

A recurring theme that comes up in all of the tests we write is the cost of a test versus the benefit of knowing that we have an automated verification for the correctness of a piece of code. Potential costs include:

  1. Execution time
  2. Reliability (false positives, false negatives, flakiness, etc.)
  3. Developer time to write and maintain the test

Because of this tradeoff there are parts of our codebase where the benefits are not great enough to warrant automated tests, such as view animations. These areas tend to be ones in which a bug making it into production, though undesirable, will not have a critical affect on the ability of the app to do what ours users need from it. By deliberately building the critical functions of Shopify POS to be testable, we are continuously working towards a software system that is provably correct in an automated manner.