Android testing with Espresso’s Idling Resources and testing fidelity
Synchronizing your app with Espresso
If you know why you need Idling Resources, you can jump to the next section
Espresso’s main advantage over other UI testing frameworks is that it synchronizes with your app. This means that it will only make assertions and perform actions when your app is idle (waiting for user input). Other testing frameworks wait until a certain condition is met (a view is displayed for example) but this has proved problematic in a lot of situations and led to flaky tests. Other frameworks rely on you guessing the time a certain operation will take. This makes tests slow and doesn’t really guarantee determinism. You should never have to use Thread.sleep in Espresso tests.
Espresso knows about Android’s internals and is capable of looking ahead to figure out what’s coming. If the main thread is busy (or will be busy in the near future), Espresso waits before running the next operation. This synchronization mechanism, and other tricks up Espresso’s sleeves, is what made it the most used UI testing framework on Android.
However, writing UI tests is not easy and most of the problems are still related with synchronization. There are some things that Espresso doesn’t know about:
- Animations. If you have an animation running indefinitely, Espresso might timeout waiting for it to finish. This is why you should disable animations in your test devices.
- Background operations. Espresso looks at the AsyncTask thread pool as well as the main thread. However, not many people use AsyncTask anymore. If you use your own thread pool executor, Kotlin coroutines or RxJava, you need a way to synchronize them with Espresso
- Other mechanisms to schedule updates. For example, Data Binding uses the Choreographer to post updates instead of the main Looper queue (what Espresso monitors).
The error that you’ll see when this happens varies, but it’s usually a NoMatchingViewExceptionbecause what is on screen is not (yet) what you told Espresso to expect.
The APIs that Espresso provides to let you indicate when the app is busy are called Idling Resources.
How Idling Resources are registered
Before a test begins you can register as many Idling Resources as you need to. Normally you do this in a test class inside a @Before method:
IdlingRegistry.getInstance().register(myIdlingResource)
You’ll learn how to create the idling resource in the next step.
Remember to unregister them after you’re done, for example inside an @After method:
IdlingRegistry.getInstance().unregister(myIdlingResource)
Creating Idling Resources
Idling Resources implement the IdlingResource interface and you could create your own:
However, I recommend you use the CountingIdlingResource that comes with Espresso which lets you call increment each time you start a task and decrement when you finish it. That way you can control more than one task running concurrently.
Example 1: Snackbars
Tests that show Snackbars are hard to UI-test because they start a pop-up animation, they stay displayed for a while and then they are hidden with another animation. Espresso doesn’t really know when your app is idle because it only sees 15 ms in the future and Snackbar.LENGTH_SHORT is 2 seconds. Even with disabled animations, the Snackbar will be shown.
You can add a callback each time you show a Snackbar and notify the Idling Resource when it’s shown and when it’s dismissed. This example is extracted from the Android Testing codelab:
You could also disable Snackbars completely from UI tests, to make them pass faster, as you’re not verifying their content this way.
See how we verify Snackbar content in unit tests in the testing codelab.
Example 2: Idling Resources used only from tests
Not all Idling Resources live in your production code. For example, in the testing codelab we use a DataBindingIdlingResource that looks for bindings in an activity and calls hasPendingBindings on them to figure out if they’re busy or idle.
These tools are only used by the tests, so they’re the perfect Idling Resource: no modification of production code needed.
Other examples are RxIdler, an Idling Resource which wraps an RxJava Scheduler and okhttp-idling-resource, for, you guessed it, OkHttp requests. We’re currently exploring how to do this with Kotlin coroutines.
Example 3: modifying production code 😵
If you load data in a background operation and there’s no other Idling Resource mechanism available, you can expose an Idling Resource from your app to the test. For example, in your ViewModel or in your data repository:
However, you’re modifying the code under test just for testing. Ideally, we shouldn’t have to.
The debate is on: should we use Idling Resources in our production code?
On the one hand, they’re very easy to use and they do make your code more testable. On the other hand, you’re adding code to an app that will be run by potentially millions of users but will do… nothing.
If you go this route, you should limit the negative effect in production. For example, if you are using a CountingIdlingResource from an object, create different versions of this object for debug and release (or for different flavors). The released version should do nothing:
Same goes with the Snackbar example where we added listeners to it. The extension function would be replaced for the release with:
The problem is that these classes will have no UI test coverage so they should be kept simple.
Alternatives to Idling Resources in production
You have to find your balance between modifying code for testing and maintaining fidelity of tests. High fidelity means that what you test is what users will use. Otherwise the test might not catch specific bugs.
Let’s say you want to solve the problem by making all background operations run sequentially instead of concurrently. You can use the main thread or even the AsyncTask thread for this as we saw before. This is great because you just need to inject some test executors or replace some dispatchers in the test. However, consider this code:
This launches two coroutines that will run concurrently in production. If you are not familiar with coroutines it’s like executing two Runnables in a thread pool. In tests we could flatten execution into a sequence of operations, but the remote data source call will always be executed first.
Is this behavior fidelitous (does it correspond to reality in production)? Almost! But you might be missing edge cases in low-end devices or limited connectivity, for example.
With this approach you can avoid Idling Resources in production, but you should be aware of the consequences and cover the missing situations. For example, you should probably add a series of deterministic unit tests to verify the cases you’re not testing in UI test where job #2 finishes before #1.
Note: the solution for coroutines is a work in progress and tracked in EspressoIdlingResource integration. The idea is to be able to set a dispatcher that can be registered as an Idling Resource.
Summary
1. If available, use the Idling Resources provided by the library you’re using to do work outside of the main thread.
2. Otherwise, decide if you want to modify production code by adding Idling Resources like CountingIdlingResource. If you are modifying code, make sure it has low impact. If you are modifying the behavior of your app to avoid Idling Resources, make sure you understand the repercussions and add coverage in a different test.
3. If the scope of your test can be smaller, you can always replace one of your dependencies for a fake. In the codelab, all “fragment” tests use a FakeTasksRepository so they’re fast. See TasksFragmentTest of the testing codelab and the rest of the @MediumTests.
