Writing fast, deterministic and accurate Android Integration tests

Introducing OkReplay — record and replay OkHttp network interaction in your tests.

At Airbnb, shipping a high quality product is of utmost importance. To accomplish this goal, we use automated testing to catch bugs before they reach our users. In this post, we’ll describe how we use a particular type of automated tests, UI tests, to ensure the quality of our Android application and will announce the open source launch of OkReplay.

Our choice for Android UI tests is Espresso, which is arguably the best, most popular and recommended library for writing integration tests. However, the extra confidence provided by a solid test suite can be quickly affected by flaky tests creeping up and undermining everyone’s trust on them. In fact, flaky tests are poison!

Integration tests are at the very top of the Test Pyramid, but that doesn’t mean that they have to be slow, brittle or expensive to write. They should be complemented by an even bigger coverage of service and unit level tests.

Martin Fowler’s Test Pyramid

As an example of Service layer tests, one could build a set of APIs that provide test fixtures that can construct fake domain models for use during the test execution (eg.: Ruby’s FactoryGirl and Forgery).

Back to UI tests, flows that rely heavily on the network (eg.: typically performing many API requests) are often affected by network instability during the test execution. This was observed through high flip rates, that is, the rate at which a test will switch from failed to successful and vice/versa between invocations. One of the ways to expose the flakiness was to schedule tests to run every hour, for example, with or without code changes in order to measure their stability. Based on our findings, we started looking into options on how to remove the network variable from this equation. Some of us were familiar with the awesome VCR library, popular among Ruby/Rails folks, but nothing similar seemed to exist for Android. Thus OkReplay was born.

Since all the Airbnb network traffic goes through OkHttp, creating an Interceptor that does the network recording and replaying seemed like the easiest way forward. Interceptors are very powerful and can modify network calls as they see fit, which is exactly what we needed. While looking for similar solutions, we also found the awesome Betamax project, which aims to solve a very similar problem, but was not designed with Android or OkHttp in mind. Regardless, since most of that project’s goals were similar to our needs, we decided to work off of its codebase and modify as needed to make sure it’s Android and OkHttp friendly. Some of these goals were:

1. Seamless integration with existing tests
2. No production impact or changes to the app behavior. It should be explicitly enabled during tests
3. Little or no dependencies on third-party libraries
4. Works with OkHttp, JUnit and Espresso
5. Recorded interactions should be easily readable, modifiable and committed to source control
6. Does not require running an external service or proxy in order to work

Betamax checked almost all the boxes, except for numbers 4 and 6. Another option was Wiremock, however it did not satisfy item 6. Thankfully, extending Betamax was simple enough due to its awesome modular architecture and test coverage. Additionally, having clearly defined constraints, especially the hard dependency on OkHttp, allowed us to build a much simpler solution since we don’t need to solve it for every HTTP client out there!

Under The Hood

In a nutshell, OkReplay boils down to a simple OkHttp Interceptor that, when turned on, looks at every outgoing request and either matches them against a pre-recorded set of “network interactions” and replays them, or captures the network response as soon as it comes down the wire and saves them. These recording are called “tapes”. This term was inherited from VCR where tapes are simple files in YAML format where the network interactions are stored. You can think of these files as simple test fixtures.

For Android Espresso tests, tapes are stored by default in src/androidTest/assets/tapes and loaded (read) during test execution as regular assets using Android's AssetManager. For recording (writing), they are stored in the device's external storage directory and automatically pulled out of the device after the test execution using a simple Gradle Plugin that comes with OkReplay. This distinction happens because the test APK is not able to overwrite its own package contents to modify the YAML asset files during runtime, so our only option was to write it to external storage instead and overwrite the files using the Gradle plugin. The tapes should be easily readable and may be modified as needed.

Sample OkReplay tape

There's a few other key ingredients to OkReplay. First, is the Recorder, which consists of a JUnit TestRule responsible for loading the appropriate test tape, starting and stopping the Interceptor and writing the tape file after the test is finished. Second, the TapeLoader, as the name says, loads and writes (YAML) tape files. Finally, the MatchRules, which define ways for matching requests against each other based on, either a predefined set of built in rules, or a custom logic based on each application's specific needs.

Using OkReplay

Using OkReplay is just as simple as annotating your test method(s) with @OkReplay, eg.:

The annotation allows you to optionally specify a few configuration options like the tape name, TapeMode and MatchRules.

Before it can be used, you'll need to add the OkReplayInterceptor to your OkHttpClient instance and register the OkReplay JUnit TestRule:

Setting up the TestRule

That’s it! Any network requests being made while running the testFoo() with OkHttp will automatically go through OkReplay now. By default, the TestMode is READ_ONLY, which means it will fail any requests until you've recorded a tape for them. In that case, while watching the device's Logcat output, you'd see an error message like the one below:

Error message on Logcat

While running, by default, OkReplay will not allow any network requests to hit the network unless the TapeMode is “writeable” (either READ_WRITE or WRITE_ONLY). This is to ensure tests are repeatable and deterministic.

Once you have all your interactions recorded, you could even run your tests while the device is in Airplane Mode and still see them work fine, assuming you don’t have any other hard dependencies on the network being available.

Finally, since OkReplay is a library meant to be used only while testing, you can use its no-op variant as a releaseCompile dependency to make sure your release application ships with a stubbed version of the OkReplayInterceptor.

Conclusion

By open sourcing this library and sharing it with the broader Android community, we hope to incentivize developers to write more tests and ship quality apps with more confidence!

As always, feel free to fork it, send us feedback, bug reports and pull requests. If you’d like to join us in finding more creative solutions to fun challenges like this one, we’re hiring! 😃