Writing fast, deterministic and accurate Android Integration tests

Introducing OkReplay — record and replay OkHttp network interaction in your tests.

Felipe Lima
Jun 21, 2017 · 5 min read

At Airbnb, shipping a high quality product is of utmost importance. To accomplish this goal, we use automated testing to catch bugs before they reach our users. In this post, we’ll describe how we use a particular type of automated tests, UI tests, to ensure the quality of our Android application and will announce the open source launch of OkReplay.

Our choice for Android UI tests is Espresso, which is arguably the best, most popular and recommended library for writing integration tests. However, the extra confidence provided by a solid test suite can be quickly affected by flaky tests creeping up and undermining everyone’s trust on them. In fact, flaky tests are poison!

Integration tests are at the very top of the Test Pyramid, but that doesn’t mean that they have to be slow, brittle or expensive to write. They should be complemented by an even bigger coverage of service and unit level tests.

Martin Fowler’s Test Pyramid

As an example of Service layer tests, one could build a set of APIs that provide test fixtures that can construct fake domain models for use during the test execution (eg.: Ruby’s FactoryGirl and Forgery).

Back to UI tests, flows that rely heavily on the network (eg.: typically performing many API requests) are often affected by network instability during the test execution. This was observed through high flip rates, that is, the rate at which a test will switch from failed to successful and vice/versa between invocations. One of the ways to expose the flakiness was to schedule tests to run every hour, for example, with or without code changes in order to measure their stability. Based on our findings, we started looking into options on how to remove the network variable from this equation. Some of us were familiar with the awesome VCR library, popular among Ruby/Rails folks, but nothing similar seemed to exist for Android. Thus OkReplay was born.

Since all the Airbnb network traffic goes through OkHttp, creating an Interceptor that does the network recording and replaying seemed like the easiest way forward. Interceptors are very powerful and can modify network calls as they see fit, which is exactly what we needed. While looking for similar solutions, we also found the awesome Betamax project, which aims to solve a very similar problem, but was not designed with Android or OkHttp in mind. Regardless, since most of that project’s goals were similar to our needs, we decided to work off of its codebase and modify as needed to make sure it’s Android and OkHttp friendly. Some of these goals were:

  1. Seamless integration with existing tests
  2. No production impact or changes to the app behavior. It should be explicitly enabled during tests
  3. Little or no dependencies on third-party libraries
  4. Works with OkHttp, JUnit and Espresso
  5. Recorded interactions should be easily readable, modifiable and committed to source control
  6. Does not require running an external service or proxy in order to work

Betamax checked almost all the boxes, except for numbers 4 and 6. Another option was Wiremock, however it did not satisfy item 6. Thankfully, extending Betamax was simple enough due to its awesome modular architecture and test coverage. Additionally, having clearly defined constraints, especially the hard dependency on OkHttp, allowed us to build a much simpler solution since we don’t need to solve it for every HTTP client out there!

Under The Hood

For Android Espresso tests, tapes are stored by default in src/androidTest/assets/tapes and loaded (read) during test execution as regular assets using Android's AssetManager. For recording (writing), they are stored in the device's external storage directory and automatically pulled out of the device after the test execution using a simple Gradle Plugin that comes with OkReplay. This distinction happens because the test APK is not able to overwrite its own package contents to modify the YAML asset files during runtime, so our only option was to write it to external storage instead and overwrite the files using the Gradle plugin. The tapes should be easily readable and may be modified as needed.

Sample OkReplay tape

There's a few other key ingredients to OkReplay. First, is the Recorder, which consists of a JUnit TestRule responsible for loading the appropriate test tape, starting and stopping the Interceptor and writing the tape file after the test is finished. Second, the TapeLoader, as the name says, loads and writes (YAML) tape files. Finally, the MatchRules, which define ways for matching requests against each other based on, either a predefined set of built in rules, or a custom logic based on each application's specific needs.

Using OkReplay

The annotation allows you to optionally specify a few configuration options like the tape name, TapeMode and MatchRules.

Before it can be used, you'll need to add the OkReplayInterceptor to your OkHttpClient instance and register the OkReplay JUnit TestRule:

Setting up the TestRule

That’s it! Any network requests being made while running the testFoo() with OkHttp will automatically go through OkReplay now. By default, the TestMode is READ_ONLY, which means it will fail any requests until you've recorded a tape for them. In that case, while watching the device's Logcat output, you'd see an error message like the one below:

Error message on Logcat

While running, by default, OkReplay will not allow any network requests to hit the network unless the TapeMode is “writeable” (either READ_WRITE or WRITE_ONLY). This is to ensure tests are repeatable and deterministic.

Once you have all your interactions recorded, you could even run your tests while the device is in Airplane Mode and still see them work fine, assuming you don’t have any other hard dependencies on the network being available.

Finally, since OkReplay is a library meant to be used only while testing, you can use its no-op variant as a releaseCompile dependency to make sure your release application ships with a stubbed version of the OkReplayInterceptor.


As always, feel free to fork it, send us feedback, bug reports and pull requests. If you’d like to join us in finding more creative solutions to fun challenges like this one, we’re hiring! 😃

Airbnb Engineering & Data Science

Creative engineers and data scientists building a world…

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store