The 3 tiers of the Android test pyramid
The test pyramid is a concept that any automated testing strategy should have more low-level unit tests than high-level UI tests. Let’s take a look at how the test pyramid can be applied to modern Android development.
Today we have many testing tools available on Android including JUnit, Mockito, PowerMock, Robolectric, Espresso, Robotium, and UI Automator. But where does each tool fit into the test pyramid and why?
Unit tests are the fastest and least expensive tests you can write. They should cover most if not all of your application’s logic (but not its dependencies). Dependencies should be replaced with mocks or stubs in the test harness. Unit tests can be run frequently as a regular part of the software development process. On Android JUnit, Mockito, the mockable Android JARs give us a nice combination of tools to run fast unit-style tests in the local JVM.
Integration tests validate how your code interacts with other parts of the system but without the added complexity of a UI framework. For this layer Robolectric has become my tool of choice. Robolectric executes your code against real (not mock) Android JARs in the local JVM. This is made possible using bytecode manipulation to simulate low-level system components (like the UI). The end result is a more realistic test harness without the overhead of deploying to an emulator or device. Using a working version of the real Android framework does however slow test execution and add a degree of brittleness when compared to pure unit tests.
UI tests are the slowest and most expensive kind of test. They are slow because you must deploy your app to an emulator or device and drive it using the UI. They are expensive because this often means maintaining an internal device lab and CI server, or paying for a cloud testing service. Nonetheless automated UI tests can be an essential component of any testing strategy since they can uncover issues related to UI, hardware, firmware, and backwards compatibility. The primary tools we have for UI tests on Android are Espresso, Robotium, or UI Automator. UI tests can also be a valuable second line of defense in catching issues that may have slipped through the cracks in the other tiers of the pyramid.
If you get a failure in a high level test, not just do you have a bug in your functional code, you also have a missing unit test. Thus whenever you fix a failing end-to-end test, you should be adding unit tests too. — Martin Fowler
