Android Testing: Making the right choice.
In the last article, I talked about continuous integration and why a developer should do it. Continuous integration(CI) is also required for Test Driven Development (TDD) and is key when it comes to meet a certain level of quality standards. You may see this great post by Rashi Karanpuria on Test Driven Development.
When you are following Test driven development, the real question is: Which type of framework is to be picked under which circumstances?
Instrumentation tests require access to Android components and hence require an emulator or Android device so they are slow to execute.
Unit tests are fast because they are executed in the JVM and hence do not require the dexing, packaging, and installing-on-the-emulator steps, reducing test cycles from minutes to seconds so you can iterate quickly and refactor your code.
If you have a large suite of integration tests, then doing CI will take a lot of time. That’s no way to do TDD.
If you want a simple and fast way to test your code on a build server, you should pretty early integrate Robolectric in your project. Key benefits of using Robolectric is:
- Higher speed of execution compared to instrumented tests. As a result of not having to launch an emulator or a device, Espresso(An integration test framework)works slower than Robolectric.
- Easily executable on Continuous Integration server. Setting up instrumentation tests on CI is difficult and complicated.
Beware, Robolectric covers almost all Android API but not all of them. So it might so happen that newest APIs launched by Google is not testable by Robolectric.
One piece of advice though, relying completely on Robolectric will be immature because there are certain things such as testing multi-touch ,which can be done with instrumentation tests very easily.
When Google launched Espresso back in Google IO 2013, it really excited the developers because there wasn’t a good approach to write tests.
The main benefit Espresso provides over Robolectric is it’s reliability. Espresso tests are much more reliable because it runs on real device using real resources. Also, espresso is extremely easy to use.
But as stated above, espresso tests run on real devices, its slow and is complicated to set up on build server.
I would say to go along with both testing frameworks Robolectric and Espresso. It isn’t a “either…or” question. Espresso is the most reliable and secure way of testing but if you’re part of a group with several developers and a central repository, I would highly recommend the usage of a continuous integration server executing Robolectric unit tests.
The ratio of Unit tests, Integration tests and UI tests should roughly follow the following criteria:
- Unit tests 70–80%
- Integration tests 10–20%
- UI tests 5–10 %
Again, it’s not a hard and fast rule, but something to keep in mind to make TDD feasible.