Why 100% unit test coverage should not be a priority
Lessons learned from writing unit tests for Mitra Bukalapak Android App.
Our journey in unit testing started with a target: by the end of 2019, unit test coverage must achieve 25%. Unfortunately, it was not achieved.
In 2020, our unit test coverage went from 11% to 58%. We had started with little knowledge on how to write unit tests and went from writing not-so-good unit tests to figuring out how a good unit test should be written. A lot of good principles are taken from xunitpatterns.
Generally, Android app developers only care about 2 kinds of tests: UI and business logic¹. UI tests are created to make sure that views can be scrolled, buttons can be clicked, adapters show correct items, and so on. Business logic tests are for everything else.
Here are some tips that we’ve learned to avoid unit test pitfalls:
Unit is not function
During my observation of our codebase, most programmers when writing unit tests tend to think that a unit is a function. A test for a function is usually written after the function is done, or inversely if one is doing test-driven development. The problem with that is, those tests do not necessarily represent business requirements. We will need to write another type of test above those unit tests, which is called integration test. This feels redundant.
Instead, think of a unit as a unit of work, which is a chunk of business requirements that can be translated into a test case. Below is an example of a use case for when a user logs-in using a wrong phone number:
1. Given user open app
2. Then user click Mulai button
3. And user see non-login home page
4. Then user click Masuk button
5. And user see phone number page
6. And user fill phone number "<invalid_phone_number>" field
7. And user click Lanjut button
8. Then user see "Format salah." warning on headerA unit test should be written to verify this use case. Since the Android framework can only test a single Activity, we can’t combine steps 1–8 as a unit of work because they involve 3 Activities. Instead, the unit of work starts from the Login screen, which is steps 5–8. We can then easily implement the unit test using “front-door first” principle.
Below is a snippet of the Espresso test for the above use case. Note that we can have 3 or 5 functions in LoginFragment to achieve this use case but there is only one unit test.
@RunWith(AndroidJUnit4::class)
class LoginTest { @Test
fun userLoginWrongNumber_shouldShowWarning() {
inputFieldView().perform(typeText("54321"))
buttonView().perform(click())
warningHeaderView().check(matches(isDisplayed()))
}
}
Smaller units are just train stops on the way to the main station and can have more fakes/mocks that are not representing real use case. — The Art of Unit Testing
The trick of remembering this technique is, once you’re done writing production code, you have to forget it. Forget all the functions that you’ve written and start writing tests by thinking from the user’s perspective.
Use front-door first
UI layer and domain layer often have a 1–1 relationship. Let’s say you have a LoginFragment which is a layer that consists solely of presenting the state to the UI and a LoginController which is a domain layer that can be in the form of Jetpack ViewModel, MVP Presenter, MVI Action, or MVC Controller.
It might be tempting to create two test classes for those:
- LoginFragmentTest using ActivityScenario
- LoginControllerTest using probably a simple JUnit
Why does that approach often feel desirable? Because sometimes developers can’t figure out how to do UI testing so they do business logic testing first or somehow business logic testing has higher priority than UI testing. Unfortunately, that strategy can lead to a disconnected test flow. The LoginControllerTest could pass with 100% coverage but if there is a bug in LoginFragment, the result is still a bug in production.
Writing tests against UI components rather than calling handlers directly faithfully simulates user interactions. — Google Testing Blog
We applied this principle in conjunction with the “unit of work” principle and the tests are more understandable as they are more high-level, more maintainable, and represent closer to real user interactions.
UI testing via instrumentation
Even though it can run via Robolectric, instrumentation testing using an emulator is essential which is why we tried to get it right from day one. By running on emulators or real devices we can actually see the user interaction. Furthermore, Robolectric still has some issues with AndroidX that sometimes it’s difficult to make UI tests runnable 100% on JVM.
Espresso already provides us with many useful things like View assertions, granting runtime permissions, stubbing Intent results. Still, there are some things that are difficult to test even with Espresso, such as the Android system’s ContentProvider. For that, a mocking framework may come to the rescue.
Mocking framework should be used very minimally
Using a mocking framework seems convenient at times, but this habit has dangerous side-effects. In particular, it can be very easy to slip into the carelessness of testing the mocked object itself. One should be aware of this rule: don’t modify System Under Test, which also means don’t mock or spy SUT at all. For example, let’s look at the code below:
class MyTest { @Test
fun `when do something then do something else`() {
// Given
var sut = spyk(MyClass())
every { sut.doSomethingElse() } returns Unit // When
sut.doSomething() // Then
verify { sut.doSomethingElse() }
}
}
Unless there is a unit test for the function doSomethingElse() somewhere, the above test is unreliable because the function body of doSomethingElse() can be deleted but the test still passes.
Also, avoid mocking Android classes whenever you can. This includes but not limited to Context, View, Fragment, Activity. Let’s review one by one:
- ❌:
mockk<Context>()
✔️:ApplicationProvider.getApplicationContext() - ❌:
mockk<View>()→ 99% doesn’t make sense, why would you need to mock a View?
✔️: use Espresso - ❌:
mockk<Fragment>()
✔️: use FragmentScenario - ❌:
mockk<Activity>()
✔️: use ActivityScenario - ❌:
mockk<ViewModel>()→ this one seems popular on StackOverflow
✔️: never mock ViewModel, use the real class
When you mock Android classes, you depend on the mocking framework to provide you with the correct stub implementations. But, what if they don’t? What if someday you upgrade the framework and suddenly many of your tests break? It already happened to us and we had to do a major clean up. It’s better to cut the middleman and either use Espresso or create fakes by yourself.
Prioritize quality rather than quantity
Good unit tests will lead to good coverage, while the opposite is not true. Writing bad unit tests is exhausting and the long-term consequence will be messy. At some point in the future, we have to revisit those bad unit tests and do that all over again.
There’s no point in writing a bad unit test, unless you’re learning how to write a good one. If you’re going to write a unit test badly without realizing it, you may as well not write it at all. — The Art of Unit Testing
By applying good principles and creating unit tests based on user interactions, we are more confident of our tests in preventing production bugs. Our focus onward is to implement more best practices, cover more user interactions through unit tests, try to do shift-left testing, and better test suites management.
High test coverage is a byproduct of good implementation but it should never be the first priority to achieve.
