Our Journey Towards a Testable Android App

How we refactored our legacy Android codebase to make it testable.

Photo by Patrick Fore on Unsplash

The Wantedly Android app was first launched in 2014, which was some time ago, hence our codebase for our Android app is quite old. When I joined Wantedly 3 years ago, there were quite a few issues with the test suite that was in place. The number of test cases was low and it was not possible to produce a test coverage report. The test cases that we have were also quite flaky and difficult to maintain. Whenever we want to update or add a new test case, it required a lot of complicated setups and it was very difficult to test a single component of the app independently.

We knew that we need to improve this if we want to scale our development and improve the productivity of the Android engineers. With this post, I want to share what we did to make our Android codebase testable.

Defining the Objectives

I think it’s really important that we define what we need to implement and achieve. In our case, we created a document that outlines the goals, benefits, strategy, and conventions of writing tests. This document also served as the centerpiece of discussions for the engineers and the stakeholders so that everybody is on the same page, and can also be used for references for new members in the future. I’ll try to briefly show what our document looks like.

Goals: To create a solid framework and infrastructure to easily build, test, and release our apps with ease of mind.

Benefits: A reliable test suite allows engineers to make changes quickly and release the app confidently. It provides a shorter feedback cycle during development, making it easier to debug issues in the application. It also allows us to simulate hard-to-replicate edge cases consistently. A well-documented test suite can also be used as documentation to help in knowledge sharing.

Strategy: Structure the test suite with the testing pyramid (we’ll talk more about this in detail later). We also need each “component” in the app to be independently testable, but still can be tested together if needed. We will run the test suite on every pull request. Any pull requests to the main branch need to have added/removed test cases, and we’ll also ensure that the coverage doesn’t drop.

Conventions: Ensure that the test cases are consistent across the codebase. We will use a unified test method naming (something like givenCondition_whenAction_thenResult) to ensure consistency and make it easier to write and maintain. Consistent test method naming allows us to quickly identify which test case is failing in CI. The test cases should be small and test only 1 aspect of the code.

Our Approach to Testing

The best strategy to structure your test suite will depend on your team and codebase. Here, I will describe what our approach was in introducing testable architecture into our legacy codebase.

Testing Pyramid

Testing pyramid allows us to group tests into buckets depending on the number of components involved, execution time, and maintenance effort of each group. It also gives us an idea of how many test cases we need to write for each group. Ideally, you want to write a large number of test cases that are easy to maintain. Hence, you want a smaller number of tests as you go up the pyramid. Typically, from the top, you’ll have the end-to-end tests, then integration tests, and then finally the unit tests at the bottom of the pyramid.

Here’s how we structured our test suite using the testing pyramid:

At the top of the pyramid, we have our end-to-end testing with Firebase Robo Test. Then, we have our Fragments’ Espresso UI Tests, which are run as instrumented tests. We test the Fragments together with the ViewModel as our Fragments are tied heavily with the implementation of our ViewModel. We also replace any dependencies that are required by the ViewModel (such as repository classes) with test doubles to allow easier testing. Finally, we have repository and utility class tests that are run as local unit tests with Robolectric.

Note: Repository classes that interact with SQLite database, should be run as instrumented tests.

Test Doubles

Test doubles are replacement objects for classes that we use in production, whose behaviors can be controlled during testing. We use test doubles to create a predictable condition for our tests. They are also very important to allow us to test our components in isolation. In Wantedly, we primarily make use of 2 types of test doubles, mocks, and fakes.

Mock: a type of test double that can respond to a set of method calls whose answers have been defined. It can also check whether a method in the test double has been or has not been invoked. Some of the popular libraries that can help you create mock classes in Android are Mockito and MockK.

interface UserRepository {
  suspend fun getUser(userId: Int): User
}
// Create a mock UserRepository with MockK
val mockRepo = mockk<UserRepository>()
// Set getUser to return a fake User object every time it's called.
coEvery { mockRepo.getUser(any()) } answers {
  User(
    id = firstArg(),
    firstName = "John",
    lastName = "Smith"
  )
}
// Usage
launch {
  mockRepo.getUser(1) // Will return User(1, "John", "Smith")
}

Fake: a type of test double that uses some sort of “shortcut” implementation to act like the real class. If you use interfaces to declare your class dependencies, then creating a fake implementation is quite straightforward.

interface UserRepository {
  suspend fun getUser(userId: Int): User
}
// Create a fake implementation of a UserRepository
class FakeUserRepository : UserRepository {
  override suspend fun getUser(userId: Int): User {
    return User(
      id = userId,
      firstName = "John",
      lastName = "Smith"
    )
  }
}

Writing Tests

As I mentioned above, we introduced several conventions, such as test method naming, to ensure that the tests are consistent across the codebase. This helps us describe the scenario being tested, and also helps us quickly identify regression in our code. We also treat our test code as production code, meaning that SOLID principle still applies to the test code.

Other than that, we also use other third-party libraries to help us write test cases more easily. For example, we are using Kakao, a Kotlin DSL wrapper for Espresso, to write our UI test as we find the DSL is much more readable. We also wrote a library to create fake objects using reflection to help generate fake data for testing.

Since most of the components that we’re testing are similar to each other (Fragments, ViewModels, etc). We created a few JUnit rules to make setting up those tests easier. For example, we have a FragmentTestRule that contains several other JUnit rules such as InstantTaskExecutorRule, TaskSchedulerRule, and DataBindingIdlingResourceRule to ensure that the Fragment tests are deterministic and not flaky.

Our approach was UI testing to test UI logic, not the appearance of the UI element. Checking whether a Button is displayed correctly in a specific color or size with Espresso is difficult, however, you can easily check whether that button has the correct label or whether it is clickable. If you want to test the appearance of a Button, I’d suggest you take a look at screenshot testing instead.

Testing Components in Isolation

The ability to test components independently is important to ensure that we can focus on writing the test cases for each specific component. To achieve that, we make use of dependency injection such as Dagger and constructor parameters to pass in the required dependencies for each class. This allows us to easily replace those dependencies with test doubles during testing. Consider the following example:

interface UserApi {
  suspend fun getUser(userId: Int): User
}
interface UserRepository {
  suspend fun getUser(userId: Int): Flow<User>
}
class UserRepositoryImpl(
  val userApi: UserApi, 
  val userDb: UserDB
) : UserRepository {
  
  override fun getUser(userId: Int): Flow<User> { 
    // Fetches user data from UserAPI and store it 
    // in the local DB if necessary.
    ... 
  }
}
class UserViewModel(
  private val userId: Int,
  private val userRepository: UserRepository
) : ViewModel() {
  private val _user: MutableLiveData<User> = MutableLiveData()
  val user: LiveData<User> = _user
  
  init {
    viewModelScope.launch {
      // Observe changes from UserRepository.
      userRepository.getUser(userId).collect { user ->
        _user.value = user
      }
    }
  }
}
class UserFragment : Fragment() {
  
  // Assume that UserViewModel is injected through Dagger.
  @Inject
  lateinit var viewModel: UserViewModel
  
  override fun onViewCreated(view: View, savedInstanceState: Bundle?) {
    super.onViewCreated(view, savedInstanceState)
    ...
    // Observe state changes from LiveData.
    viewModel.user.observe { ... }
 
    someButton.setOnClickListener {
      // Use Navigation Component to handle navigation events.
      findNavController().navigate(R.id.to_other_destination)
    }
  }
}

When testing UserRepository, we can replace the implementation of UserApi like so:

@RunWith(AndroidJUnit4::class)
class UserRepositoryTest {
  @JvmField
  @Rule
  val repositoryTestRule = RepositoryTestRule()
  
  @MockK
  lateinit var userApi: UserApi
  
  @Before
  fun setUp() {
    MockKAnnotations.init(this)
  }  
  
  @Test
  fun givenUserApiReturnsUser_ThenEmitUser() = runBlockingTest {
    
    // Set up mocked UserApi to return a fake user.
    coEvery { userApi.getUser(any()) } answers { 
      User(
        id = firstArg(),
        firstName = "John",
        lastName = "Smith"
      )
    }    
    
    // Replace the dependencies of UserRepository with mocks.
    val userRepository = UserRepositoryImpl(
      userApi, 
      repositoryTestRule.db
    )
    
    // Assert that repository emits the fake user.
    val user = userRepository.getUser().single()
    assertEquals("John", user.firstName)
    assertEquals("Smith", user.lastName)
  }
  ...
}

When testing UserViewModel, we only need to replace the UserRepository implementation:

@RunWith(AndroidJUnit4::class)
class UserViewModelTest {
  @JvmField
  @Rule
  val viewModelTestRule = ViewModelTestRule()
 
  @MockK
  lateinit var userRepository: UserRepository
  @Before
  fun setUp() {
    MockKAnnotations.init(this)
  }
  
  @Test
  fun givenUserRepostiroyEmitsUser_ThenUserLiveDataIsUpdated() {
    
    // Set user repository to emit a fake user.
    coEvery { userRepository.getUser(any()) } answers { 
      flowOf(
        User(
          id = firstArg(),
          firstName = "John",
          lastName = "Smith"
        )
      )
    }
    
    // Replace the dependencies of UserViewModel with mocks.
    val viewModel = UserViewModel(1, userRepository)
    
    // Assert that user LiveData emits the fake user.
    val user = viewModel.user.getOrAwaitValue()
    assertEquals("John", user.firstName)
    assertEquals("Smith", user.lastName)
  }
}

You can find getOrAwaitValue extension here.

As I mentioned before, we usually test our Fragment and ViewModel together, hence here’s how we test UserFragment:

@RunWith(AndroidJUnit4::class)
class UserFragmentTest {
  @JvmField
  @Rule
  val fragmentTestRule = FragmentTestRule()
 
  @MockK
  lateinit var navController: NavController
 
  @MockK
  lateinit var userRepository: UserRepository  
  
  @Before
  fun setUp() {
    MockKAnnotations.init(this)
  }
  
  // Launch the fragment using FragmentScenario.
  private fun launchFragment(): FragmentScenario<UserFragment> {
    return launchFragmentInContainer {
      
      // Create the Fragment manually and replace any
      // dependencies as needed. 
      UserFragment().also { userFragment ->
        userFragment.viewModel = UserViewModel(1, userRepository)

        fragmentTestRule.dataBindingIdlingResourceRule
          .setFragment(userFragment)
      }
    
    }.onFragment { userFragment ->
      
      // Replace the NavController with a mocked NavController
      // to test UserFragment in isolation.
      Navigation.setViewNavController(
        it.requireView(), 
        navController
      )
    
    }
  } 
  
  @Test
  fun givenUser_ThenDisplayUserDetails() {
    
    // Set fake user similar to ViewModel.
    coEvery { userRepository.getUser(any()) } answers { 
      flowOf(
        User(
          id = firstArg(),
          firstName = "John",
          lastName = "Smith"
        )
      )
    }
    
    launchFragment()
        
    // Assert views with Kakao.
    onScreen<UserScreen> {
      firstNameTextView.hasText("John")
      lastNameTextView.hasText("Smith")
    }  
  }
  
  @Test
  fun whenClickSomeButton_ThenNavigateToOtherDetailsFragment() {
    launchFragment() 
    
    // Trigger UI interactions with Kakao.
    onScreen<UserScreen>{ 
      someButton.click()
    }
    
    // Verify navigation events using the mocked NavController.
    verify { navController.navigate(R.id.to_other_destination) }
  }
}

You can also use TestNavHostController as a replacement for your NavController.

This kind of setup also allows us to easily combine components during integration tests. For example, we can use Dagger to replace only the UserApi with a mock API client during testing, and keep the rest of the dependency with the real implementation.

Pull Requests and Coverage

We run our test suite for each pull request on our CI server and this allows us to measure the code coverage of each branch. However, we don’t use the coverage percentage to guide us on how many test cases we have to write. We believe that we need to focus on the most critical path of the codebase, rather than the number of test cases.

Given that we are dealing with legacy code, we need to introduce the test suite gradually. It’s impossible to refactor the whole application at once because, with that approach, we will not be able to develop any feature in parallel. Without parallel development, we will not be able to deliver any updates or fixes to our users. Thus, we decided to write test cases for components that we’re are currently working or refactoring on, and track our overall progress using code coverage.

Code coverage report allows us to easily find out which components have been or have not been tested. In addition, we also introduce a minimum coverage threshold and increase that threshold gradually to ensure that we don’t add more code that has not been tested into the codebase.

Issues that we Encountered

The whole journey towards a testable app was not all smooth. Here are several issues that we encounter during the refactoring and how we dealt with these issues.

• Mismatches between actual implementation and the testing document. When we first created our test document, there were a lot of unknowns. What we ended up implementing is quite different from what we initially wanted. Hence, we needed to iterate on the testing document. We needed to adapt to technical changes and adjust our approach accordingly. For example, we wanted to make use of gherkin syntax with Spek to help us structure our test suite. However, Spek does not currently support instrumented tests for Android. Given that we want to maintain consistencies between our local unit test and instrumented test (as we were hoping for Project Nitrogen), we decided not to use Spek.
• Writing and maintaining an automated test suite requires a lot of time investment. Implementing proper test cases will inevitably require a lot more time. We as engineers need to communicate that to the stakeholders and manage their expectations. However, I believe that in the long run, writing automated tests is a good investment to make since most of the manual tests are quite repetitive and can be automated. Investing in an automated test suite also allowed us to make changes to our code faster.
• Execution time gets longer as the test suite grows. The number and type of tests in the test suite will heavily impact the execution time of the testing pipeline. If there’s a large number of UI tests, the execution time can be very long and it can slow down the development. For us, we have about 1000 unit tests and 300 UI/ integration tests. Running our test suite on our CI server takes about 5 minutes for unit tests and 15 minutes for UI tests. Executing UI tests takes a lot of time even after we introduced Flank to shard our tests. If you have a bigger test suite, you might want to take a look at Dropbox’s approach to selectively run the test suite for code that changed or that could have been affected by the change.
• Generating code coverage with Jacoco can be difficult. Even though support for Jacoco comes out of the box with Android Gradle Plugin. Setting it up can be quite difficult, especially when upgrading AGP’s version. In our experience, Jacoco will often throw compilation errors or show incorrect coverage reports when AGP is updated. As you can see in our branch coverage graph, we encountered a big dip in the coverage due to this issue when we updated AGP to 4.2. There are also other issues that you have to pay attention to, such as this, this, and this.

Wrap up

While it took a lot of time to see the benefit and result of our effort to implement a testable app, I believe that the testing setup for our Android app has improved significantly in the last 2 years. Of course, we can’t just stop here because there’s still a lot of room for improvement. Creating a testable app needs consistent efforts from everybody, and it’s very important to get buy-in from your team members and stakeholders before starting your journey in implementing a testable architecture. Writing automated tests should be an integral part of your development process and it should not slow down the pace of the development.

An automated test suite can help you quickly identify regression during development. However, I don’t think you can 100% rely on an automated test suite because there are problems that a manual QA process can find more easily, such as animation issues, UI or layout issues, incorrect use of test doubles, or unhandled edge cases, etc. So, it’s still important to have a manual test in your development cycle.

Thank you for reading this article. I hope it can give you some ideas on how to implement a testable architecture in your Android app.