Android BDD with Robolectric and Espresso — how to refactor your Android project tests

I like Android testing not because it’s easy, but because it’s challenging. It takes brainstorming beforehand at the start, safeguards the source code from my own (or anyone else’s) mistakes while ultimately bringing higher quality to the projects I work on. Such a holistic approach is quite challenging as it makes me feel like I work on two different projects at the same time (the Android main project and the testing project of that main project). Apart from the numerous benefits and excellent results, I would also say that testing on Android had my hair turned grey as it proved to be quite a rocky area with the constant issues such as flakiness, different styles to write the different type of tests and delicate details, especially with the UI testing.

In this article I would like to share my experience with all types of tests on Android and how I managed to achieve a nice degree of unification by making all my tests look the same (unit and instrumentation).

In addition, I shall also present how to run the same instrumentation test both on JVM and on Device powered by Android platform(real device or emulator).

If you had the chance to look at my previous articles about BDD with Cucumber and Espresso you had probably noticed how well is writing Instrumentation tests with BDD Cucumber and Espresso working. However, unfortunately, this approach has its own downsides like:

• The need to have Cucumber plugin, dependencies, etc.
• The need to use Gherkin syntax
• The extra caution how to write your steps (key sensitive)
• The need to use Cucumber Annotations, own Cucumber runner, etc.
• When the Cucumber tests number grows over time, opening the tests and highlighting the Gherkin syntax takes extra time
• In the likely events when Gradle updates, Kotlin updates or Android SDK updates, very often some issues occur with Cucumber has issues and all you could do is simply wait for someone address and resolve them
• The inconsistency between The Cucumber tests and Unit tests and how they look like because the way they are written is simply different

Compare between standard Cucumber UI test with standard Unit test

I could continue with the list about Cucumber weaknesses I’ve noticed out of my experience, but that is not the point of the article and I’d rather concentrate on some new things I want to share more importantly.

What do we need:

We need an Android project with the following dependencies:

project dependencies

As you may noticed from our project dependencies, what we are using are some basic Android dependencies, Kotlin for the programming language, MockK for mocking dependencies, Robolectric and Espresso for the UI tests and JUnit for the tests run and the assertions.

For the purpose of the article I shall use classic MVP architecture whereby I have an Activity implementing a Presenter.View interface and a Presenter class where I keep my business logic. Later in this tutorial we shall unit test our Presenter and we shall UI test our View (our Activity class).

I tried to keep the article’s GitHub repository similar to what I used in my previous article (Android BDD with Cucumber and Espresso) so that we can compare it, if necessary.

I used again the Robot pattern for designing my Unit and UI tests as the idea is to achieve full consistency between my UI and Unit tests styles. Here is the link where you can learn more about the Robot pattern. Feel free to choose any other pattern (PageObject pattern or similar) as it shouldn’t be that difficult to adapt my way to your way.

Before we proceed with the code, please pay attention that each of our test classes has its own robot and this robot separates the test steps from the actual test work (setup, mock, assertions, tears down etc.).

Let’s use Kotlin DSL instead of Cucumber

I like Kotlin as much as most of the Android engineers and that is why I’m working towards finding out more ways of developing Android by using the language features. One of these features is definitely the DSL (Domain Specific Language) syntaxes . If you are not familiar with Kotlin DSL, I would encourage you to do so. The Kotlin DSL feature is everywhere — in Gradle scripts, in libraries etc. and below I’m recommending a nice read to start with https://www.raywenderlich.com/2780058-domain-specific-languages-in-kotlin-getting-started.

Next, have a look at the following DSL functions to get an idea later how a test written with them looks like:

Generally, what we have is:

• A simple BaseRobot class which is meant to be parent for each of our test robots (from which they extend from). For example, If I have a test class called “MyTestClass.kt” this class should have its own private robot called Robot which extends from BaseRobot
• A data class called TestRun, representing individual test run metadata (containing same data about the test itself or something useful)
• A few extension functions of our TestRun class, called GIVEN, THEN, WHEN and AND
• A couple of high order functions called RUN_UI_TEST and RUN_UNIT_TEST which require robots as parameters and some block of code which is going to run inside. These functions return an instance of the TestRun class, which allows us to use the TestRun class extension functions GIVEN, THEN, WHEN and AND directly in the RUN_UI_TEST or RUN_UNIT_TEST blocks
• And finally, as you probably spotted, I added some useful printing (logs) in RUN_UI_TEST and RUN_UNIT_TEST methods, helping me to understand when the test starts, finishes, how long it took to complete etc. You can happily experiment and print as many useful things as you might wish.

Perhaps at this point you are asking yourself: “Well, where shall I best place all these DSL functions? Maybe in the test directory of my project? Or better in the androidTest directory of my project?”

Considering our goal to use the same style (same DSL) in the unit tests as well as in the UI tests, it seems we can’t place them neither in the test directory, nor in the androidTest directory. If we do place the DSL functions in the project test directory, we can use them in the unit tests, but not in the Instrumentation tests. Absolutely the same is valid if we place our DSL functions in the androidTest directory.

What shall we do then?

The solution is to create a shared place to put the DSL functions in order to use them for both unit tests run and instrumentation test run. The steps to achieve this are:

1. Create a new folder called “sharedTest” in you app module

2. In your app module build.gradle script add custom sourceSets { } block like this:

android {
   ……….
   sourceSets {
      test {
           java.srcDirs += "src/sharedTest/java"
      }
      androidTest {
           java.srcDirs += "src/sharedTest/java"
      }
    }
    ……..
}

3. Place the DSL functions in the “sharedTest” folder. As soon as you sync the Gradle scripts, they should be ready to use in both unit and instrumentation tests.

On this link you can find some good read on the matter in case you still experience difficulties in the set up.

Ultimately, you can try and even create your own library (SDK), where you can have the DSL functions and use it as a dependency for testImplementation and androidTestImplementation, but you will still need that sharedTest directory, because your have to host your instrumentation tests there.

The Unit testing

Once we have our DSL functions placed in the sharedTest directory, we can easily use them to write our unit and instrumentation tests. The unit tests should be placed in the app module test directory. Here is how our unit test looks like:

What do we have in the unit test:

• We are unit testing a Presenter class called LoginPresenter (here is reference to the LoginPresenter code in the project repository)
• We have two unit tests for the LoginPresenter: shouldClearErrorsOnLogin() and shouldDisplaySuccessfulLogin()
• As mentioned earlier, our test class should have its own private robot class called Robot which extends from BaseRobot. The DSL function RUN_UNIT_TEST is designed for unit tests run, working by way of simply passing the test robot instance inside and later calling the Robot method directly in the GIVEN, WHEN, THEN steps
• Please pay attention to the test construction. Note that each test holds only the steps of the test, but NOT the actual code which builds and runs the test.The responsibility of the latter belongs to the test class robot. In the GIVEN, WHEN, THEN blocks we only call the corresponding Robot’s methods directly thanks to the fact that in these blocks we have our robot instance immediately (directly) available
• In the private Robot class we mock some dependencies, verify some values and do all the test actual work

If you want to draw a simple comparison how the same test would look without the DSL functions and without the Robot pattern, it would end up looking like something of the kind:

With the simple unit test style, we have everything in one place — the tests steps and the test implementation source code. It is inevitable that over time, when the tests number increases and some of the test cases require more code to implement, then it will become really difficult to understand and have a clear picture of what is going on with such tests.

Also with the simple unit test style, we end up duplicating the test implementation code a lot. We keep writing the same implementation in hundreds of unit tests and we don’t share it at all. If for some reason, we are required to change the duplicated code, we shall have to spend days make it different everywhere. Another disadvantage is that we avoid to optimise our tests, because they are so difficult to change and we stick with flaky and legacy test forever.

The Instrumentation testing

With AndroidX test, Robolectric 4.0 and Project Nitrogen , you can write tests in Espresso and run them in either Robolectric on the JVM or in an emulator/real device called Android Run Time (ART). If you go back where we added our project dependencies you will see that we have included Espresso core dependency for the unit tests builds along with the instrumentation test builds. Exactly this step enables us to use Espresso in Robolectric (JVM) runs for verifications and view selections. You can see that Robolectric is not a dependency of the androidTest builds where we use robolectric annotations dependency only, because we need it in our shared tests (see the UI test code below). More about Robolectric and Android X can be found here.

The approach to writing our UI test (placed in the sharedTest directory) is very similar to the way we used to write our unit tests above.

IMPORTANT: At the beginning of this article, where I explained how we can achieve a similar style in writing unit tests and instrumentation tests, I also mentioned that we can have our instrumentation tests running at the same time both on ART and on JVM (without real device). Note that we need to place the instrumentation tests in the sharedTest directory (rather than the usual androidTest directory). The reason is simply because we would like to run our tests with Robolectric along with Espresso too. If we place them in the androidTest directory, we wouldn’t be able to run them on JVM. I really hope that this limitation is gonna be resolved by the Android’s team very soon.

The screen which we are trying to UI test looks like:

Our project UI

The UI test class looks like:

What do we have in our instrumentation test:

• We test an Activity class called LoginActivity (please refer to the project repository to see the LoginActivity code if you are interested)
• We have two UI tests for the LoginActivity: shouldDisplayPasswordLoginError() and shouldDisplaySuccessfulLogin()
• As mentioned earlier, our test class should have its own private robot class called Robot which extends from BaseRobot. The DSL function RUN_UI_TEST is designed for UI tests run, working by way of simply passing the test robot instance inside and later calling the Robot method directly in the GIVEN, WHEN, THEN steps. It’s absolutely the same approach and technique from the unit tests
• The UI tests have only the steps of the test, but not the actual code which builds and runs the test. This responsibility again is delegated to the test class robot
• We use Espresso testing framework to select our Views on the screen and to assert their states (visible, gone, with text etc.)
• The test class is annotated with @RunWith(AndroidJUnit4::class), but that is just because we would like Robolectric to run the test on JVM (and not on ART)
• If we run the same test on ART, then the runner is not the defined one in @RunWith, but what is defined in the app module build gradle file as testInstrumentationRunner. In my setup I use my own runner called MyInstrumentationTestRunner
• I use my own custom test application class called MyTestApplication, which I set up in MyInstrumentationTestRunner for the instrumentation tests (ART) run and in the annotation @Config(sdk = [24], application = MyTestApplication::class) for the Robolectric (JVM) run;
• We have annotation @LooperMode(LooperMode.Mode.PAUSED) on top of our test class. By default, operations run synchronously using this looper which means that many operations shall not happen in the same order that they would occur on a real device. A while ago, this has been an issue with Robolectric and it is good they have come up with a fix for it by adding the annotation before test classes. Additional information on the LooperMode annotation can be accessed here

Few notes and scenarios on running the UI tests:

1. Due to the fact that the UI test is placed in sharedTest, every direct run from Android Studio by using the test play icon (the green play button), is going to run the test as an Instrumentation test. In this case an attached device/emulator is needed before the run.
2. In case a gradle command is used to run the UI test as Instrumentation test, the command in use is:

./gradlew connectedDebugAndroidTest -Pandroid.testInstrumentationRunnerArguments.class={full_path_to_your_test_class}

3. If you want to use gradle command to run the UI test as Robolectric (JVM) test you can use:

./gradlew testDebugUnitTest --tests {full_path_to_your_test_class}

4. If you prefer to run your tests with Android Studio’s run button, you shall have to create Run configurations in AS similar to:

Some considerations and conclusions :

1. Sooner or later, in the course of your project’s lifetime, all Espresso tests will start taking longer and longer to run and complete. The nice option of running them on JVM as well (thanks to Robolectric 4.0 and Espresso) however could be very helpful in this case. You can still keep on running them on a device whenever needed without being restricted to using only this way. I would say beyond doubt that implementing such a flexible testing mechanism shall significantly reduce your tests running time
2. Our tests shouldn’t worry about the running environment, they should only care about asserting expected behaviour. That is why if we have the chance to enhance the run of the slow Android tests by using the JVM faster option, we should definitely use it
3. Avoiding usage of any third party dependency in your project (like Cucumber and Gherkin language) is preferable especially as long as we are able to achieve the same effect with the Android native stack like Kotlin in particular
4. Adopting BDD style in both unit tests and instrumentation tests is amazing and at some point even allows for sharing code ( for instance maybe Robots, maybe data factories etc ). As the project tests pile up and grow and in case there are different types of tests, having all the tests consistent and tidied is such a relief. The differences between the Unit tests and the UI tests should live in the test Robots only, where the test implementation stays
5. Some of you are probably well aware of some custom testing frameworks existing on the market, for example Kaspresso Its effect looks similar to what I’ve described here above. However, bear in mind that it means adding another dependency to your project. Apart from that, I don’t think the UI tests written with Kaspresso are able to run on JVM and also I don’t think you can use Kaspresso to write your unit tests. UPDATE: It looks like Kaspresso could be used for running Instrumentation tests on ART and JVM environments , for more information please check the comments section in this article.
6. Don’t forget that the ideal, always up to date documentation, stays in the tests of the project. If the project tests, no matter what kind of tests, are messy your project documentation is gonna be messy too and nobody shall be able to understand it. By having clear test steps for each test on any occasion (unit, UI etc.) you basically have clear documentation about your project
7. Because I use MockK as mocking library I spotted some flakiness in my Instrumentation tests run when I run them on emulators. There is an open issue on GitHub about that. The workaround at the moment is to use x86_64 processor architecture when you create your emulator where the tests are going to run. MockK is a great mocking library but not the perfect mocking library. I don’t think the perfect mocking library exists for Android at the moment, especially when we are talking about Instrumentation tests. Also If you stick with MockK as I did, I encourage you to run your UI tests on Android 9, 10 or 11 otherwise you may see again some flakiness.
8. Add these settings to your app module build.gradle file:

testOptions {
   unitTests.includeAndroidResources = true
   unitTests.returnDefaultValues = true
}

This will set Robolectric to include Android resources, because Robolectric is not an actual emulator or device, many Android system calls do not actually do anything. The unitTests.returnDefaultValues makes them return a dummy default value in those instances, instead of throwing an exception, which will save the tests from failing.

I hope I managed to exhibit my ideas above as clearly as possible so that you might find the article useful. I would be glad if you share my enthusiasm for this innovative approach to testing in Android. Please, feel free to express your opinion, ideas, modifications, additions or disagreement to my tutorial and add them in the comments sections down below.

Ooo and don’t forget to clap, if you really liked the article and the topic, up to 50 times 🙂.

Thank you!