Android BDD with Cucumber and Espresso — the full guide
Intro
I have read a lot of articles and documentation about how to write Android UI tests (E2E, acceptance etc.), but as you may also have observed, all the information had been sprinkled in multiple sources. Apart from being incompletely descriptive, there had been some small part or detail missing, which made me spend extra time for additional searches at different places. That is why I decided to put my thoughts on this issue and write an article, aiming at more detailed explanation, while having my focus on the main task — taking a step by step approach on how to setup and test the UI in our Android projects with BDD (Behavior-Driven Development).
If you have ever been dealing with Android UI tests, you should know how “painful” this could be. The most unpleasant part probably is that all GUI tests are usually really slow, mostly due to synchronization problems. When the testing framework simulates some actions, it is quite not certain if the GUI is ready for them. Usually, different screens need different time to render all of the UI on the screen, because they have complicated logic or too many requests etc., That is why there is no guarantee that the UI is ready to accept the testing framework actions. In case the UI is not ready for that, even if the tests are correct and the app works as expected, the results of the tests could be flaky and unstable. Therefore, the need for any unnecessary test result checks, would slow down the development process and the releases. In a testing frameworks like Appium, you can see lots of tests with random delays, which at the end of the day means increased total amount of testing time. That is why, when you have a huge amount of test suits, or you pay some money to cloud test labs, you should be really picky for the testing framework. In Android world, the Espresso tests are the fastest thing ever, when we talk about UI Instrumentation tests. Espresso elegantly solves this problem by synchronising the test framework with the queue of GUI events that have to be processed by the system, thus removing finally all the need to introduce arbitrary waits in the tests, along with the main cause of test flakiness.
Below is the link to the article GitHub repository.
Update: If you are interested in a different approach of what we are gonna have in this article, you can check my tutorial about Android BDD with Robolectric and Espresso — how to refactor your Android project tests.
Why recommending Espresso?
- It is a native testing framework, designed specifically for Android native app instrumentation tests;
- It has perfect synchronisation between the tests and the IDLE (UI thread) of the Android platform;
- It has significantly faster execution time than Appium, Calabash or any other testing frameworks for Android;
- The Espresso framework would be placed alongside the app codebase, in the same repository like the source code and thus providing much easier navigation between the tests and the source code of the app;
- Most android developers understand the Espresso testing framework just because it is part of their development kit.
Why adding Cucumber in our UI tests?
- It runs automated acceptance tests written in BDD;
- It works well with Espresso Framework;
- It supports huge number of programming languages and most importantly — all JVM languages;
- It enables anyone to write plain-text descriptions of the desired behaviour in any spoken language and runs automated tests. The latter being possible thanks to its plain language parser called Gherkin, that allows expected software behaviours to be specified in a logical language, understandable for all customers, stakeholders, managers, developers, QAs etc.;
- It provides you with the best documentation about your app.
3, 2, 1 Gooo
To begin with, let us install some plugins in AS, which shall help us with the BDD UI testing approach in our Android project.
Gherkin plugin
In Android Studio we need to install Gherkin plugin. This plugin provides Gherkin language support. Gherkin is the language that Cucumber uses to define test cases (test scenarios). It has been designed to be non-technical and human readable, collectively describing use cases relating to a software system. In a proper software development environment, these use cases would come from the acceptance criterias in the development tickets.
You can find a useful tutorial about how to install plugins in AS here.
Once the Gherkin plugin installation has been completed, you should proceed by restarting your AS.
Next, in order to create .feature files, where you shall describe your feature tests and test scenarios, you need to teach AS how to create these files. In AS, just right click on some of the code directories (for example androidTest) and select New -> Edit File Templates
In the following Dialog, under File tab, press + icon and write down “Cucumber feature file”in the name field and “feature” in the extensions field. After that, press Apply and OK buttons to confirm the new item creation.
As seen from the screenshots above, there is already an existing file extension in place in my AS setup, because I have already completed such setup before. Nevertheless, in your case, it is going to be a newly added one. Next time, when you click right button to create a new file, you shall have the option to create .feature files.
Cucumber for Java Plugin
If you are planning to use Java language in your project, in Android Studio you need to install Cucumber for Java plugin. This plugin enables Cucumber support with step definitions written in Java. It should be borne in mind that omission to install and enable the plugin would mean that the feature files wouldn’t be able to recognise your step definitions in the Java code. In case you intend to use Kotlin syntax in your step definition files, you need to install Cucumber for Kotlin Plugin. For the purpose of illustrating this article I shall use Kotlin and I need to install Cucumber for Kotlin accordingly.
Once the necessary plugins have been installed, the next step is to implement an Android project, where we shall apply the BDD UI testing approach.
You can do that with any of your own existing projects or you can create a new one, using the Android Studio (AS) new project wizard.
What I did is using AS start new project wizard and selecting Login Activity. As a result, I got UI generated with some sample source code. That is enough to write some Cucumber tests with Gherkin and Espresso.
In order to get some source code and UI elements, I simply selected Login Activity generation in the wizard. Later on, I removed some of the source code that AS generated for me, because I did not need it. You can see the final source code here (just copy/paste if you follow my steps).
By reason of keeping the article simple and to the point, I would not include in the project any real HTTP requests. However, If you have an app where you need to test screens with internet requests, I suggest you to mock your servers and return hard coded data in you requests. I consider mocking the server responses as one of the best practices in terms of testing Android UI. It allows test results independance from the network stability issues which enables them to run much faster. It has been widely agreed that the Android UI tests are the slowest tests in the Android testing stack. That is why it is important to seize every opportunity to make them faster and reliable. You can setup mock web server and mock your responses by following this tutorial.
There is also my source code about BDD in the project with HTTP requests , available to have a look here. Although the project is not that simple, in app module androidTests folder, you can see how to mock the requests and test the UI.
Now, let’s return back to our article’s steps and continue with the setup.
Having the project setup in place, what we need is a few new gradle dependencies for Cucumber to add. In app module gradle file just add these new dependencies:
dependencies {androidTestImplementation 'info.cukes:cucumber-android:1.2.5'androidTestImplementation 'info.cukes:cucumber-picocontainer:1.2.5'}
I assume you already have some Espresso dependencies included in your app module build gradle file. If don’t, however, and you don’t know how to do it, it might be helpful to follow this link.
We need these dependencies, because we are planning to write the Cucumber step definition implementations with Espresso.
This is how my app module build gradle file looks like with the Espresso dependencies included
dependencies {androidTestImplementation 'com.android.support.test:runner:1.0.2'androidTestImplementation 'com.android.support.test.espresso:espresso-core:3.0.2'androidTestImplementation 'info.cukes:cucumber-android:1.2.5'androidTestImplementation 'info.cukes:cucumber-picocontainer:1.2.5'}
The next step is to create some packages where we shall store:
1. Our Cucumber configuration setup;
2. Our Cucumber feature files(scenarios, step definitions);
3. Our Espresso test implementations.
How to setup the Cucumber configuration in the project
In app module -> androidTest folder create new package, named test and create one new class file named CucumberTestCase (the name does not really matter) and put this source code:
import cucumber.api.CucumberOptions;import cucumber.api.CucumberOptions@CucumberOptions(features = ["features"],glue = ["com.sniper.bdd.cucumber.steps"],tags = ["@e2e", "@smoke"])@SuppressWarnings("unused")class CucumberTestCase
CucumberTestCase class should stay empty and we don’t need any implementation inside. This is just placeholder class where CucumberOptions annotation lives. In addition, you may also noticed @SuppressWarnings(“unused”) , which indicates that we are not contemplating to use this class at all.
The CucumberOptions annotation is mandatory and relevant only for a particular class amongst all in the test project — the first annotated class. The one that has been found will be used by Cucumber, any others will remain ignored. If no class is annotated, an exception is thrown. What I tried was to put CucumberTestCase in some random package, but that didn’t work well and Cucumber complained about missing initialisation annotation. Then I found out that it was better to place the annotated class in a separate package named test. Once having CucumberTestCase in the test package, Cucumber was able to initialise the tests and find the annotation. The following image shall provide you with an idea of how your androidTest folder should look after this step has been accomplished.
Path to the .feature files
Although CucumberOptions annotation has a lot of different optional settings, I consider above all else to specify at least the features path. In our example this is done by features = “features”. Basically, what the setting says is “Features files must be placed inside assets/features/ of the test project”.
What Cucumber would do is to look for the feature definition files (.feature files), based on the path setup in features setting. The root directory , however, remains always assets.
We should opt for creating assets folder in androidTest folder as well as another directory features in the assets folder, where we shall place our Cucumber feature step definitions. You can name the directory as you wish, but make sure to change the path in CucumberTestCase file accordingly.
Now in features directory you can place your Cucumber .feature files.
Path to the glue files
The glue setting in the CucumberOptions annotation is about Cucumber path for the Step implementations source code. This option is not mandatory, I would not regard this indispensable as it rather helps Cucumber and people to understand where the Espresso source code is. In our case we proceed by creating in androidTest folder new package, named cucumber and then another new one, named steps in cucumber folder.
Note that later, within in this steps folder, we shall put out Kotlin test files with the actual step implementations.
The setting for the BDD test tags
In CucumberTestCase class we have setup the tags setting, with
tags = ["@e2e", "@smoke"]This is another option in Cucumber where we can specify our custom tags and apply these to the Cucumber feature definitions in the .feature files. It provides highly practical solution when you deal with many different .feature files, covering all the different functionality of your application. You may encounter a situation in your daily work, where you aim at executing just a subset of all UI tests (@smoke or @e2e or something else). In this case you would find Cucumber tags setting most helpful. It clearly makes possible running your Cucumber BDD tests with the desired specific tag and thus avoiding to activate the entire UI tests bundle. Consequently, you may appreciate how helpful running the tests only for specific app feature could prove in terms of time saving.
For any additional Cucumber annotation settings have a look this link.
How to write Cucumber feature definitions (.feature files)
Before we go into details about our test step definitions, we should be aware of how our UI looks like. This shall help to understand the necessity to define certain BDD steps. Let us have a picture of how the test app UI looks now:
And this is how it looks after successful Login:
Next, it is time to create out first .feature file where to apply the test step definitions written in Gherkin language.
For the purpose, go to assets/features folder and create new file, named login_details.feature. Then place the following Gherkin code:
Feature: Enter login details@smoke@e2eScenario Outline: Successful loginGiven I start the applicationWhen I click email fieldAnd I enter valid email <email>And I close the keyboardAnd I click password fieldAnd I enter valid password <password>And I close the keyboardAnd I click sign in buttonThen I expect to see successful login messageExamples:| email | password || abv@mail.com | 123456 |
If you haven’t been dealing with Gherkin and Cucumber before, I would recommend as a starting point the following reading on the topic here. It is a helpful document as regards the syntaxis and the setup for the feature steps.
This Gherkin code itself is quite descriptive about what we are going to test. It is perfect documentation and perfect acceptance criteria for our login feature in my opinion. Let us examine the Gherkin code in detail:
- Feature: Enter login details — it is just a text description of the feature file;
- @smoke and @e2e — are our custom Cucumber tags that should help us to run subsets of BDD UI tests. If you remember we defined them in CucumberTestCase class where our CucumberOption annotation stays;
- Scenario Outline: Successful login — Outline keyword added to Scenario keyword allows you to execute the same scenario with different parameters. Instead of defining the parameters as part of the scenario, you can add an examples section with your test parameters ( email and password in the above example). Indeed, every single row in the Examples section will present a new UI test executed in the same scenario, but with different values for our particular parameters. Of course, it should be noted that expanding the rows number in Examples section inevitably would increase the testing time.
- Given, When, And, Then — Each step starts with Given, When, Then, And, or But. These are additional Gherkin keywords, helping us to define our test steps. Cucumber executes each step in one scenario at a time, in the sequence you’ve written them in. When Cucumber tries to execute a step, it looks for a matching step definition implementation to execute. In our case we will implement the steps with Espresso framework. More useful information about the above mentioned special keywords is available here.
Considering that we don’t have yet any implementations about our Cucumber steps until now, AS will prompt us to create them.
How to write Espresso step definition implementations
There are two options available here. The first one is to hover the cursor in AS on one of the steps and press alt + Enter (option + Enter for Mac). This will suggest you to create new step definition:
If you have already had some Kotlin file available where you could add the new step definition for your scenario, you may reuse it. Alternatively, you can create new Kotlin file and store it there.
The second option is to manually create Kotlin class file and then start adding the step definition implementations there. I suggest that we follow this path now for the purpose of this article.
Let‘s create new Kotlin class in our steps folder. I choose to name it LoginDetailsSteps.kt. Feel free to put any name you wish though.This class will hold all of the step definitions for our test scenario. Here is the source code for the new file:
class LoginDetailsSteps {private val robot = LoginScreenRobot()private val activityRule = ActivityTestRule(LoginActivity::class.java, false, false)@Given("^I start the application$")fun i_start_app() {robot.launchLoginScreen(activityRule)}@When("^I click email field$")fun i_click_email_field() {robot.selectEmailField()}@And("^I close the keyboard$")fun i_close_the_keyboard() {robot.closeKeyboard()}@And("^I enter valid email (\\S+)$")fun i_enter_valid_email(email: String) {robot.enterEmail(email)}@And("^I click password field$")fun i_click_password_field() {robot.selectPasswordField()}@And("^I enter valid password (\\S+)$")fun i_enter_valid_password(password: String) {robot.enterPassword(password)}@And("^I click sign in button$")fun i_click_sign_in_button() {robot.clickSignInButton()}@Then("^I expect to see successful login message$")fun i_expect_to_see_successful_login_message() {robot.isSuccessfulLogin()}}
To summarise what we have here:
We have ActivityTestRule, which help us to start our login screen (LoginActivity.kt).
We have lots of methods annotated with @Given, @When, @And, @Then. Basically, these methods represent the glue between Cucumber and Espresso. These are our step definition implementations. The only thing that has left for us to do in these methods is to simply call the corresponding robot method and the robot shall do the work for us. It is noticeable that some of the methods have parameters. If you trace the steps back in login_details.feature file, it becomes clear that these are our scenario parameters indeed, so nothing complicated here.
We also have LoginScreenRobot class, where we actually use Espresso framework to test the UI elements. Using the robot pattern allows us to reuse the same robot implementation in some other UI tests. Further, the same is applicable in cases of End-to-End tests where we may want to inject few robots in one big test as well as to test multiple screens at the same time (for more complicated scenarios). It is worth trying to encapsulate this code so that you can reuse it among different step definitions or share it with the regular Espresso tests (not BDD). This way avoids code duplication, helps you separate concerns and keeps your codebase clean.
I’ve been using test robots in the tutorial. A good example for implementation for Android can be found here.
Next, we move to creating our login robot class. In androidTest folder we should have an espresso folder. We are going to create new package (name it login) there. This is where we need to create a new LoginScreenRobot.kt class, with the following content:
class LoginScreenRobot {
fun launchLoginScreen(testRule: ActivityTestRule<LoginActivity>) {testRule.launchActivity(null)}fun selectEmailField() {onView(withId(R.id.email)).perform(click())}fun selectPasswordField() {onView(withId(R.id.password)).perform(click())}fun enterEmail(email: String) {onView(withId(R.id.email)).perform(typeText(email))}fun enterPassword(password: String) {onView(withId(R.id.password)).perform(typeText(password))}fun closeKeyboard() {Espresso.closeSoftKeyboard()sleep(100)}fun clickSignInButton() {onView(withText(InstrumentationRegistry.getTargetContext().getString(R.string.action_sign_in))).perform(click())}fun isSuccessfulLogin() {onView(withId(R.id.successful_login_text_view)).check(matches(isDisplayed()))onView(withId(R.id.successful_login_text_view)).check(matches(withText(R.string.successful_login)))}}
The Robot source code above reveals that we use Espresso to select views, enter texts, perform clicks or verify things.
Setup your own Cucumber Runner class and the Cucumber tags
In order to run our BDD tests, Cucumber needs its own instrumentation class. We need to create a new package with name runner and then create a new class in that package, named CucumberTestRunner. This new class extends from AndroidJUnitRunner and initialises the CucumberInstrumentationCore. Take a look our test runner code:
public class CucumberTestRunner extends AndroidJUnitRunner {private static final String CUCUMBER_TAGS_KEY = "tags";private static final String CUCUMBER_SCENARIO_KEY = "name";private final CucumberInstrumentationCore instrumentationCore = new CucumberInstrumentationCore(this);@Overridepublic void onCreate(final Bundle bundle) {String tags = BuildConfig.TEST_TAGS;if (!tags.isEmpty()) {bundle.putString(CUCUMBER_TAGS_KEY, tags.replaceAll("\\s", ""));}String scenario = BuildConfig.TEST_SCENARIO;if (!scenario.isEmpty()) {scenario = scenario.replaceAll(" ", "\\\\s");bundle.putString(CUCUMBER_SCENARIO_KEY, scenario);}instrumentationCore.create(bundle);super.onCreate(bundle);}@Overridepublic void onStart() {waitForIdleSync();instrumentationCore.start();}}
The next thing is to tell Android how to use this instrumentation in the build gradle file. It can be accomplished by setting the testInstrumentationRunner option in gradle to our CucumberTestRunner class. It should be noted here that, this way, CucumberTestRunner would be used for every test located in your androidTest folder. However, in most cases, we would not wish to use the same Cucumber runner to run all of the UI tests, but rather the BDD tests only. Sometimes, as well, we may need to run pure Espresso tests (without BDD). In such case, we should have the option to skip our custom Cucumber test runner and use some other runner instead. Therefore, in order to setup multiple custom Instrumentation test runners, a good solution is to have a gradle method that selects the right runner for our UI tests.
Place this code in your app module build.gradle file:
defaultConfig {testInstrumentationRunner getInstrumentation()}def getInstrumentation() {project.hasProperty(‘cucumber’) ?'com.sniper.bdd.cucumber.runner.CucumberTestRunner :'android.support.test.runner.AndroidJUnitRunner'}
After that we should be able to run our BDD test with the CucumberTestRunner, by using the command line with this:
./gradlew connectedCheck -PcucumberPcucumber is just a custom parameter that we pass to gradle and then getInstrumentation() method will do the rest of the work.
Probably you noticed BuildConfig.TEST_TAGS and BuildConfig.TEST_SCENARIO variables in our CucumberTestRunner class. These constants are helping us to test individual tests by selecting test tags or test scenarios.
In addition, we need to add a little bit more code in the app module build.gradle file:
android {
.....
.....buildTypes {
....
....debug {buildConfigField ‘String’, ‘TEST_TAGS’, ‘“‘+getTestTags()+’”’buildConfigField ‘String’, ‘TEST_SCENARIO’, ‘“‘+getTestScenario()+’”’}}
}
def getTestTags() {project.getProperties().get('tags') ?: ''}def getTestScenario() {project.getProperties().get('scenario') ?: ''}
Having completed this, now we can run a single test or subsets of tests via annotations with number of commands, for example:
./gradlew connectedCheck -Pcucumber -Ptags="@smoke"Or
./gradlew connectedCheck -Pcucumber -Pscenario="Successful login"The first command will run all feature tests annotated with @smoke, while the second one will run only the feature with name “Successful login”. Right now, both commands run the same test, because we have just one test, but in real project you will have much more single tests and probably more annotations and then this separation becomes handy.
It is time to have a glance at how the project looks like finally:
Some last words and tips
- Before and After
One thing to pay attention to is when running your BDD tests to make sure you shut down your activities (tests) after each scenario completes, so the next scenario will run on time and without any problems. The consequence of neglecting this is that it would make Android Studio throw exceptions and errors. Even if your tests are well written, you are still going to experience failures. In order to shut down your activity after each scenario ends, you need to use Cucumber @Before and @After hooks. Beware not to mix them with the JUnit @Before and @After ones. In this relation, you may go to the git repository of the article and make reference about the way I’ve been using them. Broadly speaking, @Before helps to setup stuff before the Cucumber test runs (for example init some data etc.), while @After aids to clear your tests from mock servers, mock data, running processes , Activities etc, after Cucumber finishes with the scenario. Additional information on this subject matter can be found here.
2. Testing Pyramid
Despite having both unit tests and integration tests, you would probably still want to go for a small number of end-to-end tests to verify the system as a whole. Achieving the right balance between all three test types has been very well presented by the testing pyramid below:
We can see that the unit tests are at the bottom as they form the base of the pyramid. Moving up the pyramid means your tests get larger, while at the same time the number of tests (the width of your pyramid) gets smaller.
Some experts suggests a 70/20/10 split between these tests: 70% unit tests, 20% integration tests, and 10% end-to-end tests, but I would say that it is not one-fits-all solution. More important, in my view, is to have unit tests whenever possible, to create Integration and E2E tests within some reasonable amount of execution time. Some of you might find useful a strategy to slit their Integration and E2E tests in subsets (with annotations and tags) and run only few of them with PRs and nightly CI jobs, and run all of them just before releases (final regression tests). Think of the E2E tests as of your worst possible case. If you have a small number of E2E tests, the overall runtime of all your tests will still be quite reasonable. However, if you use E2E tests predominantly, then your test runtime (and the number of test flakes) will inflate significantly which shall have an impact on the team’s productivity and efficiency. Additional information on the topic could be found here.
IMPORTANT NOTE:
When we are talking about Mobile testing the pyramid above is not really accurate and doesn’t represent what exactly is happening on mobile testing. For more information I would suggest you to read this article where the real mobile testing pyramid is actually flipped.
3. Setup your UI Test results in slack
I’d like to note that it is always good to have your test result reports printed in a report channel, available to everybody there to view. I think Slack is probably one of the best ways to print some results after your UI tests finished. Here is a useful link how this could be done.
4. Start from the UI tests
Some of you may observed, that this article doesn’t actually follow BDD style development. I did that with a purpose, because otherwise it will make the things a little bit more complicated. The main focus here was how to setup Cucumber + Espresso and make the tests run, not how to write your UI with BDD. I advice you to give a try the real BDD style development, where you start from writing test scenarios and finish in the implementation of the UI.
To put some last words, I hope that all interested in the topic will find this long article feasible and helpful for the Android UI test setup with BDD and Cucumber.
Update: For those who use AndroidX (probably everybody), please visit the new Cucumber Android project here and use the latest version of the cucumber Android library. This will help you to solve any issues with the tests run.
There is an open Github project here where you can see simple implementation of the latest version of Cucumber and Android X. I don’t think there is a much difference between the original article and the usage of AndroidX and latest Android Cucumber libraries. The only major difference so far is the moment when we need to extend our custom test runner. With AndroidX and latest Cucumber version we need to extend from CucumberAndroidJUnitRunner.java (this is Cucumber class), where in the old versions of Cucumber (the method described in my article) we had to extend from AndroidJUnitRunner.java (this is Android SDK class).
