Into the Belly of the Beast: UI Testing an Android App with Cucumber and Espresso

It’s 3a.m. You’re on your 5th Redbull, this one might have had an extra dash of something in it to dull the pain. You look at the screen. It blinks red back at you.

FAILURE: Build failed with an exception.

Fuck.

Anyone working in the world of Android testing has probably seen this message hundreds of times — but why is it so difficult to get around?

In short, its the general state of testing on the Android platform. Essentially this is the wild west of test automation. You go in with a few ideas and guns blazing and hopefully after a few tries, the survivors stumble out, dazed and confused, but alive. UI testing for mobile apps has only been around as a field for 8 years and tools are just now coalescing into sets of standards and best practices. Information is scattered, documentation is out of date. Libraries are compiled multiple times with conflicting versions.

If you’ve ever tried to link a few of these tools together, this phrase might hold a special place in your heart.

So, what’s the answer?

The Behavior Driven Design of Cucumber. Plus a turbo charged shot of Espresso 2.0

Cucumber is a behavior driven development tool. In short — the tests you write for your automation suite also act as acceptance criteria for the feature development. Cucumber has been around for a while — since 2008 — and chances are you have stumbled across it. Espresso is a more recent addition to the testing family. Released in November 2013 with version 2.0 coming out in December 2014, Google finally gave the Android testing world a tool it needed. Espresso is lightweight, fast, and reliable. The question was how these two mix. Can I apply the human readable and behavior driven framework of cucumber to the data-driven approach of espresso?

Yes, but it’s not easy.

Phase 1: Understand that Espresso and Cucumber can work together well, but aren’t necessarily designed to work well together.

Cucumber, as mentioned above, is meant to be the driver for User Interface tests in plain english format. Its appeal is being user friendly for Product Managers, Dev Managers, Business executives, and anyone else with a non-technical or moderately technical interest in seeing how the application in question is being tested. It’s also appealing for being language agnostic in how step definitions are executed.

Espresso, on the other hand, is supposed to be a testing framework by engineers, for engineers. It interacts directly with Android’s R.id framework and Data adaptor layers with onView and onData respectively. It’s not made to be English friendly, but app development friendly. Espresso isn’t designed for layering abstractions such as Cucumber on top of it.

Phase 2: Adding layers without making it a house of cards.

Ultimately, every tester wants a stable framework to work with. The next few sections will run through the setup of what should be an error-free Cucumber layer on top of Espresso tests.

Dependencies are hard.

Test frameworks can be notoriously finicky in what they expect in order to operate. Cucumber + Espresso + Gradle is unfortunately no different. Without careful dependency management, you’ll end up mired in build issues which is the developer equivalent of purgatory. Below is a sample Gradle file from a working project I’m using to test a real-world app. It looks like a mess, but this is the result of cleaning up a seemingly endless string of dependency errors from library compilation collisions.

//Runner
androidTestCompile( 'com.android.support.test:runner:0.4.1' ){
    exclude module: 'junit'
}
//Cucumber
androidTestCompile('info.cukes:cucumber-junit:1.1.4') {
    exclude module: 'cucumber-jvm-deps'
    exclude module: 'cucumber-core'
    exclude group: 'org.hamcrest', module: 'hamcrest-core'
}
androidTestCompile('info.cukes:cucumber-android:1.2.4@jar') {
    exclude module: 'cucumber-jvm-deps'
}
androidTestCompile('info.cukes:cucumber-picocontainer:1.2.4') {
    exclude module: 'cucumber-core'
    exclude module: 'cucumber-jvm-deps'
}
androidTestCompile('info.cukes:cucumber-jvm:1.2.4') {
}
androidTestCompile('info.cukes:cucumber-core:1.2.4') {
    exclude module: 'cucumber-jvm-deps'
}
androidTestCompile('info.cukes:cucumber-jvm-deps:1.0.3') {
}
//Espresso
androidTestCompile( 'com.android.support.test.espresso:espresso-core:2.2.1' ){
    exclude module: 'junit'
    exclude module: 'runner'
}
androidTestCompile( 'com.android.support.test.espresso:espresso-contrib:2.2.1' ){
    exclude module: 'espresso-core'
    exclude module: 'support-v4'
}

If you run into this error while setting up your cucumber dependencies in Gradle

Error:Execution failed for task ':app:packageAllDebugClassesForMultiDex'.
> java.util.zip.ZipException: duplicate entry:

You’ll need to track down which libraries are compiling more than once in your project and remove the duplicate modules or groups. Try running

gradlew :your_app_module:dependencies

That will spit out a list of all the dependencies being compiled by your project. Now get to tracking down those duplicates.

Once you can build your app with no conflicting dependencies, it’s time to start adding the files which make Cucumber run the test suite. Your tests project should be in the app/java/androidTest/ directory. Under this directory add an Instrumentation class like so:

package com.yourexample.test
import android.os.Bundle;

import cucumber.api.android.CucumberInstrumentationCore;

public class Instrumentation extends android.support.test.runner.AndroidJUnitRunner {

    private final CucumberInstrumentationCore instrumentationCore = new CucumberInstrumentationCore(this);

    @Override
    public void onCreate(final Bundle bundle) {
        super.onCreate(bundle);
        instrumentationCore.create(bundle);
    }

    @Override
    public void onStart() {
        waitForIdleSync();
        instrumentationCore.start();
    }
}

Instrumentation allows for cucumber to spin up and create the test bundle.

Now, let’s add a runner.

package come.yourpackage.test
import cucumber.api.CucumberOptions;
import cucumber.api.junit.Cucumber;

@CucumberOptions(features = "features",
        glue = {"com.mypackage.test"},
        monochrome = true,
        plugin = { "pretty"} //"html: cucumber-html-reports",
               // "json: cucumber-html-reports/cucumber.json"}
)
public class CucumberRunner {
}

That’s it. The class doesn’t need to be called from anywhere else. Cucumber will look for something like this in order to know how to find features and where to set up reports.

That’s the core of getting Cucumber rolling with your Android project. You should also add the following to your build.gradle file under the android section. The sourceSets reference here points your androidTest environment in the right direction to find both the cucumber features and test code.

sourceSets {
    androidTest {
        assets {
            assets.srcDirs = ['src/androidTest/assets']
        }
        java {
            java.srcDirs = ['src/androidTest/java/com/yourpackage/test']
        }
    }
}

Additionally, add the following to your default config section in build.gradle. This lets androidTest know what your test package is named and which type of Instrumentation runner you want to manage running the tests.

testApplicationId "com.yourpackage.test"
testInstrumentationRunner "com.yourpackage.test.utils.Instrumentation"

Note: here I have my Instrumentation class located in a subfolder of androidTest called utils. Now you should be able to run gradlew -connectedCheck and get the “0 tests” message.

Phase 3: Layers Acquired. It’s testing time.

Create a subdirectory under the assets folder in your project called ‘features’. You can create multiple subdirectories under this and call them whatever you want, but the first folder needs to be features. This is where the Runner we added above will look for feature files.

Add a simple feature file. Save it as FirstTest.feature. Or really whatever you want, as long as it’s there.

Feature: My First Test
Scenario: Make Sure Cucumber Works
Given my app is launched
Then Success

Run gradlew -connectedCheck (or gradlew -cC) and now you should have failing tests. The helpful output from the console should give you some code snippets to add as step definitions. Create a new class under the androidTest directory.

package com.yourpackage.test
import android.app.Activity;
import android.content.Context;
import android.test.ActivityInstrumentationTestCase2;

import com.fanatics.activities.MainActivity;
import com.fanatics.test.libs.Driver;
import com.fanatics.test.utils.ActivityFinisher;

import cucumber.api.java.After;
import cucumber.api.java.Before;
import cucumber.api.java.en.Given;
import cucumber.api.java.en.Then;
import cucumber.api.java.en.When;


public class MainActivitySteps extends ActivityInstrumentationTestCase2<MainActivity> {
    private Activity mActivity;
    private Context mInstrumentationContext;
    private Context mAppContext;

    public MainActivitySteps(){
        super(MainActivity.class);
    }

    @Before
    public void setUp() throws Exception {
        super.setUp();
        mInstrumentationContext = getInstrumentation().getContext();
        mAppContext = getInstrumentation().getTargetContext();
        mActivity = getActivity(); // Start Activity before each test scenario
        assertNotNull(mActivity);
    }

    @After
    public void tearDown() throws Exception {
        ActivityFinisher.finishOpenActivities();
        getActivity().finish();
        super.tearDown(); // This step scrubs everything in this class so always call it last
    }

You will also need an ActivityFinisher class to make the tearDown() step work. This class is necessary for any application with multiple activities as it closes all open activities and allows multiple tests to run in order. If this is not included, a runtime error will occur. Create this class under the androidTest directory.

package com.yourexample.test.utils;

import android.app.Activity;
import android.os.Handler;
import android.os.Looper;
import android.support.test.runner.lifecycle.ActivityLifecycleMonitor;
import android.support.test.runner.lifecycle.ActivityLifecycleMonitorRegistry;
import android.support.test.runner.lifecycle.Stage;

import java.util.ArrayList;
import java.util.EnumSet;
import java.util.List;

/**
 * Finishes Activities at end of individual tests - required for apps with multiple activites
 * in order to avoid runtime errors with resume Intent after @tearDown is called.
 */
public final class ActivityFinisher implements Runnable {

    private final ActivityLifecycleMonitor activityLifecycleMonitor;

    private ActivityFinisher() {
        this.activityLifecycleMonitor = ActivityLifecycleMonitorRegistry.getInstance();
    }

    public static void finishOpenActivities() {
        new Handler(Looper.getMainLooper()).post(new ActivityFinisher());
    }

    @Override
    public void run() {
        final List<Activity> activities = new ArrayList<>();

        for (final Stage stage : EnumSet.range(Stage.CREATED, Stage.STOPPED)) {
            activities.addAll(activityLifecycleMonitor.getActivitiesInStage(stage));
        }

        for (final Activity activity : activities) {
            if (!activity.isFinishing()) {
                activity.finish();
            }
        }
    }
}

Add in the code snippets you got from running the cucumber with no step definitions, add your test logic, and off you go.

Structuring Your Tests

It can be tempting to build all of the logic for a test within the step definition. If you feel this urge, kill it. Murder it in it’s sleep. It’s bad. The layer from cucumber > step definition should not contain test logic. The logic should be abstracted out. You want your step definitions to be as reusable as possible. This involves a few layers of abstraction, but trust me in the end, it makes debugging and maintenance easier.

Here’s an example of how logic should be laid out.

Cucumber Step > Step Definition > Driver Logic > Element locator strategy > Driver step uses element to run > Step definition passes once driver executes > Cucumber reports success.

Here’s an example of the process.

When I tap “HOME”

Calls

@When("^I tap \"([^\"]*)\"$")
public static void I_tap_element(int element){
    Driver.tapElement(element);
}

Which executes

public static void tapElement(String element)
    Field newElement = findElement(element);
    onView(withId((int) newElement.get(new Constants())))
            .perform(click());
}

Which references

public static Field findElement(String element) {
    return Constants.class.getDeclaredField(element);
}

Which then references

package com.youexample.test.libs;

import com.yourexample.R;
public class Constants {

    //Home Screen Elements
    public final static int HOME = R.id.home_button;

Note: the Constants above is a separate class which I added to the same directory as our step definitions file.

Once the findElement step locates the element we’re looking for in our initial cucumber step (“HOME”) it passes it back up the chain of commands to our tap step which then executes a tap on the desired element and gives us a passing cucumber step (as long as the element being tapped is viable and visible on screen).

Why the hell do you do it this way?

In a word: maintainability. Let’s say you build a lovely test suite using hard coded values within your test logic. It works, and it works great. Then, your team decides its time for a refactor. The devs change the name of “HOME” to “We don’t care about your tests”. You had “HOME” hardcoded everywhere it needed to be tapped.

artist credit: http://shafik.deviantart.com/art/Stairway-To-Hell-13876398

Your test suite now looks like this. That’s right. A flaming pit straight to hell. You will need to go into the test and manually change every instance of “HOME” to “We don’t care about your tests”. It takes ages, it makes tests fragile, and it’s just not fun. By using the structure outlined above, you only need to change your reference in one place. One place. One change. That’s it. Now your tests have the updated element reference everywhere.

This type of structure also increases readability and makes debugging much easier. Methods are short and succinct, references to other methods are clear, and the purpose of each function is clear. Instead of gigantic methods which perform complicated operations, a few methods containing a few lines each accomplishes everything you need.

See? Happy tests are readable, maintainable, flexible tests.

For further reading and other neat tricks, as well as a working project (which contributed to my knowledge presented here), check out https://medium.com/@neoranga55/the-evolution-journey-of-android-gui-testing-f65005f7ced8#.tlu6j27t7 and https://github.com/neoranga55/CleanGUITestArchitecture