Building a Smoke Test Suite in Espresso for Native Android Apps

I wrapped up a work week where I spent the better part of it trying to figure out how to cherry-pick a few methods from different classes to build a smoke test suite. “This will be easy,” I thought, remembering using Junit’s handy @Smoke from a previous project. Unfortunately, I discovered as soon as I annotated my first smoke test with @Smoke that feature had been deprecated.

I was so wrong. This was not easy. This was frustrating and surprisingly difficult to figure out, as everyone has a different way to accomplish this goal. I couldn’t find a lot on the topic; it seemed most people were simply running all the tests in a given class or project. I found others who were accomplishing this with Spoon, but I wasn’t keen on adding something more complicated than what was needed. The solution ended up being simple — once I figured it out.

If you’re struggling with the same problem, you now have a handy step-by-step guide on how to get quickly and easily started with chunking your Espresso tests into suites.

Why Build Suites?

Simply put, we don’t want to run all of our UI tests every single time. UI tests take a long time to run when compared to pretty much every other type of automated test. We do, however, want to run some of the most important tests at critical points. I wanted to build a smoke test suite that ran with every merge into the develop branch and tested core features. If these tests break, something critical is broken.

Our smoke tests cover login, logout, creating a new account, and page loads for our 5 main content areas (otherwise known as Activities in Android-speak). The tests take less than 2 minutes to run. Our full suite takes somewhere around 7 minutes. One can easily see how running a subset of our full test suite can save hours of time in a week, which can lead to saving thousands per year in engineer hours and infrastructure costs.

AndroidJUnitRunner != JUnit

I spent a few hours researching and setting up JUnit categories, only to find out that categories won’t work with Android’s SDK. Womp womp! I discovered Android has its own test runner for JUnit. It does not support all of the various features that JUnit supports, including categories. (It also includes some annotations like @SmallTest and @LargeTest, which may be valuable for you. It wasn’t for me.)

Once I found the official documentation for AndroidJUnitRunner, things became clear very quickly. Funny how that works, isn’t it?

After doing some chatting in the Ministry of Testing automation Slack channel, Sven (@craftingtester) suggested a solution that I liked: create my own annotations and pass an argument to Gradle telling it to only run tests with a given annotation. It worked like a charm. Thanks Sven!

Setting Up & Using Annotations

I’ll assume that you have Android Studio installed. You’ll also need to get Java 8 installed on your system. I’ll be using Kotlin too, but you can stick with Java if you’d like; just basically replace the word “Kotlin” with “Java” for the rest of this post.

Open up your project. Create a util folder under src/androidTest/kotlin/com.your.project to keep things nice and tidy, then add a new Kotlin file to your util folder.

Type in a name like SmokeTest, select Interface as the kind, and click OK. It’ll open up the new interface. Add a few imports and a couple of annotations, and you’ve got yourself a custom annotation that you can use to define your test suites.

Boom! That’s it. You’ve created your first custom annotation. Now you can use it by adding the annotation to your smoke tests. Simply find the test method and annotate it like I did below.

Now you’re going to run the tests from the command line. Make sure that you have an emulator running or a device connected, or else this will error out. Open a terminal window, navigate to your project’s directory, and run this Gradle command:

./gradlew connectedDevelopmentDebugAndroidTest -Pandroid.testInstrumentationRunnerArguments.annotation=com.your.project.util.SmokeTest

If you’re on Windows, it’ll be just gradlew. In this command, you’re using Gradle Wrapper to run the task connectedDevelopmentDebugAndroidTest with the annotation you created, SmokeTest. The task name is based on your project’s build variants, so yours will probably be different. You can find your available tasks in the Gradle projects pane in Android studio, as illustrated below.

Once you run that command with the task specific to your project, the tests that you annotated with @SmokeTest will run on the connected device. When the tests finish, you get either a success or failure message and a link to a nice Gradle-generated HTML report that you can share with your team.

If this helped you, if I left something out, or you want to know more about a topic related to software testing, let me know in the comments. Happy testing!