Exploring AndroidJUnitRunner filtering options

Photo by andrew welch on Unsplash

Intro

Have you ever wondered what happens when you right-click on an Android test in Android Studio and select Run ‘MainActivityTest’? How does Android Studio know that you want to run a single test and not the entire test suite? Also, how to achieve something similar from the command line?

In this article I would like to uncover some of the magic behind AndroidJUnitRunner, especially how it can run only a subset of all declared tests and how launching tests in Android Studio works. I’ll shed some light on some of runner arguments as well.

What is this ‘runner’ class?

AndroidJUnitRunner from the Android Support test library is an Instrumentation that runs JUnit tests against an Android package (application). When building with Gradle we declare it in build.gradle like this:

android {
  defaultConfig {
    testInstrumentationRunner 'android.support.test.runner.AndroidJUnitRunner'
  }
}

You can pass additional execution options to this runner to influence what’s being run.

What are execution options?

Execution options are nothing more than a bunch of arguments that we can pass to the runner, which it can later use to perform custom actions. In this article I’ll cover the ones that can be used for test filtering so that we run only a subset of the existing tests.

These arguments are actually a built-in mechanism of Instrumentation and are a set of key-value pairs placed in a Bundle which the Instrumentation receives in its onCreate method:

public void onCreate(Bundle arguments) {
}

How do I pass these options to the runner?

adb shell am instrument

This is the command that’s actually used under the hood when running with Gradle or within the Android Studio. The command itself is explained in more detail here. To pass additional arguments to the runner we need to use the -e flag with the test option key and value like so:

-e <key> <value>

E.g.

$ adb shell am instrument -w -e package com.android.test.package1 \
> com.android.test/android.support.test.runner.AndroidJUnitRunner

This would pass a package argument with com.android.test.package1 as its value to the runner.

With Gradle

Android Gradle Plugin offers a way of passing test options via a testInstrumentationRunnerArguments property in defaultConfig e.g.

android {
  defaultConfig {
    testInstrumentationRunnerArguments ['size': 'medium', 'foo': 'bar']
  }
}

Test runner arguments can also be specified from the command line:

$ ./gradlew connectedAndroidTest \
> -Pandroid.testInstrumentationRunnerArguments.size=medium
$ ./gradlew connectedAndroidTest \
> -Pandroid.testInstrumentationRunnerArguments.foo=bar

With Android Studio

When right-clicking on a test class located in androidTest folder in your project and selecting Run 'MyActivityTest' we start an instrumentation test.

When doing so Android Studio does the following under the hood:

1. Builds the application to test and the testing application.
2. Installs both APKs on the device.
3. Executes adb shell am instrument with a class test option.

We can see that in the Run tool window’s logs in Android Studio:

Testing started at 09:04 ...
02/13 09:04:23: Launching MyActivityTest...
$ adb push /path/to/myproject/app/build/outputs/apk/debug/app-debug.apk /data/local/tmp/my.package.id.debug
$ adb shell pm install -t -r "/data/local/tmp/my.package.id.debug"
Success
$ adb push /path/to/myproject/app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk /data/local/tmp/my.package.id.debug.test
$ adb shell pm install -t -r "/data/local/tmp/my.package.id.debug.test"
Success
Running tests
$ adb shell am instrument -w -r   -e debug false -e class 'my.package.id.screens.MyActivityTest' my.package.id.debug.test/android.support.test.runner.AndroidJUnitRunner

class test option is also used when right-clicking on a test method in our test class. We can also right-click on a test package and run all tests in there. package test option would passed then to the runner.

Note: there was a way to pass any test options to the instrumentation before Android Studio 3.0 came along. Sadly this feature has been removed — see the issue in Google issue tracker.

OK, so what options exactly are there?

AndroidJUnitRunner supports a number of arguments by default.

class

Used to run all tests in a class/classes and/or test methods.

Running all tests in a class:

adb shell am instrument -w -e class com.android.foo.FooTest com.android.foo/android.support.test.runner.AndroidJUnitRunner

Running a single test (test method):

adb shell am instrument -w -e class com.android.foo.FooTest#testFoo com.android.foo/android.support.test.runner.AndroidJUnitRunner

Running all tests in multiple classes:

adb shell am instrument -w -e class com.android.foo.FooTest,com.android.foo.TooTest com.android.foo/android.support.test.runner.AndroidJUnitRunner

notClass

Used to run all tests except those in a class/classes and/or test methods.

Running all tests except those in a particular class:

adb shell am instrument -w -e notClass com.android.foo.FooTest com.android.foo/android.support.test.runner.AndroidJUnitRunner

Running all but a single test:

adb shell am instrument -w -e notClass com.android.foo.FooTest#testFoo com.android.foo/android.support.test.runner.AndroidJUnitRunner

testFile & notTestFile

The file should contain a list of line separated package names or test classes and optionally methods.

Running all tests listed in a file:

adb shell am instrument -w -e testFile /sdcard/tmp/testFile.txt com.android.foo/com.android.test.runner.AndroidJUnitRunner

Running all tests not listed in a file:

adb shell am instrument -w -e notTestFile /sdcard/tmp/notTestFile.txt com.android.foo/com.android.test.runner.AndroidJUnitRunner

package & notPackage

Running all tests in a java package:

adb shell am instrument -w -e package com.android.foo.bar com.android.foo/android.support.test.runner.AndroidJUnitRunner

Running all tests except a particular package:

adb shell am instrument -w -e notPackage com.android.foo.bar com.android.foo/android.support.test.runner.AndroidJUnitRunner

size

Running a specific test size i.e. annotated with SmallTest or MediumTest or LargeTest:

adb shell am instrument -w -e size [small|medium|large] com.android.foo/android.support.test.runner.AndroidJUnitRunner

annotation & notAnnotation

If used with other options, the resulting test run will contain the intersection of the two options. E.g. -e size large -e annotation com.android.foo.MyAnnotation will run only tests with both the LargeTest and com.android.foo.MyAnnotation annotations.

Filter test run to tests with given annotation:

adb shell am instrument -w -e annotation com.android.foo.MyAnnotation com.android.foo/android.support.test.runner.AndroidJUnitRunner

Filter test run to tests without given annotation:

adb shell am instrument -w -e notAnnotation com.android.foo.MyAnnotation com.android.foo/android.support.test.runner.AndroidJUnitRunner

Filter test run to tests without any of a list of annotations:

adb shell am instrument -w -e notAnnotation com.android.foo.MyAnnotation,com.android.foo.AnotherAnnotation com.android.foo/android.support.test.runner.AndroidJUnitRunner

filter

Used when the above are not enough for you use case. With filter you can specify your own custom logic for which tests to run.

Filter test run to tests that pass all of a list of custom filter(s):

adb shell am instrument -w -e filter com.android.foo.MyCustomFilter,com.android.foo.AnotherCustomFilter com.android.foo/android.support.test.runner.AndroidJUnitRunner

For details see How to create a custom Filter? section.

Can I use all of them together?

Nope.

We cannot use package and class filtering at the same time. This means that we cannot provide class + package or class + notPackage arguments together. This also means that in testFile we cannot have classes & packages together.

If we do, the following error gets thrown by the instrumentation:

java.lang.IllegalArgumentException: Ambiguous arguments: cannot provide both test package and test class(es) to run

How to read runner args ?

You might want to pass some additional data to your runner or obtain the arguments supported by default and do something more based on them. As explained earlier, all runner args are passed via a Bundle so to that you need to extend AndroidJUnitRunner and simply access them like this:

public class MyAndroidJUnitRunner extends AndroidJUnitRunner {
  
  @Override
  public void onCreate(final Bundle bundle) {
    super.onCreate(bundle);
    String myRunnerArgumentValue =  bundle.getString("myRunnerArgumentKey");
    if (myRunnerArgumentValue != null) {
      //do some custom stuff here
    }
  }
}

How to create a custom Filter?

Filter is an abstract JUnit class. When provided to the filter test option it must be public and must provide a public constructor of one of the following patterns. They are searched in order and the first one found is the one that is used.

1. <init>() - a no arguments constructor. This is for filters whose behavior is hard coded.
2. <init>(Bundle bundle) - accepts a Bundle that contains the options passed to this instance. This is for filters whose behavior needs to be configured through additional options to am instrument.

To write a custom filter we need to extend it and implement two methods:

/**
 * @param description the description of the test to be run
 * @return <code>true</code> if the test should be run
 */
public abstract boolean shouldRun(Description description);
/**
 * Returns a textual description of this Filter
 *
 * @return a textual description of this Filter
 */
public abstract String describe();

The magic lies in the shouldRun method. This is where we apply our custom filtering logic. Description class contains all the information we need about our test or test suite. shouldRun will be called for each test and test suite found.

When passing a class or other supported test option AndroidJUnitRunner actually creates a Filter class under the hood. See android.support.test.internal.runner.TestRequestBuilder class from com.android.support.test:runner:1.0.1 for examples of these filters.

Custom filter example

In the next article I’ll describe a custom filter we’ve created for our multiple-flavored project in which we can filter which tests to run depending on Gradle flavors. So stay tuned for more :)

Read more about the technologies we use or take an inside look at our organisation & processes. Interested in working at StepStone? Check out our careers page.