Exploring AndroidJUnitRunner filtering options
Intro
Have you ever wondered what happens when you right-click on an Android test in Android Studio and select Run ‘MainActivityTest’
? How does Android Studio know that you want to run a single test and not the entire test suite? Also, how to achieve something similar from the command line?
In this article I would like to uncover some of the magic behind AndroidJUnitRunner, especially how it can run only a subset of all declared tests and how launching tests in Android Studio works. I’ll shed some light on some of runner arguments as well.
What is this ‘runner’ class?
AndroidJUnitRunner from the Android Support test library is an Instrumentation that runs JUnit tests against an Android package (application). When building with Gradle we declare it in build.gradle like this:
android {
defaultConfig {
testInstrumentationRunner 'android.support.test.runner.AndroidJUnitRunner'
}
}
You can pass additional execution options to this runner to influence what’s being run.
What are execution options?
Execution options are nothing more than a bunch of arguments that we can pass to the runner, which it can later use to perform custom actions. In this article I’ll cover the ones that can be used for test filtering so that we run only a subset of the existing tests.
These arguments are actually a built-in mechanism of Instrumentation and are a set of key-value pairs placed in a Bundle which the Instrumentation receives in its onCreate
method:
public void onCreate(Bundle arguments) {
}
How do I pass these options to the runner?
adb shell am instrument
This is the command that’s actually used under the hood when running with Gradle or within the Android Studio. The command itself is explained in more detail here. To pass additional arguments to the runner we need to use the -e
flag with the test option key and value like so:
-e <key> <value>
E.g.
$ adb shell am instrument -w -e package com.android.test.package1 \
> com.android.test/android.support.test.runner.AndroidJUnitRunner
This would pass a package
argument with com.android.test.package1
as its value to the runner.
With Gradle
Android Gradle Plugin offers a way of passing test options via a testInstrumentationRunnerArguments
property in defaultConfig
e.g.
android {
defaultConfig {
testInstrumentationRunnerArguments ['size': 'medium', 'foo': 'bar']
}
}
Test runner arguments can also be specified from the command line:
$ ./gradlew connectedAndroidTest \
> -Pandroid.testInstrumentationRunnerArguments.size=medium$ ./gradlew connectedAndroidTest \
> -Pandroid.testInstrumentationRunnerArguments.foo=bar
With Android Studio
When right-clicking on a test class located in androidTest
folder in your project and selecting Run 'MyActivityTest'
we start an instrumentation test.
When doing so Android Studio does the following under the hood:
- Builds the application to test and the testing application.
- Installs both APKs on the device.
- Executes
adb shell am instrument
with aclass
test option.
We can see that in the Run
tool window’s logs in Android Studio:
Testing started at 09:04 ...02/13 09:04:23: Launching MyActivityTest...
$ adb push /path/to/myproject/app/build/outputs/apk/debug/app-debug.apk /data/local/tmp/my.package.id.debug
$ adb shell pm install -t -r "/data/local/tmp/my.package.id.debug"
Success$ adb push /path/to/myproject/app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk /data/local/tmp/my.package.id.debug.test
$ adb shell pm install -t -r "/data/local/tmp/my.package.id.debug.test"
SuccessRunning tests$ adb shell am instrument -w -r -e debug false -e class 'my.package.id.screens.MyActivityTest' my.package.id.debug.test/android.support.test.runner.AndroidJUnitRunner
class
test option is also used when right-clicking on a test method in our test class. We can also right-click on a test package and run all tests in there. package
test option would passed then to the runner.
Note: there was a way to pass any test options to the instrumentation before Android Studio 3.0 came along. Sadly this feature has been removed — see the issue in Google issue tracker.
OK, so what options exactly are there?
AndroidJUnitRunner supports a number of arguments by default.
class
Used to run all tests in a class/classes and/or test methods.
Running all tests in a class:
adb shell am instrument -w -e class com.android.foo.FooTest com.android.foo/android.support.test.runner.AndroidJUnitRunner
Running a single test (test method):
adb shell am instrument -w -e class com.android.foo.FooTest#testFoo com.android.foo/android.support.test.runner.AndroidJUnitRunner
Running all tests in multiple classes:
adb shell am instrument -w -e class com.android.foo.FooTest,com.android.foo.TooTest com.android.foo/android.support.test.runner.AndroidJUnitRunner
notClass
Used to run all tests except those in a class/classes and/or test methods.
Running all tests except those in a particular class:
adb shell am instrument -w -e notClass com.android.foo.FooTest com.android.foo/android.support.test.runner.AndroidJUnitRunner
Running all but a single test:
adb shell am instrument -w -e notClass com.android.foo.FooTest#testFoo com.android.foo/android.support.test.runner.AndroidJUnitRunner
testFile & notTestFile
The file should contain a list of line separated package names or test classes and optionally methods.
Running all tests listed in a file:
adb shell am instrument -w -e testFile /sdcard/tmp/testFile.txt com.android.foo/com.android.test.runner.AndroidJUnitRunner
Running all tests not listed in a file:
adb shell am instrument -w -e notTestFile /sdcard/tmp/notTestFile.txt com.android.foo/com.android.test.runner.AndroidJUnitRunner
package & notPackage
Running all tests in a java package:
adb shell am instrument -w -e package com.android.foo.bar com.android.foo/android.support.test.runner.AndroidJUnitRunner
Running all tests except a particular package:
adb shell am instrument -w -e notPackage com.android.foo.bar com.android.foo/android.support.test.runner.AndroidJUnitRunner
size
Running a specific test size i.e. annotated with SmallTest
or MediumTest
or LargeTest
:
adb shell am instrument -w -e size [small|medium|large] com.android.foo/android.support.test.runner.AndroidJUnitRunner
annotation & notAnnotation
If used with other options, the resulting test run will contain the intersection of the two options. E.g. -e size large -e annotation com.android.foo.MyAnnotation
will run only tests with both the LargeTest
and com.android.foo.MyAnnotation
annotations.
Filter test run to tests with given annotation:
adb shell am instrument -w -e annotation com.android.foo.MyAnnotation com.android.foo/android.support.test.runner.AndroidJUnitRunner
Filter test run to tests without given annotation:
adb shell am instrument -w -e notAnnotation com.android.foo.MyAnnotation com.android.foo/android.support.test.runner.AndroidJUnitRunner
Filter test run to tests without any of a list of annotations:
adb shell am instrument -w -e notAnnotation com.android.foo.MyAnnotation,com.android.foo.AnotherAnnotation com.android.foo/android.support.test.runner.AndroidJUnitRunner
filter
Used when the above are not enough for you use case. With filter
you can specify your own custom logic for which tests to run.
Filter test run to tests that pass all of a list of custom filter(s)
:
adb shell am instrument -w -e filter com.android.foo.MyCustomFilter,com.android.foo.AnotherCustomFilter com.android.foo/android.support.test.runner.AndroidJUnitRunner
For details see How to create a custom Filter? section.
Can I use all of them together?
Nope.
We cannot use package and class filtering at the same time. This means that we cannot provide class
+ package
or class
+ notPackage
arguments together. This also means that in testFile
we cannot have classes & packages together.
If we do, the following error gets thrown by the instrumentation:
java.lang.IllegalArgumentException: Ambiguous arguments: cannot provide both test package and test class(es) to run
How to read runner args ?
You might want to pass some additional data to your runner or obtain the arguments supported by default and do something more based on them. As explained earlier, all runner args are passed via a Bundle so to that you need to extend AndroidJUnitRunner and simply access them like this:
public class MyAndroidJUnitRunner extends AndroidJUnitRunner {
@Override
public void onCreate(final Bundle bundle) {
super.onCreate(bundle);
String myRunnerArgumentValue = bundle.getString("myRunnerArgumentKey");
if (myRunnerArgumentValue != null) {
//do some custom stuff here
}
}}
How to create a custom Filter?
Filter is an abstract JUnit class. When provided to the filter
test option it must be public and must provide a public constructor of one of the following patterns. They are searched in order and the first one found is the one that is used.
<init>()
- a no arguments constructor. This is for filters whose behavior is hard coded.<init>(Bundle bundle)
- accepts aBundle
that contains the options passed to this instance. This is for filters whose behavior needs to be configured through additional options toam instrument
.
To write a custom filter we need to extend it and implement two methods:
/**
* @param description the description of the test to be run
* @return <code>true</code> if the test should be run
*/
public abstract boolean shouldRun(Description description);/**
* Returns a textual description of this Filter
*
* @return a textual description of this Filter
*/
public abstract String describe();
The magic lies in the shouldRun
method. This is where we apply our custom filtering logic. Description class contains all the information we need about our test or test suite. shouldRun
will be called for each test and test suite found.
When passing a class
or other supported test option AndroidJUnitRunner actually creates a Filter class under the hood. See android.support.test.internal.runner.TestRequestBuilder
class from com.android.support.test:runner:1.0.1
for examples of these filters.
Custom filter example
In the next article I’ll describe a custom filter we’ve created for our multiple-flavored project in which we can filter which tests to run depending on Gradle flavors. So stay tuned for more :)
Read more about the technologies we use or take an inside look at our organisation & processes. Interested in working at StepStone? Check out our careers page.