Running subset of espresso tests locally and gcloud

Test filtering is a very important part of an automation framework; for various reasons you don’t want all your tests to be run. eg: some only work on physical devices, some only work in local environments etc.

One of the shortcomings of espresso is that its JUnit based but not TestNG which has a lot of niceties like test filtering, test data filtering and many more. I feel jealous of the selenium/appium Java folks of all the built in functionality they have including test level parallelism. Also the espresso test runner AndroidJunitRunner only supports Junit4 and JUnit3, so even though Junit5 have some great features espresso test can’t use them yet.

Now, Junit4 has quite a few mechanisms for filtering like @Category and @Suite but they don’t work with AndroidJunitRunner. This framework supports some filtering by using built in annotations LargeTest, MediumTest, FlakyTest etc. After tagging the tests you can run only LargeTest like this

Various triggers for Large

from instrumentation commandline (use -e size large)

adb shell am instrument -w -r -e size large -com.mycompany.app/android.support.test.runner.AndroidJUnitRunner

from gralde espresso kickoff

./gradlew connectedAndroidTest -Pandroid.testInstrumentationRunnerArguments.size=Large

An even better way is to put the argument in the app.gradle file

android {
    ...
    defaultConfig {
  ....         
        "android.support.test.runner.AndroidJUnitRunner"
        testInstrumentationRunnerArgument 'size', 'Large'
    }
}

Keep in mind that if you run the tests from Android Studio by right clicking, you need to add the extra flags option because it doesn’t look at app.gradle file.

Custom Test Attributes

Other than the espresso given attributes, it is also possible to create extra annotations and a colleague at work helped me figure a lot of this

package com.testcategories;

import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;
//Needed to have annotations at a test level
@Target({ElementType.METHOD, ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
public @interface TestEnvOnly {
}

Your testcase would look something like

@TestEnvOnly
@RunWith(AndroidJUnit4.class)
public class MyTestsClass {

    @Rule
    public ActivytTestRule<Activity> activityTestRule
            = new ActivytTestRule<>(Activity.class);
    @TestEnvOnly //either class or method level
    @Test
    public void testASpecifcbehaviour() {
  ... 
    }
}

Now you can use similar triggers

For invoking TestEnvOnly tests

from instrumentation commandline (use -e size large)

adb shell am instrument -w -r -e annotation com.testcategories.TestEnvOnlyTest -com.mycompany.app/android.support.test.runner.AndroidJUnitRunner

from gradle espresso kickoff

./gradlew connectedAndroidTest Pandroid.testInstrumentationRunnerArguments.annotation=com.testcategories.TestEnvOnly

build.gradle…

android {
    ...
    defaultConfig {
....         
"android.support.test.runner.AndroidJUnitRunner"
        testInstrumentationRunnerArgument 'annotation', 'com.godaddy.gdm.telephony.uitests.testcategories.TestEnvOnly'
    }
}

Android Studio

One extra wrinkle if you are using BuildVariants and want to be able to exclude tests only for some variants is that you have to use defaultConfig.

android {
productFlavors {

    all {..}
TestEnv {
defaultConfig.testInstrumentationRunnerArgument 'annotation', 'com.testcategories.TestEnvOnly'
}
DevEnv {
defaultConfig.testInstrumentationRunnerArgument 'notAnnotation', 'com.testcategories.TestEnvOnly'
}

If you are using firebase test lab for cloud based testing you can exclude these tests by using the environmental variables flag

something like this

gcloud beta test android run \
— type instrumentation \
— app app/build/outputs/apk/*unaligned.apk \
— test app/build/outputs/apk/*androidTest-unaligned.apk \
— device-ids Nexus6 \
— os-version-ids 24 \
— environment-variables notAnnotation=com.testcategories.TestEnvOnly