Running your tests on the command line with Gradle

“Matrix Code” by David.Asch licensed under CC BY-NC-ND 2.0

In addition to running your tests in Android Studio you can also run them from the command line. This is useful for automated build scripts, CI servers, or when Android Studio is crashing on launch and needs to be reinstalled.

Use the command ./gradlew test to run all tests.

$ ./gradlew test
Incremental java compilation is an incubating feature.
:app:preBuild UP-TO-DATE
:app:preDebugBuild UP-TO-DATE
:app:checkDebugManifest
:app:preReleaseBuild UP-TO-DATE
:app:prepareComAndroidSupportAnimatedVectorDrawable2501Library
:app:prepareComAndroidSupportAppcompatV72501Library
:app:prepareComAndroidSupportDesign2501Library
:app:prepareComAndroidSupportRecyclerviewV72501Library
:app:prepareComAndroidSupportSupportCompat2501Library
:app:prepareComAndroidSupportSupportCoreUi2501Library
:app:prepareComAndroidSupportSupportCoreUtils2501Library
:app:prepareComAndroidSupportSupportFragment2501Library
:app:prepareComAndroidSupportSupportMediaCompat2501Library
:app:prepareComAndroidSupportSupportV42501Library
:app:prepareComAndroidSupportSupportVectorDrawable2501Library
:app:prepareComAndroidSupportTransition2501Library
:app:prepareDebugDependencies
:app:compileDebugAidl
:app:compileDebugRenderscript
:app:generateDebugBuildConfig
:app:generateDebugResValues
:app:generateDebugResources
:app:mergeDebugResources
:app:processDebugManifest
:app:processDebugResources
:app:generateDebugSources
:app:incrementalDebugJavaCompilationSafeguard
:app:compileDebugJavaWithJavac
:app:compileDebugJavaWithJavac - is not incremental (e.g. outputs have changed, no previous execution, etc.).
:app:incrementalDebugUnitTestJavaCompilationSafeguard UP-TO-DATE
:app:preDebugUnitTestBuild UP-TO-DATE
:app:prepareDebugUnitTestDependencies
:app:compileDebugUnitTestJavaWithJavac
:app:processDebugJavaRes UP-TO-DATE
:app:processDebugUnitTestJavaRes UP-TO-DATE
:app:compileDebugUnitTestSources
:app:mockableAndroidJar
:app:assembleDebugUnitTest
:app:testDebugUnitTest
:app:checkReleaseManifest
:app:prepareReleaseDependencies
:app:compileReleaseAidl
:app:compileReleaseRenderscript
:app:generateReleaseBuildConfig
:app:generateReleaseResValues
:app:generateReleaseResources
:app:mergeReleaseResources
:app:processReleaseManifest
:app:processReleaseResources
:app:generateReleaseSources
:app:incrementalReleaseJavaCompilationSafeguard
:app:compileReleaseJavaWithJavac
:app:compileReleaseJavaWithJavac - is not incremental (e.g. outputs have changed, no previous execution, etc.).
:app:incrementalReleaseUnitTestJavaCompilationSafeguard UP-TO-DATE
:app:preReleaseUnitTestBuild UP-TO-DATE
:app:prepareReleaseUnitTestDependencies
:app:compileReleaseUnitTestJavaWithJavac
:app:processReleaseJavaRes UP-TO-DATE
:app:processReleaseUnitTestJavaRes UP-TO-DATE
:app:compileReleaseUnitTestSources
:app:assembleReleaseUnitTest
:app:testReleaseUnitTest
:app:test

BUILD SUCCESSFUL

Total time: 6.517 secs

If we dig into the output we can see that both :app:testDebugUnitTest and :app:testReleaseUnitTest were executed.

Unfortunately that is all we see and it is buried in a long list of build steps. A quick glance at the output might leave you wondering if all of your tests ran or even any at all.

If there is a failing test we at least get a bit more output about that specific test case.

$ ./gradlew test      
Incremental java compilation is an incubating feature.
:app:preBuild UP-TO-DATE
:app:preDebugBuild UP-TO-DATE
:app:checkDebugManifest
:app:preReleaseBuild UP-TO-DATE
:app:prepareComAndroidSupportAnimatedVectorDrawable2501Library UP-TO-DATE
:app:prepareComAndroidSupportAppcompatV72501Library UP-TO-DATE
:app:prepareComAndroidSupportDesign2501Library UP-TO-DATE
:app:prepareComAndroidSupportRecyclerviewV72501Library UP-TO-DATE
:app:prepareComAndroidSupportSupportCompat2501Library UP-TO-DATE
:app:prepareComAndroidSupportSupportCoreUi2501Library UP-TO-DATE
:app:prepareComAndroidSupportSupportCoreUtils2501Library UP-TO-DATE
:app:prepareComAndroidSupportSupportFragment2501Library UP-TO-DATE
:app:prepareComAndroidSupportSupportMediaCompat2501Library UP-TO-DATE
:app:prepareComAndroidSupportSupportV42501Library UP-TO-DATE
:app:prepareComAndroidSupportSupportVectorDrawable2501Library UP-TO-DATE
:app:prepareComAndroidSupportTransition2501Library UP-TO-DATE
:app:prepareDebugDependencies
:app:compileDebugAidl UP-TO-DATE
:app:compileDebugRenderscript UP-TO-DATE
:app:generateDebugBuildConfig UP-TO-DATE
:app:generateDebugResValues UP-TO-DATE
:app:generateDebugResources UP-TO-DATE
:app:mergeDebugResources UP-TO-DATE
:app:processDebugManifest UP-TO-DATE
:app:processDebugResources UP-TO-DATE
:app:generateDebugSources UP-TO-DATE
:app:incrementalDebugJavaCompilationSafeguard UP-TO-DATE
:app:compileDebugJavaWithJavac UP-TO-DATE
:app:incrementalDebugUnitTestJavaCompilationSafeguard UP-TO-DATE
:app:preDebugUnitTestBuild UP-TO-DATE
:app:prepareDebugUnitTestDependencies
:app:compileDebugUnitTestJavaWithJavac
:app:processDebugJavaRes UP-TO-DATE
:app:processDebugUnitTestJavaRes UP-TO-DATE
:app:compileDebugUnitTestSources
:app:mockableAndroidJar UP-TO-DATE
:app:assembleDebugUnitTest
:app:testDebugUnitTest

com.ecgreb.mytestapp.ExampleUnitTest > thisTestShouldFail FAILED
    java.lang.AssertionError at ExampleUnitTest.java:23

3 tests completed, 1 failed
:app:testDebugUnitTest FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':app:testDebugUnitTest'.
> There were failing tests. See the report at: file:///usr/local/code/my-test-app/app/build/reports/tests/debug/index.html

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 1.622 secs

Wouldn’t it be nice to see how many tests were run, which passed, which failed, and how many were skipped? With a small change to your build.gradle file you can.

tasks.withType(Test) { 
  testLogging {
    exceptionFormat "full"
    events "started", "skipped", "passed", "failed"
    showStandardStreams true
  }
}

Add this block to your build.gradle then re-run your tests on the command line. Go ahead — I’ll wait.

$ ./gradlew test
Incremental java compilation is an incubating feature.
:app:preBuild UP-TO-DATE
:app:preDebugBuild UP-TO-DATE
:app:checkDebugManifest
:app:preReleaseBuild UP-TO-DATE
:app:prepareComAndroidSupportAnimatedVectorDrawable2501Library
:app:prepareComAndroidSupportAppcompatV72501Library
:app:prepareComAndroidSupportDesign2501Library
:app:prepareComAndroidSupportRecyclerviewV72501Library
:app:prepareComAndroidSupportSupportCompat2501Library
:app:prepareComAndroidSupportSupportCoreUi2501Library
:app:prepareComAndroidSupportSupportCoreUtils2501Library
:app:prepareComAndroidSupportSupportFragment2501Library
:app:prepareComAndroidSupportSupportMediaCompat2501Library
:app:prepareComAndroidSupportSupportV42501Library
:app:prepareComAndroidSupportSupportVectorDrawable2501Library
:app:prepareComAndroidSupportTransition2501Library
:app:prepareDebugDependencies
:app:compileDebugAidl
:app:compileDebugRenderscript
:app:generateDebugBuildConfig
:app:generateDebugResValues
:app:generateDebugResources
:app:mergeDebugResources
:app:processDebugManifest
:app:processDebugResources
:app:generateDebugSources
:app:incrementalDebugJavaCompilationSafeguard
:app:compileDebugJavaWithJavac
:app:compileDebugJavaWithJavac - is not incremental (e.g. outputs have changed, no previous execution, etc.).
:app:incrementalDebugUnitTestJavaCompilationSafeguard UP-TO-DATE
:app:preDebugUnitTestBuild UP-TO-DATE
:app:prepareDebugUnitTestDependencies
:app:compileDebugUnitTestJavaWithJavac
:app:processDebugJavaRes UP-TO-DATE
:app:processDebugUnitTestJavaRes UP-TO-DATE
:app:compileDebugUnitTestSources
:app:mockableAndroidJar
:app:assembleDebugUnitTest
:app:testDebugUnitTest

com.ecgreb.mytestapp.ExampleUnitTest > mainActivity_shouldNotBeNull STARTED

com.ecgreb.mytestapp.ExampleUnitTest > mainActivity_shouldNotBeNull PASSED

com.ecgreb.mytestapp.ExampleUnitTest > addition_isCorrect STARTED

com.ecgreb.mytestapp.ExampleUnitTest > addition_isCorrect PASSED
:app:checkReleaseManifest
:app:prepareReleaseDependencies
:app:compileReleaseAidl
:app:compileReleaseRenderscript
:app:generateReleaseBuildConfig
:app:generateReleaseResValues
:app:generateReleaseResources
:app:mergeReleaseResources
:app:processReleaseManifest
:app:processReleaseResources
:app:generateReleaseSources
:app:incrementalReleaseJavaCompilationSafeguard
:app:compileReleaseJavaWithJavac
:app:compileReleaseJavaWithJavac - is not incremental (e.g. outputs have changed, no previous execution, etc.).
:app:incrementalReleaseUnitTestJavaCompilationSafeguard UP-TO-DATE
:app:preReleaseUnitTestBuild UP-TO-DATE
:app:prepareReleaseUnitTestDependencies
:app:compileReleaseUnitTestJavaWithJavac
:app:processReleaseJavaRes UP-TO-DATE
:app:processReleaseUnitTestJavaRes UP-TO-DATE
:app:compileReleaseUnitTestSources
:app:assembleReleaseUnitTest
:app:testReleaseUnitTest

com.ecgreb.mytestapp.ExampleUnitTest > mainActivity_shouldNotBeNull STARTED

com.ecgreb.mytestapp.ExampleUnitTest > mainActivity_shouldNotBeNull PASSED

com.ecgreb.mytestapp.ExampleUnitTest > addition_isCorrect STARTED

com.ecgreb.mytestapp.ExampleUnitTest > addition_isCorrect PASSED
:app:test

BUILD SUCCESSFUL

Total time: 6.428 secs

There isn’t that better? Now at a glance we can see which tests ran and the result of each. Twice. Once for debug and once for release. Of course you can customize the logging level to taste by adding or removing an event in the task filter.

One last note — if you want to run your tests for only one build type (this can be important for a large test suite) you can use ./gradlew testDebug or ./gradlew testRelease.

$ ./gradlew testDebug
Incremental java compilation is an incubating feature.
:app:preBuild UP-TO-DATE
:app:preDebugBuild UP-TO-DATE
:app:checkDebugManifest
:app:preReleaseBuild UP-TO-DATE
:app:prepareComAndroidSupportAnimatedVectorDrawable2501Library
:app:prepareComAndroidSupportAppcompatV72501Library
:app:prepareComAndroidSupportDesign2501Library
:app:prepareComAndroidSupportRecyclerviewV72501Library
:app:prepareComAndroidSupportSupportCompat2501Library
:app:prepareComAndroidSupportSupportCoreUi2501Library
:app:prepareComAndroidSupportSupportCoreUtils2501Library
:app:prepareComAndroidSupportSupportFragment2501Library
:app:prepareComAndroidSupportSupportMediaCompat2501Library
:app:prepareComAndroidSupportSupportV42501Library
:app:prepareComAndroidSupportSupportVectorDrawable2501Library
:app:prepareComAndroidSupportTransition2501Library
:app:prepareDebugDependencies
:app:compileDebugAidl
:app:compileDebugRenderscript
:app:generateDebugBuildConfig
:app:generateDebugResValues
:app:generateDebugResources
:app:mergeDebugResources
:app:processDebugManifest
:app:processDebugResources
:app:generateDebugSources
:app:incrementalDebugJavaCompilationSafeguard
:app:compileDebugJavaWithJavac
:app:compileDebugJavaWithJavac - is not incremental (e.g. outputs have changed, no previous execution, etc.).
:app:incrementalDebugUnitTestJavaCompilationSafeguard UP-TO-DATE
:app:preDebugUnitTestBuild UP-TO-DATE
:app:prepareDebugUnitTestDependencies
:app:compileDebugUnitTestJavaWithJavac
:app:processDebugJavaRes UP-TO-DATE
:app:processDebugUnitTestJavaRes UP-TO-DATE
:app:compileDebugUnitTestSources
:app:mockableAndroidJar
:app:assembleDebugUnitTest
:app:testDebugUnitTest

com.ecgreb.mytestapp.ExampleUnitTest > mainActivity_shouldNotBeNull STARTED

com.ecgreb.mytestapp.ExampleUnitTest > mainActivity_shouldNotBeNull PASSED

com.ecgreb.mytestapp.ExampleUnitTest > addition_isCorrect STARTED

com.ecgreb.mytestapp.ExampleUnitTest > addition_isCorrect PASSED

BUILD SUCCESSFUL

Total time: 4.017 secs

Happy testing!