Generating code coverage from espresso tests running in Firebase test lab

First, espresso runs tests through the UI but it is a not black box but a grey box, so it knows the code it is going through and hence espresso tests can be counted towards code coverage.

Second, FTL(firebase test lab) has the ability to generate code coverage numbers. Official documentation at: https://cloud.google.com/sdk/gcloud/reference/firebase/test/android/run

So the challenge is can we do this for espresso tests running on FTL cloud.

First we need the correct FTL run command

gcloud firebase test android run \
--type instrumentation \
--app app/build/outputs/apk/*-debug.apk \
--test app/build/outputs/apk/*-androidTest.apk \
--directories-to-pull /sdcard \
--device model=NexusLowRes,version=25 \
--environment-variables coverage=true,coverageFile=“/sdcard/coverage.ec

Pay extra attention to the path of the coverageFile, I haven’t had success with this in any other format. To get more information first click on the FTL link generated by google which will direct you to UI like this after correct login

Click on the device and you will see see source files button

This will take you to your bucket and all the results and output generated by your

Here when I open instrumentation.results I see Failed to generate emma coverage

Opening logcat will shed more light and give you the correct error. Search for the string coverage and will see something like this

): Failed to generate emma coverage. 02–07 17:31:22.248: I/TestRunner(6521): java.lang.reflect.InvocationTargetException 02–07 17:31:22.248: I/TestRunner(6521): at java.lang.reflect.Method.invoke(Native Method) 02–07 17:31:22.248: I/TestRunner(6521): at android.support.test.internal.runner.listener.CoverageListener.generateCoverageReport(CoverageListener.java:98) 02–07 17:31:22.248: I/TestRunner(6521): at android.support.test.internal.runner.listener.CoverageListener.instrumentationRunFinished(CoverageListener.java:70) 02–07 17:31:22.248: I/TestRunner(6521): at android.support.test.internal.runner.TestExecutor.reportRunEnded(TestExecutor.java:92) 02–07 17:31:22.248: I/TestRunner(6521): at android.support.test.internal.runner.TestExecutor.execute(TestExecutor.java:67) 02–07 17:31:22.248: I/TestRunner(6521): at android.support.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:375) 02–07 17:31:22.248: I/TestRunner(6521): at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1932) 02–07 17:31:22.248: I/TestRunner(6521): Caused by: java.io.FileNotFoundException: “/sdcard/tempDir/coverage.ec” (No such file or directory) 02–07 17:31:22.248: I/TestRunner(6521): at java.io.FileOutputStream.open(Native Method) 02–07 17:31:22.248: I/TestRunner(6521): at java.io.FileOutputStream.<init>(FileOutputStream.java:221) 02–07 17:31:22.248: I/TestRunner(6521): at com.vladium.emma.rt.RT.dumpCoverageData(RT.java:50) 02–07 17:31:22.248: I/TestRunner(6521): … 7 more

Google’s documentation misleads into adding tempDir like “/sdcard/tempDir/coverage.ec” but that will generate FileNotFoundException. If you try to do it directly like “coverage.ec” you will get a permission error.

Now, you need the correct directories-to-pull path.

If everything goes right you would see coverage.ec file underneath artifacts/ directory

I’ve stumbled on this couple of times so writing this as a post so it will be helpful for others and future me :)

Also not having SD card permission might cause this problem! https://stackoverflow.com/questions/40726208/how-to-get-code-coverage-reports-from-google-firebase-for-android-espresso-tests