Flutter Test Reports : Current state of art

This medium article aims to list what developers can currently do to monitor their Flutter tests results.

This article doesn’t include test coverage, so if your are interested you can read this excellent article from @sudhanshu0203.

For now, CIs such as Jenkins or Bitrise don’t provide Test Reports for Flutter, and SonarDart is not coming anytime soon. So as of now, we don’t have a nice way of displaying the current status of a job or workflow execution with tests results. But we will see that, by using existing tools, we can achieve something “good enough”, while waiting for the technology to come up with better solutions.

Exporting flutter results to junit format

As you surely know, the flutter test command allows you to execute tests of the flutter project. But without any other args you will just be able to check if the tests are failing or passing and go to your next CI step.

As described in the documentation, you can use --coverage to get some code coverage reports following flutter style, but you won’t have granular informations like the number of tests, and which ones are failing or passing.

flutter test --machine

This wonderful argument produces a json output for the tests, and we are going to use this.

junitreport

https://pub.dev/packages/junitreport

This Dart package allows you to convert the json output of the --machine argument to produce a junit test report.

To be able to use it in your CI, you need to make Dart & Dart Installed Packages available from the command line.

Depending of the use of Jenkins or Bitrise or others, the location of flutter pub cache & dart sdk might differ and you will have to check and replace the location by yours:

export PATH="$PATH":"$HOME/.pub-cache/bin"
export PATH="$PATH":"/usr/local/opt/flutter/bin/cache/dart-sdk/bin"

We then need to activate the dart package:

flutter pub get
flutter pub global activate junitreport

All set! Now we can launch the tests with the --machine option and let junitreport convert the test output.

flutter test --machine | tojunit --output <Your output file>.xml

Displaying jUnit Test Reports…

This article covers 2 different CI platforms: Jenkins & Bitrise.

…in Jenkins

In most Jenkins setups, the jUnit plugin is already activated but you might need to do it manually:

https://wiki.jenkins.io/display/JENKINS/JUnit+Plugin

Once the plugin is installed, you can consume the generated xml test report on a simple Jenkins pipeline step:

stage('Tests') {
    steps {
        ... your step generating the xml
    }
    post {
        always {
            junit "<location to your .xml file>"
        }
        success {
            publishHTML target: [
                    allowMissing         : false,
                    alwaysLinkToLastBuild: false,
                    keepAll              : true,
                    reportDir            : "<location of your html file (should be the same as the xml file)>",
                    reportFiles          : 'index.html',
                    reportName           : 'HTML Report'
            ]
        }
    }
}

You will then have the Test Report available on each jenkins job:

An exemple of a full Jenkins pipeline step:

stage('Tests') {
    steps {
        sh ''' #!/bin/bash
            set -e
            export PATH="$PATH":"$HOME/.pub-cache/bin"
            export PATH="$PATH":"/usr/local/opt/flutter/bin/cache/dart-sdk/bin"
            flutter pub get
            flutter pub global activate junitreport
            mkdir tests-results
            flutter test --machine | tojunit --output tests-results/TEST-report.xml                    
            '''
    }
    post {
        always {
            junit "tests-results/*.xml"
        }
        success {
            publishHTML target: [
                    allowMissing         : false,
                    alwaysLinkToLastBuild: false,
                    keepAll              : true,
                    reportDir            : "tests-results/",
                    reportFiles          : 'index.html',
                    reportName           : 'HTML Report'
            ]
        }
    }
}

And that’s all for Jenkins! Don’t hesitate to provide another step for test coverage discussed here.

…With Bitrise

The official Flutter Test Step doesn’t allow us to do much with the output. So let’s try and create a simple script step doing better !

Following Bitrise documentation, you can use the Test Report Addons to display test results if you provide jUnit test reports to the directory $BITRISE_TEST_RESULT_DIR.

test_run_dir="$BITRISE_TEST_RESULT_DIR/<Your test dir>"
mkdir "$test_run_dir"

flutter test --machine | tojunit --output $test_run_dir/<Your output file>.xml

Additionally, we need to add a test-info.json file with some basic Test Run informations (following https://devcenter.bitrise.io/testing/exporting-to-test-reports-from-custom-script-steps/). We will just provide the test run name for now:

echo '{"test-name":"<Your test run name>"}' >> "$test_run_dir/test-info.json"

And that’s everything you need! Once your build is successful, you can access the Test Reports by clicking here:

You can then enjoy your test report powered by Bitrise Test Report Addon.

An example of a complete script step for Bitrise:

#!/usr/bin/env bash
# fail if any commands fails
set -e
# debug log
set -x

export PATH="$PATH":"/usr/local/flutter/.pub-cache/bin"
export PATH="$PATH":"/usr/local/flutter/bin/cache/dart-sdk/bin"

flutter pub get
flutter pub global activate junitreport

test_run_dir="$BITRISE_TEST_RESULT_DIR/<Your test dir>"
mkdir "$test_run_dir"

flutter test --machine | tojunit --output $test_run_dir/<Your output file>.xml

echo '{"test-name":"<Your test run name>"}' >> "$test_run_dir/test-info.json"

If you wish to use an already existing step, you can modify your bitrise.yml to link to my fork of the official step.

A Pull Request is waiting for approval so that we can use the officiel Flutter Test Step instead :)

Add this step to your bitrise.yml:

- git::https://github.com/etiennecadicidean/bitrise-step-flutter-test.git:
       inputs:
       - test_run_name: <Your wished Test Run Name>
       - project_location: "$BITRISE_FLUTTER_PROJECT_LOCATION"

I hope this article will help you launch on Flutter development :)

I thank Gyl Jean-Lambert for his useful inputs on previous links & his works on Jenkins jUnit Plugin integration on our CI.