Running Flutter tests in Bamboo on AWS

Not fluttering and not on bamboo, but happy tests are running

Here in Pittsburgh, we are building mobile apps using the Flutter framework. Like all good engineering teams, we write unit tests, and want to run them in our CI system. We happen to use Bamboo and here’s what we did to get our Flutter unit tests to run on each git commit.

TL;DR

The key lessons learned from this experience are:

• Needed to install mesa-libGLU.x86_64
• Needed to workaround libstdc++.so.6 being missing
• The --machine flag causes flutter test to output result in JSON
• The junitreport package is used to convert JSON output to JUnit .xml files (via the tojunit command), a format Bamboo understands

Installing Flutter on Amazon Linux

Our Bamboo installation runs on an Amazon Linux AMI (which is for all intents and purposes, a variant of CentOS). We have a very basic installation right now, running only a single agent on one box, so keep that in mind as you view what we did below. If you are running multiple agents, you’ll need to make sure each agent has Flutter installed as well as the junitreport tool.

It’s also worth noting that we have an EFS (elastic file system) mounted at /custom. We download files into /custom/downloads and install software into /custom/opt. You, obviously, can pick your own download and install locations. Finally, we run our Bamboo instance using a bamboo user.

The installation steps we used to install flutter are outlined below.

Prerequisite

As we were getting this all to work, we ran into a problem running out unit tests where we got an error saying that ligGLU.so.1 was not found. With 20/20 hindsight, we recommend installing this library up-front. Amazon Linux, being CentOS-like, usesyum:

sudo yum install mesa-libGLU.x86_64

Install Flutter

Next we download the version of flutter we are currently using (0.6.0 when we went through this exercise — feel free to use the latest and greatest).

# Login to bamboo box as ec2-user
cd /custom/downloads
mkdir flutter
cd flutter
wget https://storage.googleapis.com/flutter_infra/releases/beta/linux/flutter_linux_v0.6.0-beta.tar.xz
cd /custom/opt 
sudo mkdir flutter
sudo chown bamboo:bamboo flutter

After downloading the install tar and creating a place to install it, we do everything else as the bamboo user.

First, we un-tar the file, which has a top-level directory called flutter.

sudo su - bamboo
cd /custom/opt/flutter
tar xvf /custom/downloads/flutter/flutter_linux_v0.6.0-beta.tar.xz

Then, anticipating having multiple versions of flutter in the future, we rename the extracted flutter directory to v0.6.0 and create a latest symbolic link to this version.

mv flutter v0.6.0 
ln -s v0.6.0 latest

We ran into an issue about libstdc++.so.6 being missing, so followed this recommendation to fix the problem:

cd latest/bin/cache/artifacts/engine/
cp android-arm64-profile/linux-x64/gen_snapshot android-arm-profile/linux-x64/gen_snapshot
cp android-arm64-release/linux-x64/gen_snapshot android-arm-release/linux-x64/gen_snapshot

Then we set some paths so we could run Flutter and Dart commands (you may want to add this to the ~/.bashrc):

export FLUTTER_HOME=/custom/opt/flutter/latest
export PATH=${FLUTTER_HOME}/bin:${PATH}
export PATH=${FLUTTER_HOME}/bin/cache/dart-sdk/bin:${PATH}

Finally, we install the junitreport tool (which translates Dart JSON test output to JUnit format) and add the install location to the PATH so we can verify it installed.

pub global activate junitreport
export PATH=~/.pub-cache/bin:${PATH}
tojunit --help 

If tojunit prints out a usage message, you are good to go. If not, then retrace your steps and engage those debugging skills.

It’s important to note that pub global activate installs executables in the current user’s home directory. The reason we do all this as the bamboo user is so this executable can easily be found when running a Bamboo task (obviously there are other ways to solve for this, but we took the easy path).

Configuring Bamboo

We will assume you know a bit about configuring Bamboo and how to setup plans. For our Flutter repository, we setup a simple plan with 3 tasks.

Task List

Task 1

Task 1 is the default repository source code checkout from git.

Task 2

Task 2 is where the magic happens. We chose an inline shell script. Note that this script assumes it is running from the root of the Flutter project (e.g., the directory that contains the pubspec.yaml, lib and test directory).

Here is the script:

#!/usr/bin/env bash
# exit on any error
set -e
 
# setup paths
export FLUTTER_HOME=/custom/opt/flutter/latest
export PATH=${FLUTTER_HOME}/bin:${PATH}
export PATH=${FLUTTER_HOME}/bin/cache/dart-sdk/bin:${PATH}
export PATH=~/.pub-cache/bin:${PATH}
# Make sure output dir is there
RESULTS=build/test-results/flutter-tests.json
mkdir -p build/test-results

# Get all packages
flutter packages get

# Run unit tests
flutter test --no-pub --machine test/unit | tee ${RESULTS}

# Parse JSON to JUnit XML
echo "Parsing test results from ${RESULTS}"
cat ${RESULTS} | tojunit -o build/test-results/flutter-tests.xml

A couple of notes on the above script:

• Results are output into the build/test-results directory
• The --machine flag is a hidden option of flutter test that causes test results to be output in JSON format.
• The JSON output is what tojunit knows how to parse — it turn it into standard JUnit .xml output
• We separate our unit tests and integration tests into unit and integration directories respectively under the test directory. This is why you see test/unit in the flutter test command. If you do not have a similar separation, you can eliminate the test/unit parameter (which will run all tests under test.

Task 3

The 3rd task, set as a “final” task (so it runs even if things fail), is a JUnit Parser task which looks for test files using this pattern: **/test-results/*.xml.

What this does is allow Bamboo to interpret the Flutter test results as it does Java JUnit results. It will track what fails/succeeds on a build-by-build basis.

End Result

Our Bamboo plan is triggered with each checkin to our repository. It successfully runs our Flutter unit tests and emails us when the build breaks (which I can assure you happens infrequently :-).

While we used Bamboo for our builds, this should be easily adaptable to other CI systems like Jenkins.

A Successful Flutter Bamboo Build