UI tests in Flutter with Azure Pipelines

Continuous integration (CI) is the practice of merging all developers’ working copies to shared mainline several times a day.

One of the most important aspects of CI is keeping the mainline healthy. That’s why it’s crucial to have proper tests, both unit and UI, that can run against any PR and/or commit to the mainline. At the same time, these tests should also be fast. It’s usually not a problem with unit tests, but UI tests, especially if you want to run them on multiple devices, can take a rather long time.

As a balance between speed and reliability, for PRs into the mainline, we use smoke UI tests that only prove the main functionality is working. Proper UI testing with manual QA verification is done only when we’re preparing a new release candidate.

In this article, I will show you how we’ve set up Azure Pipelines for running automated UI smoke tests for a Flutter app.

Unfortunately, Azure Pipelines don’t provide agents with Flutter pre-installed, and, as we use Microsoft-hosted agents, we need to perform a bunch of actions on each job run (you can avoid these steps by setting up and using self-hosted agents). These actions are simpler to invoke as bash scripts, so almost every step in pipeline configuration just delegates the work to some script.

We keep these scripts in a separate file: pipelines/scripts.sh.

We use theset -e option, so that each failed command will fail the job, then add the Flutter installation directory to $PATH.

"$@" allows us to use functions defined in this file in other scripts — if we have a function install_flutter() defined in this file, we can call it later with sh scripts.sh install_flutter.

Install Flutter

First, we need to install Flutter:

Here we download a stable branch of Flutter, install it, accept all licenses, and print doctor output (you can skip this, but it can be useful to check when dealing with certain problems).

Install AVD

Next, install and launch AVD:

We use Android SDK 29 without Google APIs. You can use a different SDK, just replace 'system-images;android-29;default;x86' with the other option. You can get a full list of options by running sdkmanager --list | grep system-images | sort | uniq.

Screen recording

In Google Play automated tests there’s a cool feature: it records the screen while running tests, so you can download and check it later. It’s a good idea to have it in our tests as well. For that, we can use the built-in screenrecord utility. Unfortunately, it has a hard-coded max length of three minutes. So we need a hack to fix this:

(No worries, this doesn’t mean that all five records will be created with each job — we’ll stop recording once tests are done.)

Run tests

Now we’re ready to create a script that will build and run tests:

(If you don’t have any code generation in your project, remove the build_runner action.)

This script runs unit tests with flutter test and UI tests with flutter drive --target=test_driver/app.dart. As you can see here, before running UI tests we launch AVD and start video recording in the background. And after the test is done, we kill the running screenrecord command. Thanks to theset -e option defined in the script, it will also end the start_recording command.

Download record

We also need a helper script for getting the record out of the device:

(If you provide a directory as the pull argument it will download all files in that directory.)

Pipeline configuration

Now let’s create a pipeline configuration file putting everything together:

(If you’re not sure where to put this file, check this tutorial.)

Pay attention to the “Pull screen video record” and “Publish screenshots” steps: the condition parameter is set to always(), as we want them to run even if tests fail.

What about the UI test itself? Let’s create a basic smoke test that allows us to check the app state (you can find a detailed tutorial on the official site).

First, create a test_driver folder and put an app.dart file into it with the following content:

This file contains an “instrumented” version of the app. While it can have any name that makes sense, for the sake of simplicity we just call it app.dart.

Now let’s create a file that contains the test suite which drives the app and verifies that it works as expected. The name of the test file must correspond to the name of the file that contains the instrumented app, with _test added at the end. In our case, it means that name should be app_test.dart:

As you can see, in the setUpAll function we create screenshots directory — we will put screenshots there if needed (and screen recording as well).

All the helper functions, such as loginToCommander or goToRooms are defined in the app_test_resources.dart file, e.g.:

We also have a couple of extensions for the driver:

(We replace “:” with “-” in the file name because Azure Pipelines fails to publish a file to artifacts if it contains “:” in the name.)

saveScreenshot is called automatically when the driver fails to find an element. In that case, we want to be able to look at a screenshot and figure out what the problem is. It can also be called manually if we want to screenshot the app at some checkpoint.

Here’s the downloaded video with screen recording:

That’s it for running our UI tests on an Android emulator. Although, despite the cross-platform nature of Flutter, it could be useful to run these smoke tests on an iOS simulator as well, right? But that’s a topic for another article. Stay tuned ;)

To find out more about Mews and the dev team, check out our Github and follow us on Twitter and Facebook.