Boosting Flutter Integration tests execution speed in pipeline by 3x!

Hello, dear Medium readers! In this short article, I will share a simple yet effective method to significantly increase the speed of executing integration tests in your Flutter pipeline, up to 3 times faster. If you have been facing the challenge of long execution times for Flutter tests, this approach can help optimize your development process.

Background

Our integration tests were running overnight on various platforms, but we encountered a problem: executing around 100 tests on Android took about 1 hour 20–30 minutes, and on iOS, it was even longer, around 1.5–2 times more. Such long execution times were unacceptable, especially considering the limitations on job execution in our pipeline.

The Challenge of Parallelizing Flutter Integration Tests

In an effort to speed up our tests, we considered parallelization. However, we faced some limitations. Flutter does not provide built-in support for the parallel execution of integration tests, which can be challenging when dealing with a rapidly growing test volume in your project. If you try to run multiple flutter test integration_test/test.dart commands from the same repository, the test commands can interact with each other, resulting in only one command being executed while the rest are ignored. This can lead to inconsistent results, incorrect reports, and slow overall test execution.

Solution:

Let’s dive into the solution! First, you should divide your integration tests into 3 logical groups, following the recommendations in this article by my colleague:

https://medium.com/@ZeroCode999/flutter-improving-the-speed-of-integration-testing-f87d2c0670fe

This will already speed up your testing process significantly.

Next, set up emulators. You will need 3 dynamically spawned emulators, as described in detail in this article:

https://medium.com/@ZeroCode999/flutter-dynamically-creating-and-using-android-emulator-for-integration-tests-92db2cd4e307

Now that our setup is ready, let’s tackle the problem of long test execution:

#!/bin/bash

# Command to run tests and save results in reports
flutter test -r json > integration_test/1_report.json integration_test/logical_group1_test.dart  -d "$DEVICE1" & P1=$!
if [ -z "$P1" ]; then
  echo "Failed to start logical_group1_test"
else
  echo "Started logical_group1_test with PID $P1"
fi
sleep 90

flutter test -r json > integration_test/2_report.json integration_test/logical_group2_test.dart -d "$DEVICE2" & P2=$!
if [ -z "$P2" ]; then
  echo "Failed to start logical_group2_test"
else
  echo "Started logical_group2_test with PID $P2"
fi
sleep 90

flutter test -r json > integration_test/3_report.json integration_test/logical_group3_test.dart -d "$DEVICE3" & P3=$!
if [ -z "$P3" ]; then
  echo "Failed to start logical_group3_test"
else
  echo "Started logical_group3_test with PID $P3"
fi

wait $P1 $P2 $P3

Let’s take a closer look at our script.

We sequentially run three test groups using the flutter test command, with each group corresponding to a logical group of tests. After launching each test group, we wait for 90 seconds. This delay is necessary for Flutter to build and install the APK before launching the next test group. You can set your own delay if additional time is required for APK build.

Each test is run in the background using the ampersand &, allowing us to execute them in parallel. We use the variables P1, P2, and P3 to obtain the process IDs of the test groups and wait for their completion using the wait command.

Although this approach may not be a conventional parallel execution, it allows tests to run in parallel, minimizing mutual influence and preventing conflicts, which contributes to faster testing and provides more reliable results.

Thank you for your attention, and I hope these simple and effective methods will help you speed up integration test execution in your Flutter project! Good luck and successful testing!