Flutter Integration Testing and CI/CD

Hello, Medium. Recently, I was a speaker at the Heisenbug. This experience enhanced my motivation as a professional. As a culmination of my series of articles and presentation, today I will compile information on implementing the integration testing process in Flutter.

Practical examples and the theoretical foundation from this material are designed to help you understand the overall concept and approach to implementing your own testing process. This is due to the fact that implementations vary across applications, and different packages are used in each project.

Today, we will implement:

• Stable and fast integration tests
• Dynamic creation and usage of Android emulators and iOS simulators on Mac mini
• Connection and configuration of Android and Flutter Docker containers.

Stable and Fast Integration Tests

I discussed how to use a testing framework for quick and stable execution of integration tests in this article Flutter: Improving the Speed of Integration Testing.

Additionally, we can enhance the stability of our test runs by automatically restarting failed tests.

Future<void> runTest({
 required Future<void> Function() run,
 required Future<void> Function() after,
 int retries = 1,
}) async {
 int currentAttempt = 0;
 dynamic lastError;

 bool hasError;

 do {
  try {
   await run();
   hasError = false;
   currentAttempt++;
  } catch (catched) {
   hasError = true;
   lastError = catched;
   currentAttempt++;
  } finally {
   await after();
  }
 } while (hasError && currentAttempt <= retries);

 if (hasError) {
  throw lastError.toString();
 }
}

Execution Cycle:

• In the try block, we attempt to run the test. If the test completes successfully, the hasError variable is false, and currentAttempt is incremented by 1.
• If an error occurs within the try block, control is passed to the catch block. Here, hasError it is set to true, lastError is updated with the caught error and currentAttempt also increases by 1.
• In the finally block, the after the function is executed. If there is an error and the current attempt is less than or equal to the maximum retry count (retries), a delay is implemented before the test is rerun.
• The cycle continues if we have an error and the current attempt is less than or equal to retries.

Dynamic Creation and Use of Android Emulator and iOS Simulators on Mac Mini.

Using Android Emulators:

In the article Flutter: Dynamically Creating and Using Android Emulator for Integration Tests, I explored examples using flutter commands to work with emulators.

Let’s delve into creating and working with an Android emulator through AVD.

#!/usr/bin/env bash

emulator_name="emulator"
device_type="pixel"
sd_card_name="/sdcard.img"

echo "no" | avdmanager -v create avd --force --name ${emulator_name} --package "system-images;android-33;google_apis;x86_64" --tag "google_apis" --device ${device_type}

emulator_arguments=(-avd ${emulator_name} -sdcard ${sd_card_name} -verbose)

echo "Rendering: Headless swift shader (software) rendering mode is enabled"
emulator_arguments+=(-no-window -gpu swiftshader_indirect)

echo "Snapshots: Emulator will be run without snapshot feature"
emulator_arguments+=(-no-snapshot)

emulator_arguments+=(-wipe-data -no-boot-anim -no-audio -partition-size 8192)

# emulator uses adb so we make sure that the server is running
adb start-server

echo "no" | emulator "${emulator_arguments[@]}"

The main goal of the script is to automate the process of creating and launching an Android emulator with specific parameters.

Variable definition:

• emulator_name: the name of the Android Virtual Device (AVD).
• device_type: the device type (in this case, Pixel).
• sd_card_name: the path to the SD card image.

Creation of the AVD using the avdmanager command. This is done using the previously defined variables along with some other parameters.

Define arguments that will be passed to the emulator:

• avd ${emulator_name}: the name of the AVD.
• sdcard ${sd_card_name}: path to the SD card image.
• verbose: display detailed information during the emulator launch.
• wipe-data: clears user data on start.
• no-boot-anim: disables the boot-up animation.
• no-audio: disables sound.
• partition-size 8192: sets the partition size to 8192 MB.

Using iOS Simulators:

The simctl tool will be used to create the iOS simulator.

Let’s start by writing a bash script for simulator creation:

#!/bin/bash

MODEL="$1"
FILE="$2"

UUID=$(xcrun simctl create "Test-iPhone-${MODEL}" "com.apple.CoreSimulator.SimDeviceType.iPhone-${MODEL}" iOS16.4)
echo $UUID > ${FILE}.txt
xcrun simctl bootstatus $UUID -b

echo $UUID

This bash script performs the following actions:

1. Using the xcrun simctl create command, it creates a new device in the iOS Simulator with the specified iPhone model. This command returns a unique identifier (UUID) for the created device.
2. Writes the acquired UUID to a file named after the second argument with the .txt extension. This allows for the UUID to be reused later.
3. With xcrun simctl bootstatus, this command starts the created device and waits for the simulator to fully load. It offers options to boot the device if it isn't running automatically. The -b option ensures the device will be started if it hasn't been started yet. The command will wait for the simulator to fully load before completing its execution.

Example CI script:

integration_test_ios:
  before_script:
    - cd ios
    - pod update
    - cd ../
    - UUID=$(bash ./create_simulator.sh "13-Pro-Max" "simulator")
    - echo $UUID
 script:
    - flutter test integration_test_folder/integration_test_file_name.dart -d $UUID
  after_script:
    - UUID=$(cat ./simulator.txt)
    - xcrun simctl shutdown $UUID
    - xcrun simctl erase $UUID
    - xcrun simctl delete $UUID

This script creates an iPhone simulator, runs Flutter integration tests on that simulator, and then stops and removes the simulator.

When using CocoaPods, remember to update dependencies with pod update.

Running tests using multiple devices.

My colleague covered this in his article Boosting Flutter Integration tests execution speed in pipeline by 3x!

I suggest enhancing this implementation by adding a script that waits for the test application to be installed on the device.

Since all tests run within a single project, the test application will be built in a shared directory with the same name. To avoid overwriting, it’s essential to ensure that the testing application is fully installed on the device before launching the next test suite with the command.

wait_for_installation.sh

function waitForAppInstallationAndroid {
  local DEVICE=$1
  local PACKAGE_NAME=$2

  while true; do
    adb -s $DEVICE shell pm list packages | grep "$PACKAGE_NAME" &> /dev/null
    if [ $? -eq 0 ]; then
      break
    else
      sleep 5
    fi
  done
}

function waitForAppInstallationIOS {
  local DEVICE_UUID=$1
  local BUNDLE_ID=$2

  while true; do
    installed_apps=$(xcrun simctl listapps "$DEVICE_UUID" 2>/dev/null | grep "$BUNDLE_ID")
    if [ ! -z "$installed_apps" ]; then
      break
    else
      sleep 5
    fi
  done
}

Function waitForAppInstallationAndroid

1. local DEVICE=$1 and local PACKAGE_NAME=$2: These lines create local variables and assign values from the function's arguments. DEVICE the device identifier PACKAGE_NAME is the application's package name on Android.
2. while true; do ... done: This loop will continue to run until a break condition is met (in our case, break).
3. adb -s $DEVICE shell pm list packages | grep "$PACKAGE_NAME" &> /dev/null: This command uses the adb utility to fetch the list of installed packages on an Android device and checks for the package's presence using grep. The command's output is redirected to /dev/null hide it from the user.
4. if [ $? -eq 0 ]; then ... else ... fi: This if block checks the return code of the last executed command. If grep finds a match, it indicates the application is installed, and the loop is interrupted using break.

Function waitForAppInstallationIOS

1. local DEVICE_UUID=$1 and local BUNDLE_ID=$2: Similarly to the Android function, DEVICE_UUID is the unique identifier of an iOS device or simulator, and BUNDLE_ID is the application's identifier on iOS.
2. while true; do ... done: This loop will also keep running until a break condition is met.
3. installed_apps=$(xcrun simctl listapps "$DEVICE_UUID" 2>/dev/null | grep "$BUNDLE_ID"): This command uses xcrun simctl to fetch the list of installed applications on an iOS simulator and check if the desired application is installed using grep.
4. if [ ! -z "$installed_apps" ]; then ... else ... fi: If the installed_apps variable is not empty, it means grep finding a match and installing the application. The loop is then interrupted with break.

Both functions use sleep 5 to delay between checks to avoid overloading the system. You can adjust this value if needed.

Then, in your main script, use the source command to include wait_for_installation.sh.

#!/bin/bash

source wait_for_installation.sh

DEVICE1="$1"
DEVICE2="$2"
DEVICE3="$3"

PLATFORM="$4"

PACKAGE_NAME="your.package.name"
BUNDLE_ID="com.your.bundle.id"

flutter test integration_test/test_cases/group_1_test.dart -d "$DEVICE1" & P1=$!
if [ "$PLATFORM" == "android" ]; then
  waitForAppInstallationAndroid $DEVICE1 $PACKAGE_NAME
elif [ "$PLATFORM" == "ios" ]; then
  waitForAppInstallationIOS $DEVICE1 $BUNDLE_ID
fi

flutter test integration_test/test_cases/group_2_test.dart -d "$DEVICE2" & P2=$!
if [ "$PLATFORM" == "android" ]; then
  waitForAppInstallationAndroid $DEVICE2 $PACKAGE_NAME
elif [ "$PLATFORM" == "ios" ]; then
  waitForAppInstallationIOS $DEVICE2 $BUNDLE_ID
fi

flutter test integration_test/test_cases/group_3_test.dart "$DEVICE3" & P3=$!

wait $P1 $P2 $P3

Connecting and Configuring Android Docker and Flutter Docker Containers.

Regarding running tests in Docker containers, I discussed it in this article Flutter integration tests: Connection and Setting Docker containers.

Although Flutter testing is still relatively new and has limitations, our experience shows that this process can be structured with the right approach so that tests run quickly and efficiently.

Particular attention should be paid to how you approach the initialization and launch of your tests, which can speed up your test run time by orders of magnitude.