Blazingly Fast Flutter Driver Tests
TL;DR: Ever wanted to speed up your Flutter Driver Tests by 750%?
Flutter has two approaches to testing user interfaces — Widget Testing and Flutter Driver Testing.
Widget Testing has more in common with Unit Testing — it does not need to display UI to test it, whereas Flutter Driver is closer to UI (similar to Appium) as it interacts with rendered components.
In your projects, you will be using Widget Testing much more as it’s more robust and faster. Using Flutter Driver tests has benefits:
- You see how your screen looks like — you can verify if everything renders properly on the different screen sizes and in different languages
- You can create png screenshots of the tests and use them for regression testing
- You can test end-to-end — backend calls work as in a normal application. In Widget Test Fake Futures are used and that prevents you from testing your UI together with the backend
There is still a couple of drawbacks of Flutter Driver tests:
They require a real device to run on
In comparison to Widget Tests, which basically run like Unit Tests (headlessly on a development machine), Flutter Driver Tests require a device that can render UI. As mobile developers, we know that maintaining an emulator or a real device is quite a pain.
They are much slower than Widget Tests
Driver Tests build the whole application to run the tests. After the application is launched, the tests need to connect to Dart VM to be able to interact with the application — that takes around 10 seconds for each test file.
It’s hard to inject different setup from test file into the application file
The application file and test file are separated. For every test file (main_text.dart
) you need one application file (main.dart
).
It’s hard from the test to change the setup of the application.
This separation is even more visible when trying to use any class from dart:ui
package: in the application file, you can obviously use Widgets
.
In _test.dart
file, you cannot use anything from dart:ui
(that includes even Locale
class) because in runtime the test will instantly fail.
Each test in _test.dart
file depends on the previous test
Because you cannot easily change the setup of the application file, you also cannot ‘reset’ the state after each test.
If the first test navigates to a certain page, the next test will start on that page.
Running one test file at a time
In flutter drive
command there is no option to specify multiple files to run — only one at a time.
Hard to debug failing test
When a test fails it will only say it did not find an element. If you are lucky you can see on the screen what’s wrong but the majority of issues won’t so obvious.
With so many challenges I am not surprised that people do not use Flutter Driver Test so often.
I really believe that if they were resolved, many more people would start using those
…so let’s fix them!
Fixing: Hard to debug failing test
You cannot debug an application when the test is running, but you can repeat the same steps of the test when you are debugging the main.dart
file.
If a test in main_test.dart
fails (eg HTTP request is not properly mocked), then just start the main.dart
as a normal application in the debug mode.
First, you need to add -t
(or — target
) parameter in the IntelliJ settings:
Then press the debug button — at this point just reproduce your test’s steps.
IMPORTANT: If you need to enter manually text into TextFields
, remember to comment out enableFlutterDriverExtension
from main.dart
.
Fixing: They require a real device to run on
Now Flutter Desktop gets handy — you can build your application for Windows/macOS/Linux so why not running UI tests against desktop as well.
You won’t be able to run every (testing device-specific components like WebView
Widget
), but from my experience, those tests are around 5% of all the Flutter Driver Tests.
Run main.dart
file as a normal desktop application:
flutter run -d windows
After the application launches you will see the message:
An Observatory debugger and profiler on Windows is available at: http://127.0.0.1:65521/vBWe3pt2_0k=/
The URL is here important as the tests require it to connect to the application
Run the main_test_dart and pass that link to it:
dart main_test.dart http://127.0.0.1:65521/vBWe3pt2_0k=/
dart
command can run any file that has main()
function.
Within the main_test.dart
file you need to extract that link form args
that you receive in main(List<String> args)
and pass it as dartVmServiceUrl
to FlutterDriver.connect
.
And done — the test is running on your desktop! Or even docker!
Fixing: Running one test file at a time
From the previous fix we’ve learned that we need to do three things to run a single file:
- Run
main.dart
withflutter run
command - Wait for the app to start and copy the Dart VM URL
- Run
main_test.dart
with given URL
To run multiple files you just need a script that:
- Looks up all the files ending with
_test.dart
in a given folder that are the tests files - Looks up files that have the same name as the tests but without
_test.dart
— those are the application files - Runs flutter run command for each application file sequentially and observe the console output for
is available at: (http://.*/)
match and copy it - Runs
dart <test file> <dart vm url>
on each file
Now you have automated running multiple test files.
Fixing: They are much slower than Widget Tests
At this point, the test from a single file runs quite quickly but when we start each test file we need to recompile the application and reconnect to the Dart VM which can take an additional 20 seconds per file.
To improve this we simply should leverage Flutter hot restart.
This includes two things:
Communicate from main_test.dart
that main.dart
should perform a hot restart
This can be achieved with FlutterDriver::requestData
method that sends a message to main.dart
. Inside main.dart
you will receive this message in enableFlutterDriverExtension
handler:
enableFlutterDriverExtension(
handler: (request) async {
// Here will be the message
},
);
Perform hot restart on main.dart
side
To do that, you need to wrap your main application in a widget that will reinsert your application’s widget into the widget tree, I use streams for the communication:
Done. Now we need to run all our tests from a single _test.dart file:
import ‘main1_test.dart’ as main_file_1;
import ‘main2_test.dart’ as main_file_2;
import ‘main3_test.dart’ as main_file_3;void main(List<String> args) {
main_file_1.main(args);
main_file_2.main(args);
main_file_3.main(args);
}
Fixing: It’s hard to inject different setup from test file into the application file
Now when we can communicate with the app we need to send serialized configuration using FlutterDriver::requestData
method.
I would recommend having a configuration class that can be passed to your MyApplication
widget.
For me, this config contains the initial route, page’s arguments or even it specifies if the application should fake or real HTTP requests.
Fixed: Each test in _test file depends on the previous test
With the ability to send a restart request with a specific configuration, we can restart the application between each test:
setUp(() async {
await restart(
driver,
config: const Configuration(
route: Routes.curves,
repeatAnimations: false,
),
);
});test(‘shows curves’, () async {
await driver.waitFor(find.byType(‘CurvesPage’));
});test(‘scroll’, () async {
await driver.scroll(find.byType(‘CurvesPage’), 0, -400,
const Duration(milliseconds: 100));
await driver.waitFor(find.byType(‘CurvesSection’));
}, retry: 1);
With this change, every test has a fresh start.
If your tests are flaky, you can now add retry
property to rerun tests if it fails.
Final Result
Here you can find an example application and run Flutter Driver Tests.
Command-line Tool
We’ve created a package + command-line tool so that you do not need to write all this boilerplate code yourself.
Final Thoughts
Before the injection of configuration and hot restart, our Flutter Driver Tests took 15 minutes for around 150 tests. After the switch, they take around 30 seconds on a local machine and 2 minutes on dockerized CI.
Personally, I think that it’s amazing to have access to the technology that allows you to run hundreds of UI tests on every commit you push.