Updates on Flutter Testing

Updated integration testing that supports Firebase Test Lab on mobile; stand-alone support for web and desktop, too!

Flutter’s goal is to provide an open framework for quickly building beautiful native apps on multiple platforms. A large part of achieving that goal is the ability for you to easily test your Flutter apps to make sure that they work and look the way you want them to on the platforms you’re targeting. Some of that testing can be handled by unit tests written using the unit test framework built into Dart. Dart unit testing works exceedingly well for non-UI testing; it runs on your development machine and doesn’t rely on the GUI of your Flutter app.

Integration testing (also called end-to-end testing or GUI testing) is a step up from unit testing because integration testing attempts to simulate a user interacting with your app by doing things like pushing buttons, selecting items, and typing on the keyboard. This testing is automated, to avoid having humans do that kind of repetitive work because, frankly, we’re not that great at it. Our initial solution to avoiding human interaction was a special way to write Flutter tests. These tests are run from the host like Dart unit tests, and drive the app running on a real or virtual device, just as a person would. These kinds of tests were called Flutter driver tests because you use the flutter_driver package and the flutter drive command line to drive the GUI of your app.

Unfortunately, Flutter driver tests have a number of issues. One issue is that tests run from the development machine and communicate with the app on the device, which means that the tests aren’t suitable for running on a device farm like Firebase Test Lab. Another issue is that having a separate process for your tests makes it difficult to check the state of your app. You can check the output, but how do you know, for example, the internal state of the app. And, finally, the flutter_driver API is more complicated than it needs to be, particularly when it comes to writing the code to find the appropriate widgets on the screen to test against.

And so, to solve these problems and continue to improve the Flutter testing experience across a growing number of targets, we’re happy to announce version 1.0 of the integration_test plugin, a simpler way to test your Flutter apps that also supports Firebase Test Lab.

Getting started with integration_test

Using the integration_test plugin requires two steps. First, add the plugin to your pubspec.yaml file as a development dependency, and use flutter pub get to pull the plugin into your project:

# pubspec.yaml
…
dev_dependencies:
  flutter_test:
    sdk: flutter
  integration_test: ^1.0.0
…

Then, use theintegration_test package in your test code:

Notice the import of the integration_test package and the call to IntegrationTestWidgetsFlutterBinding.ensureInitialized(), which ensures that the package is properly initialized. Also notice the standard WidgetTester test code. This is the same test code generated by the default Counter app template during flutter create.

While integration_test enables your tests to be bundled into your app in a stand-alone way (which is a requirement for device farms like Firebase Test Lab), it requires some build magic that we’ll cover below. During the testing development process, however, it’s handy to just be able to run your tests interactively via the command line and for that, you’ll need a new entry point:

This call to integrationDriver is a convenience wrapper around the integrationDriver plugin to enable you to run your new tests with the flutter drive command:

$ flutter drive \
  --driver=test_driver/integration_test.dart \
  --target=test/widget_integration_test.dart

This command deploys your app to the simulator, executes your tests, and shows you the results.

Flutter integration_test in action

This particular test is running on iOS, but the integration_test plugin also works for Android by changing the--device-id option, as appropriate. Furthermore, you can run your integration tests against the web and desktop targets, too, although this functionality is still is pre-release.

Running on Firebase Test Lab

After you know that your tests work locally using either virtual or physical hardware, you can set your app loose on the plethora of devices available through Firebase Test Lab.

Firebase Test Lab console

Firebase Test Lab is a cloud-based, app testing infrastructure. With one operation, you can test your Android or iOS app across a wide variety of devices and device configurations, and see the results — including logs, videos, and screenshots — in the Firebase console. One of the major advancements of the integration_test plugin is the ability to run your Flutter apps targeting Android and iOS on Firebase Test Lab, giving you the ability to test across hundreds of devices simultaneously in order to find platform, form factor, or device-specific issues before shipping your app.

To run a test on Firebase Test Lab, you need to do some configuration and use Gradle commands to build an instrumentation test for both Android and iOS, as appropriate. For those details, check out the new integration test documentation on flutter.dev.

Migrating from Flutter driver tests

For those of you with existing flutter_driver tests, moving to the new API isn’t too difficult. In addition to the appropriate initialization that was previously mentioned, you also need to move to the new WidgetTester API.

flutter_driver API (left) vs. WidgetTesting API (right)

The flutter_driver API (shown on the left) and the WidgetTester API (shown on the right) are conceptually very similar, but you can see that a lot of the details are different. For example, instead of calling the waitFor method on the flutter_driver, you call the pumpAndSettle method on the WidgetTester. The former waits for a specific widget to appear, whereas the latter waits for the app’s UI rendering phase to settle. After you have a specific widget, you act on it similarly with the two APIs, but you use different objects. The WidgetTest API is more in-line with what you’re used to seeing in Dart unit tests. As you can see, the expect method is used to ensure that the contents of a widget are what you expect.

For the details about how to move your tests from flutter_driver to integration_test and WidgetTester, see the migration docs on flutter.dev.

Native UI testing

If you have an existing Android or iOS app to which you’re adding Flutter using Add-to-App, then you’re likely to have existing integration tests for those native apps that you want to leverage. In that case, add the tests for your Flutter screens to those existing tests. For Android, you can add the tests if they’re written using the Espresso framework using the espresso plugin, which provides bindings for Espresso tests of Flutter Android apps. We’ll have a similar plugin to support native iOS tests created using Earl Grey soon.

Summary

This announcement presents a new foundation for integration testing using the new integration_test plugin for Flutter. Not only is the API simpler and more consistent, tests written using integration_test can run on hundreds of different devices using the Firebase Test Lab. The Flutter team plans to build on this new foundation by updating the flutter create template to use integration_test out of the box, adding updates to the test output to support existing testing tools using the JUnit format, adding the ability to take screenshots during tests for golden testing comparisons, and more. For the full list of details about our new integration testing recommendations for your Flutter apps and plugins, see the testing docs on flutter.dev.