Getting Started With Automating User Testing in Flutter

How you can build automated and user-centered test cases from zero.

History tells us that not testing is pretty bad. So, how can we correctly — and quickly — test our apps?

One way to do that is by working with testers. They will run our app, again and again, to discover hidden bugs, mimicking the expected (and the unexpected) user’s behavior. This is a good method, and when the number of testers increases, so does the probability to get bugs discovered. But we don’t always have enough resources to work with them: It’s necessary to find alternative methods to try to mimic the user’s behavior.

Automated Testing

When developing Flutter apps we can create a lot of automated tests: Unit tests to verify the correct response of our functions or methods, integration tests to check the behavior of some parts of our system when working together and widget tests to verify that the widget’s UI looks and interacts as expected. But we can’t really see (at least in the last one) how the app is working, how the widgets are rendered, or how they are responding to user interaction. As the Flutter team says in the Flutter Test page:

“A widget test’s environment is replaced with an implementation much simpler than a full-blown UI system.”

However, you may want to watch how every widget or interface is reacting. As shown below, the app is running and interacting with itself, but without human intervention. And more importantly, you can watch as it does so.

Today, I am going to guide you step by step to get that kind of automatic behavior. By the end of this article, you should be able to:

• Setup your Flutter app to handle the automated tests.
• Create interactive tests with JSON format.
• Import and run those tests as part of your app.

Building a testable app

What are we going to do?

You will need to follow the next steps to get your fully automatic testable app:

1. Create the app and import the necessary components.
2. Modify the app structure.
3. Add a TestController: Manage your tests.
4. Create the test steps using JSON.
5. Add a TestRunner: Provide the TestController to the widget tree.
6. Select which widgets are going to be tested.
7. Start the tests.

1. Create the app and import the necessary components

This time we are going to test the most simple Flutter app: The CounterApp. To do so we create a new Flutter project:

flutter create new_app

After that, we need to import the automated_testing_framework package into the pubspec.yaml to create the tests:

automated_testing_framework: ^1.0.3

2. Modify the app structure

The test package will need to “reset” the app before starting the tests. To simplify it, let’s rewrite the MyApp widget as a StatefulWidget . This way, we can reset the app by rebuilding the entire _MyAppState . The code should look like below:

3. Add a TestController: Manage your tests

We need to instantiate a TestController from the automated_testing_framework . It requires two arguments: A NavigatorKey navigatorKeyand a Future<void> Function onReset . The former should be the same key attached to a MaterialApp to allow the navigation within the framework and the latter is a function to reset the application to an initial state. So, inside the MyAppState we write:

You may be wondering: What is the _onReset() function? What is the TestController doing? Why is there an ‘assets/simple.json’? Let’s answer these questions.

Function onReset():

When the tests are complete, the framework will want to “reset” the app state to an initial state¹. So it’s our responsibility to tell the framework how to reset it. In this case, we just replaced the current route with the ‘/’ route (the default route). This way we can restart the entire app. As we will see soon, the framework shows us a summary of the tests: which passed and which didn’t. This summary is presented as a new page in the app, so it’s necessary to pop the routes our test has created, before restarting the state. (You can test what happens if you remove the while).

What is the TestController doing?

The TestController is the core of the package: it allows us to create, load, and execute the tests. For now, we only want to load and execute our tests. But… What tests? The framework allows us to load custom tests from JSON files (among other load methods), so we can create the tests as independent files and put inside the app as assets. How?

4. Create the test steps using JSON

First, we create the assets folder inside the app root folder. After that we need to tell the pubspec.yaml to use that folder:

flutter:
uses-material-design: true
assets:
- assets/

For this tutorial let’s start with a very simple test: Press the increment button and check if the text showing the _counter has changed. To do so, you need to create a simple.json file and paste this in it:

The id of each step is the name of the action we want to do. In this case, the press action is called"id": "tap" but which widget are we going to press? For now, let’s say we are going to tap a button with testableId equals to "fab" . It’s the same case for the next step: To check if the text has changed it’s necessary to check if its value has changed. So we need to ensure the text value is equals to something. As we only pressed the increment button once, the value changed from “0” to “1". To do this verification, we can use the assert_value test step, and check if the widget with "testableId": “text_value” has a "value" equals to "1" . The testableId values are string ids we are going to create in the next steps. You only have to remember that they are a way to identify a specific widget.

Now that the TestController is created, how do we use it?

5. Add a TestRunner: Provide the TestController to the widget tree

We need to enclose the entire app to be tested (in this case the MaterialApp) inside a TestRunner . That widget requires two parameters: the TestController and a child widget. This way the framework can access the controller, though the TestRunner also provides other properties to enable/disable tests, change the method to display the progress of the tests, etc. And also we have to add one property to MaterialApp :

• navigatorKey : To pass the _navigatorKey we created before.

Our code now looks like this:

6. Select which widgets are going to be tested

We need to tell the framework which are the widgets we are going to test (Do you remember the testableId?). To do so, the framework uses a Testable widget. This widget should wrap each widget we want to interact with: to make a press gesture, to read a value, to set a value, etc. In our example, we are going to test the counter, so we will interact with two widgets:

• FloatingActionButton to send the onPress event ( tap ).
• Text to assert the expected number value ( assert_value ).

Let’s assign to these widgets their testableId . How? Through the Testable widget:

As you can tell, both the Text and the FloatingActionButton are enclosed in a Testable widget with an id .

7. Start the tests

Now, we need to start the tests in some way. In this example, let’s start the tests right after the app is built.

Creating a _runTests() function:

To run the tests we need to:

• Load the test to the controller.
• Execute the pending tests (in this case, the only test we have).

Let’s create a simple function_runTests() to load and execute our tests:

Calling the _runTests() function:

We are going to use the initState function of our _MyAppState to initialize the process. After adding the _runTests() function to the initState, the final code should look like this:

Running the automated test :

To run the test we have created, we just need to run the app in DEBUG mode and enjoy the results.

All of the processes are automatized!

Obviously, this was a very simple test, but you can create more complex ones once you have wrapped the appropriate widgets to be identified by this framework. You would only need to add new JSON files with all the tests and register them in the TestController.

If we want to disable the tests when the app is built into RELEASE mode, the TestRunner supports a property called enabled which receives a boolean to enable or disable the test executions. This way we can be sure the tests only will run when we want to.

Conclusion

We learned how to automatize some user tests. This package is not intended to replace the Widget testing offered by the Flutter team but to run the app as if the final user were using it. This way we can ensure our code is ready to be released.

The example app code can be found here. This package has a lot of functionalities we didn’t explore here such as creating the tests inside the app, loading and saving tests from databases, gestures like double tap or drag, etc. For this reason, I will post a second part covering a few more features and capabilities of the automated_testing_framework . If you can’t wait to discover more please visit its pub.dev page.

If you have any questions, please leave them in the comments and I will be happy to help!

I hope this post has been useful to you. If you want to know how to render all types of widgets from JSON, you might also be interested to read my previous post.

[1] This would change from app to app, depending on the needs. Maybe you don’t want to reset to the very initial state but some other state.