Automated testing using ATDD in Flutter
In my last article, I talked about Flutter Driver and its methods that can be used to write integration tests that help automate feature flows and its functionalities.
In today’s article, I am going to give an overview / introduction of writing end to end integration tests using ATDD (Acceptance Test Driven Development) for Flutter apps.
What is ATDD ?
ATDD is an agile development methodology based on the communication between business customers, product owners, developers, and testers. Basically, the agile development team collaborates to define acceptance criteria for user stories before the implementation begins.
ATDD focuses on a test first approach which helps to determine test coverage for each user story, write manual or automated tests for UI layer of the app at the beginning of the sprint, execute those tests once the user story is implemented, identify and fix bugs along the way, refactor the tests as needed and, finally getting sign-off.
The tests written using ATDD are usually in simple English, so that non-technical team members can review / go through them, which helps them to understand the test scenarios, scope and what the test is actually validating.
Since ATDD is a vast topic in itself, I kept its overview short, as the main focus of this article is its usage in writing automated tests in Flutter. You can read more about ATDD here.
How to implement ATDD ?
In most cases, a combination of Cucumber and Gherkin language is used to write testing scenarios. Gherkin basically describes the features of the application that contains several scenarios that can be tested as part of test coverage for a user story and is written using a set of Given,When, Then keywords. Cucumber is the testing tool which understands and reads code written in Gherkin language based on the keywords used in the tests.
If we are to write acceptance tests for an Android app, we usually use Cucumber plugin since it supports Java. In the case of Flutter, we’ll make use of flutter_gherkin plugin which is a Gherkin parser and Flutter test runner so you can run your Gherkin scenarios in Flutter.
Demo app for test
I created a simple login screen that we’ll use to write the end to end automated acceptance tests. We’ll use the “happy path” scenario (below) for validation.
User enters email and password and hits login button -> User should navigate to next screen.
Note that, for demo purpose, we’ll not consider if the email and password is valid, but will use generic test data.
Test setup
Since our aim is to write end to end integration tests to validate login screen, we’ll need a provision to run our app in separate thread and instrument it. For this, we’ll leverage Flutter Driver.
Follow steps 2 to 4 from this link which will create a new test_driver folder under which we’ll be creating our feature and step definition files which are two important aspects of ATDD methodology.
Once you complete the above steps, inside the test_driver folder, create 2 more folders named features and steps. The features folder will have the tests we are going to write in Gherkin. The steps folder will contain the actual code for each statement. The features folder accepts files with a .feature extension. I’ve listed the steps on how to create a feature file in Android Studio here.
Next, right-click on the steps folder and create new dart file. Name it steps.
Once you’ve complete the above steps, the folder structure should look like this:
With this, we are set to write our first automated acceptance test.
In login.feature, we’ll take the happy path scenario mentioned above and write our test:
Feature: Login
User should be able to login successfully after clicking login button.
Scenario: User logs in successfully
Given I expect the user enters email
And I expect the user enters password
When user hits Login button
Then user should land on next screenFeature is the Gherkin pre-defined keyword that takes the name of the feature we are testing, followed by a one-line description.
Scenario is another keyword that takes the name of the scenario we’ll be testing for the feature mentioned above. Next, we write the statements using GWT (Given, When, Then) keywords to describe the scenario we’ll be testing.
Next, we need to write code to validate each statement and we’ll make use of the steps.dart file we created earlier. To begin with, we create a class named LoginValidation and extend it from one of the methods provided by flutter_gherkin plugin.
How do we determine which method to extend from?
The thing that will help narrow your options most when determining the method to extend is the keyword we are using for the statement to be validated. Something that makes it easier to make a final decision is that flutter_gherkin exposes various methods for each of the Gherkin keyword.
In the case of the first statement: Given I expect the user enters email, the plugin has a method named GivenWithWorld with one type of input parameter named FlutterWorld. FlutterWorld parameter is a Flutter context that allows access to the Flutter driver instance within the step along with the methods.
We need to override a couple of methods after extending GivenWithWorld class. They are:
@override
Future<void> executeStep() async {}@override
// TODO: implement pattern
RegExp get pattern => null
The first method, as its name suggests, will contain the validation code to satisfy the statement we are testing and here we will write the steps using FlutterDriverUtils methods which exposes methods such as tap, enterText, waitFor and so on. Let’s see how to do this.
@override
Future<void> executeStep() async {
String input1 = "test@test.com";
await FlutterDriverUtils.tap(world.driver, find.byValueKey('inputKeyString'));
await FlutterDriverUtils.enterText(world.driver, find.byValueKey('inputKeyString'), input1);
}This is very straightforward code which first identifies the widget by using its unique Key, taps the field and then enters the text. world.driver is an instance of FlutterWorld that allows us to use flutter driver context.
In the next method RegExp, we provide the statement as a string we are validating.
RegExp get pattern => RegExp(r"I expect the user enters email");And that completes our validation for the first statement. For the next statement, we’ll use a similar approach by extending from a different class named AndWithWorld since the statement uses And keyword. We’ll start by creating a new class to validate a password. Code for this is as follows:
class PasswordValidation extends AndWithWorld<FlutterWorld> {
@override
Future<void> executeStep() async {
String password = "test1234";
await FlutterDriverUtils.tap(world.driver, find.byValueKey('passKeyString'));
await FlutterDriverUtils.enterText(world.driver, find.byValueKey('passKeyString'), password);
}
@override
// TODO: implement pattern
RegExp get pattern => RegExp(r"I expect the user enters password");
}Similarly, code validation for the last two statements is as below:
class LoginButton extends WhenWithWorld<FlutterWorld> {
@override
Future<void> executeStep() async {
await FlutterDriverUtils.tap(world.driver, find.byValueKey('button'));
}
@override
// TODO: implement pattern
RegExp get pattern => RegExp(r"user hits login button");
}
class Nav_Validation extends ThenWithWorld<FlutterWorld> {
@override
Future<void> executeStep() async {
await FlutterDriverUtils.waitForFlutter(world.driver);
await FlutterDriverUtils.isPresent(find.byValueKey('nextScreenKey'), world.driver);
}
@override
// TODO: implement pattern
RegExp get pattern => RegExp(r"user should land on next screen");
}The complete code inside steps.dart is as below:
import 'package:flutter_driver/flutter_driver.dart';
import 'package:flutter_gherkin/flutter_gherkin.dart';
class LoginValidation extends GivenWithWorld<FlutterWorld> {
@override
Future<void> executeStep() async {
String input1 = "test@test.com";
await FlutterDriverUtils.tap(world.driver, find.byValueKey('inputKeyString'));
await FlutterDriverUtils.enterText(world.driver, find.byValueKey('inputKeyString'), input1);
}
@override
// TODO: implement pattern
RegExp get pattern => RegExp(r"I expect the user enters email");
}
class PasswordValidation extends AndWithWorld<FlutterWorld> {
@override
Future<void> executeStep() async {
String password = "test1234";
await FlutterDriverUtils.tap(world.driver, find.byValueKey('passKeyString'));
await FlutterDriverUtils.enterText(world.driver, find.byValueKey('passKeyString'), password);
}
@override
// TODO: implement pattern
RegExp get pattern => RegExp(r"I expect the user enters password");
}
class LoginButton extends WhenWithWorld<FlutterWorld> {
@override
Future<void> executeStep() async {
await FlutterDriverUtils.tap(world.driver, find.byValueKey('button'));
}
@override
// TODO: implement pattern
RegExp get pattern => RegExp(r"user hits login button");
}
class Nav_Validation extends ThenWithWorld<FlutterWorld> {
@override
Future<void> executeStep() async {
await FlutterDriverUtils.waitForFlutter(world.driver);
await FlutterDriverUtils.isPresent(find.byValueKey('nextScreenKey'), world.driver);
}
@override
// TODO: implement pattern
RegExp get pattern => RegExp(r"user should land on next screen");
}With this, we’ve completed writing the code to validate our first acceptance test. Now let’s see how to run this test and confirm it passes.
Running the acceptance test
We have a feature file and code to validate our happy path scenario. Now we need some kind of a bridge that will call our test and execute it. For this, we’ll make use of the standard configuration provided by flutter_gherkin that helps us to execute our test and also see the results in the terminal.
In app_test.dart, remove the existing code and replace it with this:
import 'dart:async';
import 'package:glob/glob.dart';
import 'package:flutter_gherkin/flutter_gherkin.dart';
import 'steps/steps.dart'; <---- Make sure to import steps.dart
Future<void> main() {
final config = FlutterTestConfiguration()
..features = [Glob(r"test_driver/features/**.feature")]
..stepDefinitions = [LoginValidation(), PasswordValidation(), LoginButton(), Nav_Validation()]
..reporters = [ProgressReporter(), TestRunSummaryReporter()]
..restartAppBetweenScenarios = true
..targetAppPath = "test_driver/app.dart"
..exitAfterTestRun = true;
return GherkinRunner().execute(config);
}Consider main() function as the entry point for test execution, which creates a FlutterTestConfiguration object that helps to locate the feature files present under thetest_driver folder, and then calls the classes we created to validate each statement in a scenario. It's important to note the classes we create in steps.dart need to be referenced by the ..stepDefinitions argument that tells the parser to execute code inside each class mentioned. The reporters argument takes the default ProgressReporter() and TestRunSummaryReporter() classes; then shows the progress of the test running and displays a summary of the test run. targetAppPath specifies the path to the app under test. We finally run the test from a terminal (project root path) using this command:
dart test_driver/app_test.dartThe above command invokes an instrumented version of the app under test on a connected device, then executes the statements for the scenario mentioned in the feature file. Below is the snippet of a terminal showing progress, as well as a summary of the test run. We can also see the execution happening on the device:
With this, we’ve successfully written and executed an automated test using the ATDD for Flutter app. Likewise, we can write acceptance tests for other scenarios to validate paths for the login screen and subsequently for other features of an app.
My objective in this article was to show how we can use ATDD to achieve automated testing, and show how to use the flutter_gherkin plugin to achieve this. There are still a lot of options to explore using this plugin which I haven’t covered here, but I hope you were able to get the general idea and see the feasibility of using ATDD in Flutter.
If you have another approach or another way to achieve ATDD in Flutter, please feel free to share it here and I’ll be happy to check it out!
Thanks for reading. I am available on Twitter, LinkedIn, and Github.
My other articles on Flutter are:
