🔍 Flutter UI Testing
First thoughts about the Flutter UI test framework
In the last few months I’ve noticed a trend of people talking about Flutter. As you might think, I’m not the only one that noticed that. Our mobile team wanted to ride the Flutter wave as well and went all-in rewriting the courier app at Stuart.
After evaluating the available tools to perform the UI tests, I realised that the only real alternative is the one that comes with the Flutter SDK. Appium doesn’t support it as everything seems to be embedded into a FlutterView. The same happens with XCUITest and Espresso, as there are no Ids nor accessibility identifiers.
As a QA engineer that wants to be up to date with everything new, I was excited to try out how the integration testing framework that Flutter provides works and be ready in case our mobile team decides to switch to Flutter (even if it’s in the long term).
Testing Out The Tests
Things that I wanted to test on this proof of concept are:
- What the tests look like
- Test execution speed
- The ability to execute tests on a CI server
- Output in case of test failure(s)
- How to mock network requests
The Flutter application runs on a single thread and the test execution is done in a separate thread that communicates through the application with a driver. Even though there’s not much information because it’s pretty new, Flutter provides you a step by step guide on how to start using it.
I literally spent an entire day testing it out before realising how amazing Futures are in Dart (and how easy it is to forget to write the await keyword in every future). My first working proof of concept was a single class that executed the typical login test.
This is how it looks:
Nice! We have our very first test!
Actually I was surprised because the Futures execution is very smooth and there’s no need to implement the custom waitForSomethingmethod (at least at the moment — I’m pretty sure that in the future it’ll be needed).
Setting Up The CI
Adding it to CI was actually pretty easy. Let me show you how I did it using a Jenkins declarative pipeline:
Now that our awesome test is running in CI for every commit, what if it fails?
I wanted to have a nice output archived in Jenkins that could help me easily identify what had failed. I found that there’s a JUnit reporter package available for Dart. What a surprise when I followed the step-by-step guide to install it:
It seems that they’ve fixed it in latest versions, but I wasn’t able to use it yet as our app targets a lower version of Dart.
On the other hand, even if there’s no test report, when a test fails it’s pretty easy to find what caused it. One of Dart’s aims is to provide meaningful error messages that helps you or give some hint about how to fix the error. Sadly, the stacktrace seems like gibberish (you don’t even see the line of your application where it crashed), but the hint is really helpful.
This actually happened to me while doing this test and it drove me crazy as I wasn’t yet familiar with Futures — imagine: you forget to write the await on line 46, where you are typing the email.
This is the stack trace of the test:
This sounded to me like — “hey, you forgot to put an await and I’m failing because of that!”
One drawback I found was the possibility to run all the tests with a single command, instead of repeating the same command that I used in the Jenkinsfile:
flutter drive --target=test_driver/yourTestClass.dart
It seems that with JUnit it’s possible to have a command to run them all and generate a report:
flutter test --machine | tojunit
Mocking Network Responses
Last but not least, I wanted to check in this proof-of-concept for how to mock network responses. I was really excited when I saw that Mockito was available in Dart!
Since the app is built using the repository pattern (you can check this article written by Pau Picas, Android developer at Stuart, where he explains it in depth) all that needs to be done is to mock the needed repositories and, with dependency injection, use the mocked repository instead of the real one during the test session.
Easy peasy, right?
Nope!
It seems that for Flutter it is not possible to do in that way, as everything under the folder test_driver cannot contain any package or reference to the main app.
Remember how flutter UI testing works: the app is running in one thread and the test code in another, communication is then done via a driver. This makes it impossible to use the mocked repositories in the app code to set expectations like:
when(getMockedAuthManager.authenticate("my.user@stuart.com","mySuperSecretPassword")).thenReturn(authenticatedUser);
There are alternatives, such as mock web server, which we’ll try to avoid as we don’t want to maintain responses. If we want to use mockito, something custom will have to be built, but I left that out of the scope of the proof-of-concept as it’ll require some thinking.
Final Thoughts
As a recap of feelings after this proof of concept, even though I found some limitations doing a very simple test, I want to say that I find the framework very promising.
As soon as you get used to Dart, the learning curve seems to not be a big deal if you have used Selenium, Espresso, XCTest or similar ones. There is still a lack of documentation but it seems that the community is growing and, of course, we’ll have the support of our amazing mobile team!
Like what you see? Join us, we’re hiring. 🚀
