🔍 Flutter UI Testing

First thoughts about the Flutter UI test framework

Victor Vargas
Mar 14 · 5 min read

In the last few months I’ve noticed a trend of people talking about Flutter. As you might think, I’m not the only one that noticed that. Our mobile team wanted to ride the Flutter wave as well and went all-in rewriting the courier app at Stuart.

After evaluating the available tools to perform the UI tests, I realised that the only real alternative is the one that comes with the Flutter SDK. Appium doesn’t support it as everything seems to be embedded into a FlutterView. The same happens with XCUITest and Espresso, as there are no Ids nor accessibility identifiers.

Screenshot of the Android Layout Inspector on a Flutter application

As a QA engineer that wants to be up to date with everything new, I was excited to try out how the integration testing framework that Flutter provides works and be ready in case our mobile team decides to switch to Flutter (even if it’s in the long term).

Testing Out The Tests

Things that I wanted to test on this proof of concept are:

  • What the tests look like

The Flutter application runs on a single thread and the test execution is done in a separate thread that communicates through the application with a driver. Even though there’s not much information because it’s pretty new, Flutter provides you a step by step guide on how to start using it.

I literally spent an entire day testing it out before realising how amazing Futures are in Dart (and how easy it is to forget to write the await keyword in every future). My first working proof of concept was a single class that executed the typical login test.

This is how it looks:

Nice! We have our very first test!

Actually I was surprised because the Futures execution is very smooth and there’s no need to implement the custom waitForSomethingmethod (at least at the moment — I’m pretty sure that in the future it’ll be needed).

Setting Up The CI

Adding it to CI was actually pretty easy. Let me show you how I did it using a Jenkins declarative pipeline:

Now that our awesome test is running in CI for every commit, what if it fails?

I wanted to have a nice output archived in Jenkins that could help me easily identify what had failed. I found that there’s a JUnit reporter package available for Dart. What a surprise when I followed the step-by-step guide to install it:

Issue installing junit report

It seems that they’ve fixed it in latest versions, but I wasn’t able to use it yet as our app targets a lower version of Dart.

On the other hand, even if there’s no test report, when a test fails it’s pretty easy to find what caused it. One of Dart’s aims is to provide meaningful error messages that helps you or give some hint about how to fix the error. Sadly, the stacktrace seems like gibberish (you don’t even see the line of your application where it crashed), but the hint is really helpful.

This actually happened to me while doing this test and it drove me crazy as I wasn’t yet familiar with Futures — imagine: you forget to write the await on line 46, where you are typing the email.

This is the stack trace of the test:

This sounded to me like — “hey, you forgot to put an await and I’m failing because of that!”

One drawback I found was the possibility to run all the tests with a single command, instead of repeating the same command that I used in the Jenkinsfile:

flutter drive --target=test_driver/yourTestClass.dart

It seems that with JUnit it’s possible to have a command to run them all and generate a report:

flutter test --machine | tojunit

Mocking Network Responses

Last but not least, I wanted to check in this proof-of-concept for how to mock network responses. I was really excited when I saw that Mockito was available in Dart!

Since the app is built using the repository pattern (you can check this article written by Pau Picas, Android developer at Stuart, where he explains it in depth) all that needs to be done is to mock the needed repositories and, with dependency injection, use the mocked repository instead of the real one during the test session.

Easy peasy, right?

Nope!

It seems that for Flutter it is not possible to do in that way, as everything under the folder test_driver cannot contain any package or reference to the main app.

Remember how flutter UI testing works: the app is running in one thread and the test code in another, communication is then done via a driver. This makes it impossible to use the mocked repositories in the app code to set expectations like:

when(getMockedAuthManager.authenticate("my.user@stuart.com","mySuperSecretPassword")).thenReturn(authenticatedUser);

There are alternatives, such as mock web server, which we’ll try to avoid as we don’t want to maintain responses. If we want to use mockito, something custom will have to be built, but I left that out of the scope of the proof-of-concept as it’ll require some thinking.

Final Thoughts

As a recap of feelings after this proof of concept, even though I found some limitations doing a very simple test, I want to say that I find the framework very promising.

As soon as you get used to Dart, the learning curve seems to not be a big deal if you have used Selenium, Espresso, XCTest or similar ones. There is still a lack of documentation but it seems that the community is growing and, of course, we’ll have the support of our amazing mobile team!

Like what you see? Join us, we’re hiring. 🚀

Stuart Engineering

Developing last mile delivery

Thanks to Naoise

Victor Vargas

Written by

QA Engineer @Stuart

Stuart Engineering

Developing last mile delivery

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade