We have all heard that testing is of crucial importance, and it has tons of benefits such as improving confidence and reducing bugs. However, end-to-end/integration tests in Flutter are often time-consuming to write, debug and maintain, and much time is wasted. This makes testing not a very pleasant piece of work and it is tempting to skip it (at least for me :/ ). Therefore, let us rethink it and try to reduce the time-wasters.
Ideal vs (old) reality
Let’s close our eyes and imagine: What should tests look like in an ideal world? Well, just tell the computer what we expect to happen without wasting a single word. That’s all. Then, when tests pass, we are confident; whenever tests fail, it should clearly point out where our business code goes wrong so we can fix it.
However, that was not the reality. In the real world, much more time is spent (read: wasted) on other aspects of tests.
A significant difference between ideal and reality is the time it needs to locate the root cause of a test failure, which may be called (generalized) debuggability. On one hand, after writing a test, it is often the case that it fails a few times before it passes, because either our test is buggy or the business code has problems. On the other hand, regression (a test that passed before fails now) happens from time to time, which indicates there are some problems in our updated code. In either case, we need to dig out the root cause to fix it. Therefore, the slow speed of bug locating contributes a lot to wasted time.
Flutter’s integration_test has, well, non-excellent developer experience for this, even though the whole Flutter framework is very wonderful and productive. Consider a typical scenario: An end-to-end/integration test taps/drags/scrolls many elements, and finally fails at an assertion. Then, how to know which step goes wrong? Let alone such failures can happen inside a CI, where we often only have logs available. One approach may be reading and analyzing the logs - if we have a thorough logging system - but it is still slow because many bugs can be spotted quickly by looking at UI or time traveling. Carefully observing what the simulator displays is also not a perfect choice, because the taps/drags often happen too quickly, and this is impossible for CI mode. Other details during cause-digging can also waste some time.
There are also other aspects. The lack of retryability wastes some time. In end-to-end/integration tests, nondeterministic events, such as network requests, happen quite frequently. We cannot know when it will end exactly. Thus, the tests have to be written with waits and retries. Flakiness, which is natural in end-to-end tests, also causes some false positives. Nobody wants to spend time examining the test “failure”, only to realize it is only flaky.
That was the real world. Can we do it better?
A solution
All the headaches above lead me to write an open-source library:
The package, flutter_convenient_test, tries to reduce the time-wasting processes mentioned above to a minimum. Surely, it is imperfect (and young), but I hope my small step could help you save some time and inspire more time-saving enhancements.
Remark: It is built on top of integration_test, so you can still use your favorite packages like integration_test, mockito, flutter_test, etc, and migrate to this library with minor modifications.
Let's see what the package does for debuggability. (All the following features are also applicable when running in CI — it will generate a data pack and we can visualize it later on the desktop.)
To begin with, we can view all actions/assertions taken in the tests, with friendly descriptions.
Next, we can do time traveling with screenshots. What did the UI look like when that button was tapped even 50 steps ago? Now we know everything.
Want to see it live instead of screenshots? No problem — we have the video records.
There are more functionalities to make bug-spotting easier. For example:
- We can switch the mode and play with the app interactively, even if we are running tests. This seems not easy in the era of
integration_test, and we had to make a full restart for that. - After modifying the code, tests can be re-executed within seconds, not minutes.
- Enhanced goldens — configurable “tolerations” (allow a portion of pixels to be different to some extent compared with the golden image); a full panel to examine golden differences with a magnifier.
- A single test or single group can be run within a click.
- …
Some aspects are reduced to zero-second jobs as well:
- For retryability, this package automatically inserts
pumpand retries whenever appropriate, so there is completely zero need for humans to intervene and write code. - When it comes to flakiness, the library fully understands it and will neither wrongly view it as failed (and maybe a big red error in CI) nor consider it has succeeded.
- …
Conclusion
In summary, we have seen testing in Flutter did contain some wasted time, including extra time to locate bug causes, lack of retryability, and flakiness. Therefore, I made flutter_convenient_test trying to reduce it to a minimum. Hope it can save your time!
