How to write an automated UI testing? That is a problem has been bothering me, a Qt developer, for many years.
It doesn’t mean I don’t know how to do that at all. I just concern about the efficiency. Basically, it is a three steps process. 1) Simulate the operation environment. 2) Simulate the user input. 3) Write a test to validate the result.
The first step is already a hard problem if components are highly coupled, it won’t be easy to break down for the test program. Step 2 and 3 are also a tight work if the test coverage requirements are high.
And worse come to worst, the UI requirements are always changing. The previously written test case may needs a rewrite.
In a project with rapid changing of UI requirements, I would simply give up the automated UI test. Only focus on the application logic and data model. Somehow It is a kind of technical debt, but I don’t really want to waste time in rewriting the test cases.
The world was changed when I found a method called snapshot testing from Jest.
Jest — Snapshot Testing
Quoted from Jest
A typical snapshot test case for a mobile app renders a UI component, takes a screenshot, then compares it to a reference image stored alongside the test. The test will fail if the two images do not match: either the change is unexpected, or the screenshot needs to be updated to the new version of the UI component.
A similar approach can be taken when it comes to testing your React components. Instead of rendering the graphical UI, which would require building the entire app, you can use a test renderer to quickly generate a serializable value for your React tree. Consider this example test for a simple Link component:
It is a very interesting method. Compare by screenshots is a well-known methodology but the setup is a bit troublesome. (You doesn’t want to store the screenshot in VCS) Moreover, it is very easy to be broken by external factor even the program is unchanged.
But keeping the snapshot in text format is a lightweight and easy to setup solution (Just save it in VCS), and it could tell you the actual difference easily.
Isn’t it a method suitable for QML too?
QML — Snapshot Testing
To validate the assumption, I have started a project to bring snapshot testing in a text format to QML.
QML Snapshot Testing
Https://github.com/e-fever/snapshottesting
Unlike the Jest version, it doesn’t convert UI into an XML description, instead, it turns that into a QML like syntax. And it has a GUI “matcher”, which could provide more detailed information.
Installation and usage will not be covered in this article. You could follow the instructions in the README.md. Now, I will talk about how could it works with TDD.
Test Driven Development
According to the FAQ of Jest:
> Is it possible to apply test-driven development principles with snapshot testing?
Although it is possible to write snapshot files manually, that is usually not approachable. Snapshots help figuring out whether the output of the modules covered by tests is changed, rather than giving guidance to design the code in the first place.
Although they are negative from using TDD with Snapshot Testing, the snapshot testing in QML is not exactly the same as Jest.
The difference:
- QML Snapshot Testing offers a GUI interface. It provides not only the information of Diff, you can also browse the full version of Snapshot.
- At the same time, the target UI component is still working and able to interact with users. Developers could validate its behaviour manually.
With the difference between these two implementations, it is not impossible to carry out the TDD approach.
The fundamental principle of TDD is 1) Write Test code first 2) Red-Green-Refactor Cycle
I will use the following example to explain how Snapshot Testing works and fulfil the two points.
Mission: Write a component (CustomItem) to display “Ready?”. Once a user clicks on the text, it should be changed to “Go!”
Task 1: Write (Failing) Test code first
Let’s start writing the test case FIRST. And that is truly simple: (Full Source Code)
And create the target component, but only a blank implementation is needed at this moment. (“Item {} “).
Then…. it is done! We could start running the tests to get a “Red” light.
“Hey, do you missed the test case to validate the display text?”
I guess you may ask this question if you have read the test code above seriously.
Normally, it should have a validation statement like this :
compare(customItem.displayText, "Ready?")And the implementation of CustomItem should provide the display text:
(The `displayText` property is not necessary for the program. It is served for testing purpose. That is a bit wasted. )
However, it is not necessary for Snapshot Testing. The `matchStoredSnapshot` is a test function and would throw an exception if the test case not matched. Ofcoz, the exact testing criteria is still not ready yet. Will input it later (without changing the test program)
Let’s start the program execution.
var snapshot = SnapshotTesting.capture(customItem);The `capture()` function converts the instance of CustomItem into a text representation, then pass it to`matchStoredSnapshot` for comparison with the previously stored snapshot.
SnapshotTesting.matchStoredSnapshot("test_CustomItem_default", snapshot);
Since it is the first time running, it doesn’t have any stored snapshot yet. Therefore, `matchStoredSnapshot` will show a GUI dialog to ask for user confirmation. CustomItem is still a blank implement, the answer is “No” for sure. Then it will throw an exception to terminate the test case.
As we have written a failing test case, the first task is completed.
Task 2: Red-Green-Refactor Cycle
There have 2 tests within our test program. First of all, we made an implementation that could pass the first test.
And then run a test program, it is stopped at the same point in the previous step and show the matcher UI.
Since it could show the “Ready?” message. It has fulfilled the requirement. It should press “yes”
The program continues to run, simulate the mouse input, but the MouseArea does not implement any callback. Therefore, it will ignore the mouse click and still show “Ready?”.
Ofcoz, it should press “no” and let the test case fails.
Let’s implement the last feature:
Run the program again. Because the UI is not modified, the first Snapshot Test will pass directly, stop again in the second test.
This time is alright. You could press “yes” and commit the changes and push to the server.
Task 3: UI specification changes (F — !)
Requirement change is the major enemy of automated UI testing. You may cry if the changes break many test cases and a rewrite is needed…
Let’s start with minor changes — Modify the display text to uppercase.
CustomItem becomes:
(Note: In fact, it could be done by setting `font.capitalization` property. But it will be ignored in this article)
And then run the test program again. The matcher UI will be shown again and tell you the actual changes via the “diff”. Pressing “yes” to all the dialog will fix the test cases.
In such kind of small changes, even it would break the test cases, it is not necessary to modify any code to get it pass again. Simply run the program, evaluate the visual component manually, then press “yes” if it works. Then it updates the snapshots file automatically. This methodology could minimise the maintenance cost of automated UI testing.
Conclusion
When it is using TDD with QML Snapshot Testing, it would be different than the traditional approach. Such that you don’t need to the write the exact test criteria. Instead, you could verify the behaviour of UI manually. Once it is alright, the condition will be captured as a snapshot then save to a file. Next time it will compare the snapshot for you. Usually, the fields to be compared will be much more than you will actually write during development.
Ofcoz that means it will be easier to be broken during changes, but updates are not a difficult task. And it could discover unexpectedly changes easily. I think this methodology could highly reduce the cost in writing and maintain automated UI testing.
If you have any questions or comments, please feel free to leave a comment for discussion.
