When you write computer programs, it’s inevitable that errors are introduced, even for the most experienced developer. To resolve an error, the first step is often reading the error message that’s displayed in the console. But, research shows that programmers, especially novices, struggle to make sense of error messages, never mind take action on them.
Admittedly, when it comes to helping you, the developer, recover from errors, Flutter hasn’t done a great job. The console output from an error is typically verbose, and it’s often unclear how an error can be traced back to a particular place in your code. Lately, we’ve been working on addressing both problems. In this post, I’m going to introduce our first attempt at increasing the signal-to-noise ratio of Flutter’s runtime error messages, and describe the research we did to arrive at the current solution.
Introducing structured error messages
In the latest release of the Flutter plugin for IntelliJ/Android Studio and the extension for VS Code, we shipped a new feature that displays error messages in a rich, yet concise, format. The logging console can now display error messages with the following improvements:
- Highlighting the summary of the error in red
- Adding whitespace between sections to make the message more scannable
- Calling out the hint in the message, if available, for resolving the error
- Collapsing long lists and trees in the message
The following screenshot shows these four improvements, identified with blue circles, in the message that’s generated by the layout overflow error:
Note that everything fits in one screen, making it possible to get a general understanding of the problem at a glance. The reduction in noise is done without deleting any information that might be useful for cases where more detail is important in order to fix the error.
In contrast, the original error message is a lot denser and lacks structure, so it’s difficult to find the information that tells you how to fix the error. See the following screenshot.
At this time, we refactored about 85% of the error messages in the Flutter framework, to take advantage of this new presentation. Improvements to the rest of the error messages will happen over time. We also plan to refine the general presentation of error messages based on user feedback.
How did we get here?
The usability of error messages is a multifaceted problem. The content, structure, and presentation of an error message all contribute to your ability to make sense of it and fix the error. We know that error messages in Flutter generally provide helpful content, but the usefulness of the content is undermined by the overwhelming amount of information that you had to sift through.
As an example, let’s take a look at the “No Material widget found” error. This error occurs when a widget expects a Material widget in its ancestor tree but one isn’t found. The message for this error was so hard to figure out that a user sought help from StackOverflow. The accepted answer pointed out the key information already in the message, but the user either couldn’t find the needed information or didn’t understand it. No new information was given.
The new way of presenting error messages
The question posted on StackOverflow is revealing. It shows that not all information in an error message is equal. This question also inspired us to think about how we could make useful information more noticeable, which meant cutting down on visual clutter. Applying theories of visual perception and techniques in UX design, we came up with three variants for that error message: colors, spaces, and ellipses.
To create the colors variant, we analyzed the relative importance of different pieces of information in the message and came up with a simple color coding scheme:
- Display a one-line summary of the error in red.
- Display the object that the error is associated with in blue.
- Display the detail that isn’t usually needed in gray.
The next variant is called spaces. We added whitespace to the error message to separate it into sections. We also added section headings such as Explanation and Potential Fix to make the message easier to skim.
Last, the ellipses variant applies the progressive disclosure technique in error message presentation and leverages the capabilities of a modern IDE. For example, we collapsed the TextField widget’s parameter list and the widget’s ancestors after the first three are displayed. To see the rest of the list, just click the ellipsis that follows the last parameter or ancestor. The resulting error message is considerably shorter than the original, increasing the likelihood that your attention is drawn to the higher-level elements in the message, all at a glance.
We had good reason to believe that these variants would help the user, but we weren’t sure whether we could justify the cost to implement them. To make those presentations real, we needed to make the Flutter framework send structured error data to the IDE, so the IDE can style different parts (e.g., summary, detail, and hints) appropriately and collapse less important information. We asked ourselves if the potential improvement to the developer experience was worth the effort to refactor Flutter’s error API and the hundreds of errors that were already written. So, we decided to run an experiment, to find out how large the upside is of having structured error messages.
The experiment
To compare the usability of the three variants with the original message, we conducted an online experiment using a scenario-based questionnaire. We recruited 52 Flutter users, who were randomly assigned to one of four experimental groups. The original error message was shown to the control group, while the variants were shown to the three treatment groups, respectively. Participants were asked to describe what was wrong and how they would resolve the error in a very limited amount of time. The detailed study design can be found in the peer-reviewed paper that we published earlier this year at CHI 2019.
The results surprised us in a very positive way. Participants’ comprehension of the error message was substantially better in any of the three treatment groups than in the control group. In other words, the proportion of participants who correctly understood the error within the time limit was much higher when any of the variants were used rather than the original. The following chart shows that, the spaces variant outperforms the original formatting of the message by about 38 percentage points when participants were given 45 seconds in total to read the error message.
Similarly, all the variants of the error message outperformed the message’s original presentation when it came to figuring out the solution.
When participants were asked to compare the original error message and the variant they used in the experiment, they explained why they liked the variants better. The following are some examples. (Words within parentheses were added by the author of this article.)
“The error on the right (the ‘colors’ variant) is a huge improvement with the critical short message in red, the affected widget in blue.”
“Hiding the whole Widget object reduces clutter (in the ‘ellipses’ variant), makes it easy to find the reason the error occurred.”
“B (the ‘spaces’ variant) is much easier to parse. The sections are clearly broken up, so it is easy when skimming to know where to jump next.”
With strong evidence from the experiment, the we decided to invest in structured error messages. It was a long journey, but we’re glad that the refactored error API now provides a solid foundation for future innovations, to help developers recover from errors in their Flutter projects.
What’s the future for error messages?
For errors thrown by UI code, we could embed diagrams, animations, and even interactive widget trees in the message to maximize their explanatory power. We plan to start some of the more ambitious experiments in Dart DevTools’ console, because we aren’t constrained with the extensibility of IntelliJ and VS Code.
How can you help us be better?
Not all Flutter errors were refactored to take full advantage of this new structured presentation. We’re working on the errors that affect the most Flutter users first. You can help by letting us know which error messages could be more useful via our GitHub issue tracker, or you can suggest ideas about refining the error presentation in IntelliJ or VS Code.
Acknowledgement
The Flutter team’s former UX Research intern Kandarp Khandwala made an instrumental contribution to the research described in this article. We also thank all the Flutter users who participated in the experiment.
