The customer is always right: user observation is an engineering superpower
It’s a meme borne of frustration, but when it comes to how a customer or user experiences a product, they cannot be wrong. It’s a tenet of user experience (UX) design that one should pay attention to what a user is doing and thinking when using a website or app. This applies even for specialist users, like data scientists or software engineers using a library. By carefully observing how a product is used in practice and understanding the cause of even the smallest mistake, an engineer is empowered to make dramatic improvements to its usability.
The StellarGraph library is part of a project of multidisciplinary experts, and the developers of the library apply this UX-focused thinking. The library is more usable, practical and has fewer avenues for common mistakes and problems because of this. We try to be as helpful as possible when a user has a problem, even if it’s something “trivial”, both by providing an immediate solution, and by thinking about how to stop the problem happening again!
StellarGraph is an open source library for graph machine learning. A graph is a collection of nodes and the edges between them, where the edges represent some connection or relationship between the nodes. Graph machine learning takes conventional machine learning models like multilayer perceptron and long short-term memory that apply to single data points (representing individual nodes) and augments them to use the information provided by each data point’s connections.
The approach
The StellarGraph team takes two simple approaches to user observation:
- User feedback: indirect observation by paying attention to the issues and questions people have.
- Usability testing: directly observing individuals actually using the library.
Both of these approaches are driven by taking raw user experiences and then sifting down to a small number of key insights and actions.
Indirect observation
Issues reported by users are a treasure trove of hints about deeper underlying problems. For StellarGraph, we went through the issues helpfully filed by people using the library and extracted some key observations.
The process was not complicated:
- Find issues from outside the team (we add an “external” label to these issues as they are filed).
- Add a row for each issue to a spreadsheet and fill out columns with information like the kind of task(s) being solved, the library functionality being used, and the kind of request: is it a new feature? A bug? Something already supported and working, but not able to be found or understood by the user?
- After filling out the spreadsheet for each issue, look for common trends, such as common categories of issues.
For StellarGraph, this process immediately flagged a few “procedural” issues that occurred often. In some contexts, these might be considered invalid or “not an issue”, but we agreed they represented problems that we could put some effort into solving. Two of the most common are:
- Version mismatches, where a user was running a demo notebook from one version with a different version of the library. Before the 1.0 release, breaking changes between versions were common, and so this mismatch resulted in confusing errors. We filed #1172 and addressed it in #1242, by adding a version check to each notebook.
- Failure to find an appropriate demo notebook, where the user didn’t manage to find one of the many demos for a task already solved by StellarGraph. We filed #1087, and progressively addressed it in #1200 and beyond, by creating clearer tables and indexing to highlight which demos are available (current version).
With this form of indirect observation, properly validating any changes is hard: we can only pay careful attention and notice whether fewer issues for each class of problem are filed. For StellarGraph, we try to release every month, to ensure users get the latest polish quickly.
Direct observation
Watching users using the product live is more effective than indirect observation. It’s more labour intensive, and more effort; but it’s much easier to notice and understand problems.
We took this approach for StellarGraph — via the “dogfooding” method — just before the library’s 1.0 release by watching colleagues from other projects use the library. This process was also not complicated:
- Create some problem sets that focus on relevant areas of the library to be tested (this is optional, if users have existing problems or tasks using the product), and let each person choose a task.
- Have someone watch each person write code using the library to solve the task, transcribing the work and any discussion as verbatim as possible (only for people who were comfortable with this).
- Otherwise, each person can take notes about confusing things, as well as discuss occasionally during the task, and then walk through the resulting code.
- After collecting all the transcriptions, notes and final code, look for common trends.
What’s most critical here is that the StellarGraph library team were just observing: not helping. By helping, explaining or defending something we may have enabled the user to solve the specific problem more easily, but we would also be contaminating the insights with our expertise and background. By observing, we got a more accurate idea of the experience of the user; someone who won’t have spent months writing the library itself, and also won’t have live access to those developers to ask questions.
We did help when someone got completely stuck, but only after understanding and writing down how and why they were stuck.
This process is fundamentally designed to test the library, not the individuals using it. If a user struggles to complete something, that represents a problem in the library, not a failure of the user. Humans are wired to feel bad if they’re unable to do something, so reassurance and a good rapport between observer and user was helpful.
If a user struggles to complete something, that represents a problem in the library, not a failure of the user.
Similar to the indirect observation, we turned the common threads into a variety of issues and worked through the most important ones before the 1.0.0 release, giving it an extra piece of polish.
Finding the nuggets of gold
How did we take the spreadsheet of issues or dogfooding notes and extract the core problems? You may see a pattern forming; this process was straightforward too.
In UX terms, we used a combination of analysis and synthesis:
- Start a spreadsheet with columns “Summary” and “Description”.
- Read through the dogfooding nodes (or spreadsheet of issues) and summarise each distinct problem into the Summary column. Quotes and more information should be included in the Description column, in order to understand the underlying cause or mental model that led to the problem. In many cases, more than one user will encounter a particular problem, and so the summary and description can be tweaked and expanded as one reads through the raw notes. We also summarised the nice experiences people shared, so this wasn’t just a long list of negative things!
- Create issues with actions to address the underlying problems.
After doing this process for both indirect and direct observation, we had a reasonable idea of the many good and bad aspects of the library. We try to improve on the good aspects, and make changes to stop or minimise the bad.
In summary —
In applying a few straightforward UX processes to our development, the team of software engineers and researchers working on the StellarGraph library gained superpowers. We could streamline the library, making it more usable and minimising common mistakes. We avoided blaming users, and recognised that even “silly” questions can represent an underlying issue that can be fixed.
You’ll see this super-powered polish in StellarGraph version 1.1, released this week. Get started with graph machine learning with pip install stellargraph, and following the node classification with GCN demo.
The UX approach here is imitating StellarGraph’s UX experts Georgina Ibarra and Nhung Nguyen. This post and the observation described was also partially inspired by a tweet thread.
This work is supported by CSIRO’s Data61, Australia’s leading digital research network.
