The User Testing Fairy Tale: There Are Two Sides to Every Story
Written by Helio Cardoso & Tania Ramos
The Story
Once upon a time, there was a product manager with a technical background. Well, this is how I usually introduce myself, but the reality is that I am an engineer by heart who, throughout my professional experience, has been exposed to a lot of UX best practices and processes (research, wireframing, prototyping, capture feedback, iterate, rinse and repeat).
A long, long time ago, or to be honest, fairly recently, I was baffled by the gods of user-centered design, which brings us to the fairy tale I want to tell you today.
The Product Manager
Back in 2019, Experience Builder kicked off with a clear statement — help our customers deliver great mobile experiences with a native look and feel — okay, we can all relate to this motto, but how do we turn it into action?
We started by researching which mobile applications deliver the best overall experience, the ones that are considered the pinnacle of CX and that users always refer to after installing a new app — “This should work as application X.”
When talking about great experiences, one must speak of B2C applications. While CX has proven to be invaluable in other business segments, B2C apps have an undeniably wider reach and a more diverse audience, and as such, offer the most significant challenge.
We rolled up our sleeves and began designing a common B2C app. Once we were happy with it, we implemented it and released it in an early access program. The fairy godmother, however, didn’t seem very interested in granting us our wish.
The Villain: A Stalemate Problem
Despite all our research and app reviews by product design and UX experts, our application still received bad feedback or no feedback at all. The latter was even worse because it meant that people did not see its value or had better alternatives.
In sum, we weren’t adding anything special to this space. So what to do now? We couldn’t even understand the underlying problem since we only had anecdotal evidence. We were at a stalemate.
Luckily, the stars aligned and the gods of user-centered design decided to answer our prayers, and I was approached by a team member of our UX team with a proposal.
“We are setting up a fully dedicated team that will bring a new variety of user research methods to inform the design process. Initiatives like the Experience Builder can submit their outcomes to user testing to get insights about how our users interact and engage with those outcomes.”
Cost vs. value
My first thought was, “I don’t have time for this”; my second thought was, “I don’t have the budget for this.” I took a breath and started analyzing the proposal in my head, which only raised more questions.
I don’t fully grasp the methodologies used, the metrics retrieved — do they consist of qualitative metrics, quantitative metrics, or a mashup of both? How can I use these metrics to improve our product? How can I compare my outcome with others in the market to correctly say we are delivering great mobile experiences? How can I make my team trust these results and welcome them with an open mind?
At the end of this (very long) breath, one thing remained: we still had a problem to solve, and a path to a solution is better than no path at all, so we embarked on this journey with optimism, but also some skepticism!
The UX Researcher
I understand your skepticism. I really do. After all, this is a usual hurdle for the Experience Builder team and many other teams. While the importance of placing the user´s voice at the center of product development is widely recognized, reconciling it with the need to have continuous and fast releases can be quite challenging.
I am a UX researcher at the R&D department, and I am very aware that research can be useless if not done fast enough to impact the product. However, for it to be trustworthy, it also needs to be done rigorously.
In fact, I think that our goal is analogous to the OutSystems mission, as stated by our CEO Paulo Rosado in the NextStep event: we need to do it fast, but we also need to do it right! Our work with the Experience Builder team is a perfect illustration of just that.
The Hero: It’s All About the Experience
The mission of Experience Builder is to accelerate the development of apps with a great user experience. But how can we know we’re achieving this goal?
Collecting real users’ feedback from an early stage is vital to ensuring that the product is right on track. Identifying and correcting usability issues early on is a cost-effective strategy because it prevents errors that can be harder (and much more costly) to fix later.
When we started collaborating with the Experience Builder team, they were working on different app templates to accelerate the development of B2C applications. Both teams worked closely to test the usability of three of those templates: mobile banking, customer insurance, and patient healthcare.
I was happy that, despite the product manager’s initial concerns, the Experience Builder team and we were 100% aligned — we also worked closely with the product designer and the technical lead.
The Experience Builder team was bold and eager to try innovative processes and also curious about the results. While relentlessly keeping their focus on delivering, they ensured that we had all the conditions to run the tests. So, we were already starting from an advantage point: the one that comes from courage and trust.
Together, we ran several usability tests with real users: moderated tests, unmoderated tests, and we even took to the streets for some guerrilla user testing.
Specifically, users (our heroes) tested application prototypes and were asked to complete tasks that are part of typical use, such as making a payment, filing an insurance claim, or booking a medical appointment.
During the tests, the users verbalized their thoughts and feelings, and we recorded their interactions (except in the guerrilla testing). In addition, we also collected several quantitative usability metrics, such as task success and error, difficulty in task completion, efficacy (i.e., time for task completion), satisfaction, system usability score (SUS), among others.
The Universal Lesson
For both teams, the results were great and insightful. On the one hand, they helped us understand what was working well. For example, most users liked the simple and clean look of the applications. They loved how easy it was to take a photo and include it in their insurance claim and were also delighted to manage both their health issues and their children’s or elderly parents’ in the same mobile app. Check, check, check. Great!
“One of my favorite things was the ease of filling the car claim and selecting to upload a picture associated with the car location.” (User 5, United States)
But, more importantly, the results uncovered usability problems that would otherwise remain unnoticed. We are not our users, and it is very hard for us to share their perspectives.
We know that this is especially the case for the products we create. We tend to overvalue our own creations and believe that others will share our views. Listening to our users is thus not only desirable — it is a reality check.
The users’ feedback was indeed revealing to us. We learned that the apps had several usability problems, including security, findability, and efficiency issues.
For example, users wanted to have more feedback and feel more confident after critical actions (such as bank transfers); they were frustrated when they couldn’t easily find information in the apps (such as their family’s profiles; see fig. 3), and they wanted more immediate ways to perform certain behaviors (including “filing a claim”).
Well, not checked, not checked, not checked. But now we had objective data about what “was working” and what “wasn’t working,” and we could learn and quickly make adjustments. Also great!
“I’m not finding my father´s profile. I have looked everywhere I can think of in the dashboard…this is very frustrating.” (User 14, United States)
The Fairy Godmother: Iterative Testing
Of course, we couldn’t stop there. So, we decided to swing our magic wands in the air. Meaning: we went for a second round of testing after tweaking the experience. Fairy dust is hard work, we assure you.
After fixes were made to resolve the problems in the first stage, we ran a second round of tests to examine if the changes had improved the experience. We were also interested in understanding if the fixes had brought new usability problems.
Although this type of usability iterative testing is considered a good practice by many UX professionals, it is often not implemented due to budget and time constraints.
We started the second round of testing in the healthcare app. To allow comparisons across tests, we kept the tasks as similar as possible and tracked the same metrics. Results showed a substantial improvement in usability, which was consistent across all metrics.
For example, the task overall success rate went up from 58 percent to 76 percent, while the percentage of “satisfied” or “very satisfied” users increased from 41 percent to 84 percent. The system usability score climbed from 69.3 (a value in the “OK” range) to 90 (a value between “excellent” and “best imaginable”). Also, the number of usability problems identified was lower and the ones identified were much less severe.
In sum, with just two rounds of usability tests, we improved the users’ experience and satisfaction substantially. This chapter of the story is never-ending: we will keep iterating, testing, and improving. We research fast, but we also do it right. And, more importantly, we do it together. This is our way. This is the OutSystems way.
The Happy Ending
So, all together now: while product managers can’t work under the premise that they fully understand all their customer end-users, doing this kind of research consumes time, effort, and costs money — true story — but the outcomes are invaluable.
We delivered a product that is easier to understand (+25%), we more than doubled users’ satisfaction (+43%), and went from an okay product to an excellent product. What’s the price tag on that?
This vital message of satisfaction transpired in the feedback received: “It is user-intuitive, easy to navigate, simple to understand, and it has just the right type of information I would need,” mentioned one of the users.
After trying out one of the apps, a UX expert designer commented, “This is an enormous boost for someone that needs to make something like this.” Such feedback from real users puts a smile on our faces!
From a strategic point of view, you can’t release an okay product in today’s world because there are thousands of okay products. If you want to become or stay competitive in the mobile market segment, you have to push your limits, know your end-users, and deliver your promises to acquire and retain a loyal customer base.
With Experience Builder, we are making our customers leverage all this knowledge to build their applications, to deliver memorable UX experiences to their users, and don’t take our word for it: the results are in plain sight!
In a matter of minutes, you can build your own app by dragging and connecting several industry-specific flows and common flows (e.g., authentication/authorization) while enjoying the peace of mind that comes from making an app your users will love.
The end.
The Authors
