Usability Testing: When Things Don’t Go the Way You Expect
What I learned as a UX Designer from an intensive month of it
Prior to the call, I got early this year to do a post-launch usability study for a HR and benefits mobile application, I had never done a usability test on something I did not design. Neither had I done one on that scale.
The app had quite a few functions to test, and the organization using it has tens of thousands of active users, as well as the means to recruit quite a number of them for the study. I was in charge of the UX side of things and would be working with a UI designer. It promised to be a great learning experience.
The travails in the details
Indeed, I soon discovered that there were details that may not have mattered as much in a smaller study, but I had to agonize over this study. For example, how should the tasks that users perform on the app be broken down?
If the scope of the app were smaller, it might be enough to simply identify user objectives and have each objective labeled as a task — things like “Book the earliest flight to Amsterdam” or “Order a skinny cappuccino to go”. But this app has a variety of different processes that allow users to do different things, like take a sick leave, apply for childcare leave, approve employee leave requests, and claim transport costs amongst other things.
To be or not to be practical
We also had about a month to recruit, test, iterate on the designs, test again, and finalize the designs. In addition, there were 3 different user personas to account for. Due to the time constraint and the complexity of the app, I could only afford to spend an hour on each tester to ensure that they will not wander and get lost trying to accomplish each task.
Out of necessity, I broke down each objective into a few significant steps and more-or-less designated each of those steps as a task. Did this make it easier for the testers to navigate the app and accomplish the tasks? Probably. Ultimately, I had to prioritize practicality given the time constraint. And, as you will see, it worked out in the end.
Is it the UI’s fault?
The app had been accumulating low ratings on the Google and Apple app stores by the time I conducted the study, so it was logical to expect testers to give similarly low scores during the test.
The client’s objective for the usability study was to investigate UI issues due to the assumption that the design must be a big part of the app’s problems, resulting in the low ratings. However, I quickly discovered that the app’s negative UX stemmed mostly from technical issues originating from the back end, issues that were not present in the testing environment.
When the app’s UI lacked polish and presented the testers with problems, they were quite willing to overlook the flaws that seemed minor in comparison to the technical problems. As a result, the usability scores they gave were much higher than the app’s ratings.
How the testing helped us redesign the app more accurately…
One of the important lessons I learned about usability testing is that there needs to be some method to score ease of use, and while the scoring system is ultimately arbitrary, it must be consistently applied across all tasks and test sessions.
In this case, I asked users to rate the ease of doing each task on a 5-point scale, with 1 being ‘very difficult’ and 5 being ‘very easy’. What score was given to a task was determined entirely by the tester’s subjective perception, so this is certainly not an objective measurement.
However, the insights don’t come from the absolute scores, but from the variations in scores between different tasks. And that is how you can tell which tasks are not as easy as they could perhaps be.
…and win over the client
After you know which tasks are relatively difficult, you can investigate by asking the testers clever questions to find out where the issues might lie.
So even if the testers are being generous, even if it’s easier to do the tasks within the testing environment, you can still derive actionable insights through a consistent scoring system and good follow-up questions. And, just as importantly, you can obtain evidence to back up your design solutions, as well as meat for a compelling story to get the client’s buy-in.
And this brings me to the next part: The resolution and happy ending.
Moving forward after the testing
After identifying users’ pain points, we had a good sense of how to redesign the app. We could then proceed with wireframing and prototyping, ahead of the second round of usability testing, which would validate our solutions.
The firm sense of direction we got was useful because we had just about a week to work on the redesign of the app.
Every significant change we made to the user flow and the layout of the screens can be explained by the results of the usability study and the feedback given by the testers. This really helped us when presenting the changes to the client, and we were able to avoid a situation where the client disagreed with the design direction.
And when the time came around for us to conduct the second round of usability testing, we were able to verify that the redesign had helped to improve the app’s UX.
The usability scores given by the testers had improved for tasks that were previously deemed ‘more difficult’.
Again, scoring was subjective and scores fluctuate from tester to tester. What I looked out for when analyzing the results were significant variations that could tell us whether a task was comparatively better or worse than the previous design.
Of course, there were still things to improve on after that. I received more feedback, which informed us of places where we could make further tweaks to improve the UX. And that brought us closer to making the app the best it could be within a one-month timeline.
Tying up loose ends
What about the technical issues that have had a major impact on the UX of the existing app? As designers, we do the best we can to mitigate such issues through design. Aside from that, it’s a matter of investigating and providing clarity on the challenges users are facing.
During testing, I had obtained specific feedback from the testers on the issues they faced as users of the actual app. For this project, the client had their own development team, so I passed the feedback on to them. This way, the team would know exactly what they needed to work on.
Conclusion
So it always pays to test an app and to speak directly to users about their experience. This, in my view, is the bread-and-butter of UX design, and my time on this project only reinforced my belief that design is only as good as the intelligence that drives it.
2359 blog features articles ranging from UIUX design pieces to technological know-how and industry trends. Read more on our design articles here:
