4 Easy Behavior and Perception Metrics in Usability Testing

I love conducting usability tests. Nothing is more humbling than getting my design in front of users. But from doing hundreds of usability tests and refining my script, capturing and analyzing 4 metrics per task based on what users say and do makes for quick and actionable insights. These metrics are suited both for moderated and unmoderated testing.

During the Test

When testing, I include the following 4 questions after each task prompt. When there’s both a moderator and notetaker, both will fill out the following criteria below after watching the task.

Behavior Metrics (Moderator fills these out based on observation)

1. Did the user successfully complete the task?

2. How confident was the user during the task?

Perception Metrics (Moderator asks the user the following questions)

3. Did you successfully complete the task?

4. On a scale from 1 to 5, 5 being easiest, how easy was the task?

After the Test

Analysis becomes really simple now with having limited quantitative measures. (I won’t get into why I didn’t include time on task or other metrics in this article.) Now it’s time for me to process the data and report back results. My next steps are to:

• Meet up with the notetaker to discuss how we each rated the behavioral metrics. Discuss if we rated any of the metrics differently, why, and then choose which rating best fits what was observed.
• Create a table per task with a breakdown of the ratings (see below). Highlight any participants that have mismatched behavior and perception values. For instance, if a user believes they didn’t finish a task, but they really did, highlight that. Similarly, if the user rated the task as easy, but they struggled, highlight that.
• Calculate each task’s overall scores. Once I’ve enter each participant, I have a final row to show the percent of observed or perceived success, confidence, and average of easy of use ratings.
• I then include an “Overall Performance” rating, using a scale of Great, Good, Okay, and Poor. At the highest level, how did the design perform?

In these results, participants 2 and 4 had mismatched experiences. Also, the 4th participant rated the ease of the task very low. This, to me, suggests looking into the details around how users were misled, and why users felt the task was difficult despite a high success rating.

Incorporating Metrics into Usability Findings

This breakdown per task becomes a really nice way for executives and stakeholders without much time to see the overall performance of users through the tasks. Because I don’t always have the luxury or time to process a lot of quantitative data, like time on task, this gives the stakeholders confidence that this isn’t just feedback but observed, measurable data.

I use these metrics to also discuss what caused the success and failures of the tasks.

• What parts of the design are working well, and thus, ensured users got through the task?
• What recommendations do I have for changes to increase the success rate?
• Why did users lack confidence? How can changes to the design boost their confidence?
• What misled users through the tasks?

While there’s a lot that could still be captured, these have become my go-to metrics for quick usability testing and analysis. It’s a language that is becoming more universal on teams I work with, too, so they know quickly how to interpret the results of the data, understand how it’s collected, and can expect how I’m going to discuss the details of recommendations that are to come.

Why Behavior and Perception?

I get this question from a lot of folks, stakeholders and UXers out there, as they think either one or the other is necessary to make design decisions. It’s all based in psychology — what we believe happened and what actually happened doesn’t always match up. Of course we want to know where there are failures in task performance, but we also need to know when users walk away feeling like they failed. (After all, doesn’t that still classify as a failure?) What happens when a user doesn’t believe they completed a task when they actually did? Users have to double check their work or are left uncertain and hoping everything works out. Did I really submit my timesheet? Email my doctor? Submit my homework? So measuring both behavior and perception helps for me to ensure that users not only are able to perform the task, but feel certain in their actions.

I’d love to hear what other UXers have done to also quickly measure and process usability data, particularly when other metrics like time on task are unable to be brought into testing.