How to measure user experience in your next usability test

Qualitative data collected in user research is subjective and often chaotic, making generating measurable and reliable insights difficult. At Kaltura, the product design team has been facing this challenge. Emotions, facial expressions, and thoughts collected in qualitative research are hard to measure, and the raw data is easily left open to interpretation.

To overcome this issue, our team adopted an approach in which we applied quantitative data points to measure, compare and track user experience within the context of a qualitative user test.

Our insights became more organized, measurable, and objective. That new direction helped us build trust with our stakeholders. We were able to share compelling visualizations and draw more attention to our findings. Plus, comparing design solutions with this approach enabled us to base our design decisions with confidence.

In this post, I will outline the process we developed for integrating UX metrics into our qualitative user research. Metrics can be used in many types of user research.

For the purpose of this post, I will focus on usability tests. These consist of evaluating a product with representative users that try to complete tasks, answer questions, and report feelings and thoughts out loud. Participants of the study are typically sending a recording of their screen, audio, and camera.

Now that we’ve set the stage, let’s dive into our process.

Step 1: Define our study goals

Planning research starts with our endgame. What questions do we want our study to answer? It’s important for us to involve many stakeholders in this step, such as product owners and UX designers, to understand our business goals.

At this stage, we discuss parts of the user experience that impact the product’s success.

Here are a few examples:

• We want to measure the discoverability of a specific feature.
• We want to rate the performance of a user completing a critical task.
• We want to evaluate whether users find it easy or delightful to use our product.

Step 2: Identify the right metrics that provide the best insight

Once we define the business questions we want to answer, we are ready to look for the right metrics. There are countless UX metrics; I will review the ones we have found to be the most valuable.

Performance metrics
These metrics evaluate how efficiently users execute tasks. Note that performance metrics do not measure user satisfaction. They are useful for spotting areas where we might find usability issues.

Here are the main ones we used in a recent test:

1. Task success: This metric shows whether the user completed the task successfully or not. Task success helped us measure the rate of successes and failures in a particular user flow. This metric can be binary, the user needs to fully complete the task or fail, or it can measure levels of success. Example of a binary task success: Did the user submit the registration form?

2. Time on task: This metric measures how long a user takes to complete a task. Time on task helps measure how effective your product is. An example of a question to measure time on task is: How much time did the user spend filling out the registration form?

3. Errors: How many errors occurred during the session or task completion? Errors help us surface usability issues. When you label errors, you can report slightly different ones into one. For example: “date input error” can correspond to each time a user made a mistake while trying to input a date.

Satisfaction metrics
Measuring satisfaction reveals how users perceive the product and highlight how valuable they find it. Surprisingly, satisfaction does not always correlate with performance, as users can easily complete a task without necessarily finding the feature useful.

Here are the main ones:

1. Rating scales: These are self-reporting metrics where we ask users to rate their experience. You can ask users to rate their level of agreement with a statement in a closed list of answers. For example:
“This feature will help me manage my daily tasks.”
Strongly disagree — Agree — Strongly agree
Another approach is to ask a question and present levels between opposite adjectives. For example:
“How valuable did you find this feature to manage your daily tasks?”
Unhelpful 0–0–0 Very helpful

2. Emotions: Measuring emotions can sound like a hard goal because it might be the most subjective, yet there are many ways to achieve it. We approach this by reporting emotion in a set of predefined tags — for example, surprise/joy/frustration/disappointment/confidence/confusion. The researcher reports one of these emotions each time they arise during a user test.

Step 3: Integrate metrics into a user test scenario

After we picked the right metrics, we need to write our user test scenario in a way that allows us to collect the data we are aiming for. Writing a scenario is challenging because we want to avoid biases that could skew our data.

When incorporating metrics in our scenario, there are considerations we need to keep in mind. Here are a few examples:

1. Aiming to collect a binary task success metric requires an unambiguous way to measure it. The outcome can be either success or failure. Therefore, the task should require from our user a clear expected outcome. For example: “Did the user successfully log in to the application?” In this example, we depend on the task’s success for a specific event.

2. While defining a rating scale to measure satisfaction, suggesting an odd number of levels allows the user to choose a neutral answer. Having an even number constrains our users from taking a stance.

An example of an unmoderated usability test scenario we use at Kaltura.

Step 4: Collect, analyze, and present user test results

As mentioned earlier, the raw data typically comes in the form of a recording for each tester. At Kaltura, we use Dovetail, a customer insight platform, to gather and analyze usability tests. It eases the process with automatic transcription, video clips, and tagging.

Preparing the review
1. The report sheet: Before we start viewing and analyzing each recording, we need to create a report sheet that allows the reviewers to take notes, report the metrics, and tag parts of the recording.

2. Tagging system: We need to set a predefined tagging system that reports data points corresponding to the metrics we defined earlier. For example, if we wanted to measure a range of emotions, we need to specify them as tags to enable reviewers to report them. Keep in mind that you might uncover issues from the raw data that needed to be defined in the planning stage. Therefore, we recommend creating a set of generic usability issue tags such as Navigation, Input errors, Readability, Misinterpretation, Misleading, etc.

Analyzing the raw data
This is what we call the “popcorn” phase. You can grab any snack while watching the recordings, but it’s important also to be equipped with your report sheet to tag, take notes and report the metrics. Dovetail allows us to easily create clips of interesting moments that can later be used in our presentation.

Presenting the data
Before we start documenting and sharing our findings, we need to make sure we gathered all the data points, and that we are able to manipulate them globally. We want to calculate different measures from our data.

Here are a few examples:

1. Task success rate: From a success metric, we can calculate a task’s success rate by calculating the mean. This measure is particularly interesting for comparing the success rate between design iterations.

2. Most recurring issues: For issue-based metrics that rely on a tag system, we can calculate the percentage of occurrences for each error type.

3. Average time to task: This lets us calculate the average time to complete any task.

Analyzing the data allows us to generate engaging visual representations that support the study’s conclusions. Our ultimate goal is to produce documents that effectively inform our stakeholders about the study’s objectives and provide them with valuable insights.

Graph representing the success rate for each step of the user flow.

Graph of the pain points tags reported during the usability test review

Impacts of measuring user experience

Introducing this methodology made a significant impact on our research practice.

Firstly, it led to more actionable results by clarifying the areas that needed improvement. It also helped us stay focused on our goals because they were streamlined throughout the entire process.

Secondly, the approach brought continuity to our studies, as the introduction of standardized measures enabled us to track progress over time.

Lastly, this new approach helped us to grow stakeholder engagement in our user research efforts.

Thanks for reading! If you have any questions or comments, please feel free to share them, and I will be more than happy to help.