Guide to Conducting Usability Testing

Main Content：

1. What’s usability testing

2. Qualitative and quantitative usability testing

3. How to conduct an usability testing?

4. What to test?

5. How many test users in an usability test?

6. What to detect in a testing? While participants:

7. The severity of usability problems

8. Tips to usability testing

9. One last word

What’s usability testing

Usability testing is qualitative research methods to validate and test the design by observing an individual’s experience with an concrete prototype as he walks through the steps of a given task.

1. can users accomplish their goals with your product? (black and white definition)
2. how easily and successfully can users accomplish their goals with your product?(shades of grey definition)

Qualitative and quantitative usability testing

All usability-testing studies involve a participant performing some assigned tasks on one or more designs. There are, however, two types of data that can be collected in a user-testing study:

• Qualitative (qual) data, consisting of observational findings that identify design features easy or hard to use. Qualitative data offer a direct assessment of the usability of a system: researchers will observe participants struggle with specific UI elements and infer which aspects of the design are problematic and which work well. They can always ask participants followup questions and change the course of the study to get insights into the specific issue that the participant experiences. Then, based on their own UX knowledge and possibly on observing other participants encounter (or not) the same difficulty, researchers will determine whether the respective UI element is indeed poorly designed.
• Quantitative (quant) data, in form of one or more metrics (such as task completion rates or task times) that reflect whether the tasks were easy to perform. Quantitative data offer an indirect assessment of the usability of a design. They can be based on users’ performance on a given task (e.g., task-completion times, success rates, number of errors) or can reflect participants’ perception of usability (e.g., satisfaction ratings). Quantitative metrics are simply numbers, and as such, they can be hard to interpret in the absence of a reference point. For example, if 60% of the participants in a study were able to complete a task, is that good or bad? It’s hard to say in the absolute. That is why many quant studies usually aim not so much to describe the usability of a site, but rather to compare it with a known standard or with the usability of a competitor or a previous design.

How to conduct an usability testing?

Planning a usability study. Tests are designed around tasks and scenario that represent typical end-user goals. Tasks should be specific, concrete and reflect actual goals of the target audience. Scenarios contextualize the task, and are written to provide extra information necessary to complete that ask.

1. Goal
2. Participants
3. Tasks and scenarios.
4. Recruiting participants
5. Designing the prototype
6. Moderating test sessions
7. Analyzing the data
8. Communicating the results

What to test?

Usability can be defined by 5 quality components.

1. Learnability: How easy is it for users to accomplish basic tasks the first time they encounter the design? Are instructions clear? Are instructions necessary? Do labels make sense? Do certain words are understandable?
2. Discoverability: Are common items easy for new users to find?
3. Efficiency: Once users have learned the design, how quickly can they perform tasks?
4. Errors: How many errors do users make, how severe are these errors, and how easily can they recover from the errors?
5. Satisfaction: How pleasant is it to use the design?
6. Memorability: When users return to the design after a period of not using it, how easily can they reestablish proficiency?

How many test users in an usability test?

• Testing with 5 people lets you find almost as many usability problems as you’d find using many more test participants. With 5 users, you almost always get close to user testing’s maximum benefit-cost ratio. Aiming at collecting insights to drive your design, not numbers to impress people in PPT.
• For really low-overhead projects, it’s often optimal to test as few as 2 users per study. For some other projects, 8 users — or sometimes even more — might be better. For most projects, however, you should stay with the tried-and-true: 5 users per usability test.

What to detect in a testing? While participants:

1. understands the task but can’t complete it within a reasonable amount of time.
2. understands the goal, but has to try different approaches to complete the task.
3. gives up or resigns from the process.
4. completes a task, but not the task that was specified.
5. express surprise or delight.
6. express frustration, confusion, or blame themselves for not being able to complete the task.
7. asserts that something is wrong or doesn’t make sense
8. make suggestion for the interface or the flow of events.

The severity of usability problems

1. Impact: Will it be easy or difficult for the users to overcome?
2. Frequency: Is it common or rare
3. Persistent: Is it a one-time problem that users can overcome once they know about it or will users repeatedly be bothered by the problem?

Tips to usability testing

1. Planning the testing to focus on important questions about the design.
2. Recruit your participants in your target audiences.
3. Using scenario to develop user task.
4. Observing the test sessions. Focus on participant behavior and their rationale.
5. Understand the “how” and “why”.
6. Before answer participants’ question, think first why he is asking, is this a potential risk?
7. Test late enough in the process that there’s a concrete design to test, and early enough to allow adjustment in the design and implementation.
8. Ask participants to perform explicitly defined tasks while thinking aloud.
9. Share findings with stakeholders. Invite some to observe.
10. Open-ended questions.

One last word

Usability testing is, at its core, a means to evaluate, not create. It is not an alternative to interaction design, and it will never be the source of that great idea that makes a compelling product. Rather, it is a method to access the effectiveness of ideas you’ve already had and to smooth over the rough edges.