Usability Testing Lab image courtesy Willow Tree

How to Conduct a Usability Test with Users on Test Day

Based on Steve Krug’s Demo Usability Test Video

So, you finally completed the formative research phase of your product. You conducted user interviews, analyzed user questionnaires and survey data, evaluated product analytics, developed market segments for specific users, and created personas based on real data. From your established research findings, you then design and build a working prototype. Once completed, it’s now time to evaluate your product’s usability with real users.

You then begin preparing for this usability test by defining your research goals with your team and identifying key metrics to leverage in future research. After laying the groundwork for your upcoming study, you write compelling tasks, create consent forms, choose the right post-study questionnaire, and determine the best screen recording software to use. As you’re doing all of this prep work, you also send out recruitment emails to these targeted participants based on your formative research. Soon after preparing for this study, the day finally comes to test your participants. It’s Test Day!

So, now what?!

Steve Krug’s Demo Usability Test image courtesy of YouTube

1. Explain to Users Why You Need Them to Test Your Product

Goal: Make it clear to your users that their input when testing this product will improve its overall design and functionality. And this outcome can only be achieved by insights from real people like them using this product.

Methodologies

Assure users that you’re “Testing the site and not them” to ease any potential anxieties that they might experience before testing.

1. Inform users that you want them to think out loud during this testing session, and that their input is crucial for your team to capture poignant thoughts and feelings relating to the product.
2. Make it clear to users that you want them to be genuinely honest in their evaluation of this product, and that their feedback won’t hurt your feelings. The primary goal of this research is to improve the product for them.
3. Let users know that they can ask questions throughout the testing session. However, you should also explain to them that their questions might not be answered during the test. If they ask why, you can tell them that your primary objective is to see how real people would actually interact with your product without any outside influence.
4. Verbally ask users for their permission to record this testing session and ensure them that this recording will only be used for research purposes. If other colleagues are observing the test remotely, be transparent with your users and tell them that is the case.
5. Politely ask your users to sign a written consent form permitting your team to record them as they participate in your test. Be sure to communicate that the only people who’ll have access to this video will be those people working on the project, and that it’ll be used for research purposes only.

Developer watching videotape of usability test image courtesy of Interaction Design Foundation

2. Chat with Users to Prime Them for the Talk-Aloud Protocol

Goal: Make users feel comfortable to ease potential test anxiety. This technique helps users become comfortable vocalizing their thoughts and feelings before starting the test.

Methodologies

Pre-Interview Demographic Screen: Ask the users about themselves to collect some basic demographic information:

1. How old are they?
2. What’s their income level?
3. What’s their highest level of education?
4. What is their occupation?
5. What kind of work does their job entail?

Pre-Test for Experience, Expertise, and Brand Knowledge: This pre-test is done through either Moderated Interviews or Questionnaires. Ask users about their internet usage to gauge their levels of experience, expertise, and brand knowledge on your product or similar products:

1. How many hours do users spend per week using the internet (work, home, mobile)?
2. Where do they use the internet (work, home, mobile)?
3. When do they use the internet?
4. How do they use the internet (work, entertainment, shopping, learning, etc.)?
5. What kinds of sites do they visit?
6. Do users have favorite sites that they like to use?
7. What sites do they tend to avoid?
8. Within these sites, what features do they like or use the most?
9. What features do they dislike or never use?
10. What kinds of devices do they use?

By asking these kinds of questions, you could potentially validate some of the results in your formative research. Moreover, you’d then be able to determine whether or not you’re testing the appropriate users. These types of questions can help refine and hone personas to be more targeted in later design iterations.

3. Asking Users about Their First Impressions of Your Product

Goal: Ensure users understand the basic concept of your product based on a landing page’s content.

Methodologies

Invite users to make sense of the page by sharing their first impressions of the product:

1. What about the home page strikes users, either positively or negatively?
2. Who do users think this product is directed to?
3. What do users think they can do with this product?
4. What do users think this product is used for?
5. What kinds of things could users do on this site?

Mobile usability testing image courtesy of Keep It Usable

Advise users that they can scroll the landing page, but they can’t click on anything at the moment.

Remember, your aim is to test whether or not users fundamentally understand the purpose of the site. By having them tour the landing page, users can potentially help identify any glaring issues with your product’s content.

4. Giving Users Tasks to Complete

Goal: Test whether or not users can easily complete their objective from the given task.

Thinking Aloud image courtesy of Nielsen Norman Group

Methodologies

1. Read the task out loud to users and offer a written copy of the task for future reference.
2. Do not answer their questions about the product. They should be describing how they’re using the site to you without your influence. You should remain quiet in order to minimize potential researcher bias. Only when they are absolutely stuck can you intervene.
3. If users stop talking during the session, encourage them to share their thoughts by asking open-ended questions or clarifying questions. This technique will prompt them to continue explaining what they’re thinking and feeling as they navigate or use your product to complete the given task.

Pro Tip: Take notes as they complete these tasks. Here are some other resources on note-taking during a usability test.

Note-taking image courtesy of UX Planet

A special note about writing tasks: Tasks shouldn’t be written to explicitly match your product’s written content. They should represent real world use cases that people might encounter when actually using your product.

5. Post-Study Questionnaire

After completing the test, you can give them a post-study questionnaire to evaluate their thoughts and feelings on your product’s usability. By incorporating this tool after the test, you can quantify and aggregate data across users to get an overall understanding of product usability.

Likert-type scale used in the Post-Study Usability Questionnaire image courtesy Try My UI Blog

6. Compensate and Thank Users for Their Time

Once the test is complete, compensate your users or provide them with a timeline for compensation, if necessary. Finally, thank them for their time and escort them out of the testing area.

Next Steps

Once testing is complete, you can then begin evaluating the results of your usability study, which will then inform future design decisions.