10 tips for designing UserTesting.com tests

from a former UserTesting Product Designer

Full disclosure: I’m a stakeholder and alumnus of UserTesting.

To get the most out of this article, you’ll need a UserTesting account.

1. Start with a Story

Our brains are wired to process stories, so take full advantage of this when creating your usability test. Instead of asking participants to imagine or pretend, tell them a simple story that sets the stage for what they’ll be doing in your test.

A story-style introduction doesn’t need to be long-winded, for example:

Your boss, Head of Marketing at SportyWear, purchased a new tool to help you with your merchandising strategy and asked you to check it out.

2. Ask Interview Questions

Even if you’ve screened your participants, I’ve found that it pays to start every test with one or more interview questions. Asking brief, open-ended interview questions will often yield interesting insights that help you evaluate and analyze a participant’s feedback.

Furthermore, if you’ve already identified personas for your users, these questions can help you validate your ideas or place participants into groups. If you don’t have personas, interview questions can provide a head start for creating some.

3. Keep It Simple

Write simple tasks. The less that participants have to remember during each step of your usability test, the more they can focus on thinking aloud as they complete your tasks. If a task doesn’t fit within two lines of text, try simplifying your language or splitting complex tasks into two or more simple tasks.

4. Mix Open-Ended & Closed-Ended Tasks & Questions

Open-ended tasks and questions are great for inviting surprises and generating new ideas. By giving participants freedom, you can often get a glimpse into their intrinsic motivations while increasing opportunities for surprising insights worth their weight in UX research gold.

A simple open-ended task might be,

Identify your biggest question about this site/app, and then try to find the answer.

And an open-ended question could be something like,

What questions do you have about this site/app?

Closed-ended tasks and questions can help you uncover problems. To discover the biggest usability issues and problems, send participants through your business’ most important conversion funnels or needle-moving user flows.

For a great close-ended task, send participants through an end-to-end user flow. For example,

Shop for a protective case for your new iPhone and stop when you’re asked to enter payment information.

Closed-ended questions, like yes/no or rating scale questions, provide constraints that facilitate visualizing data in charts or graphs. These types of questions are particularly useful when looking for trends or patterns in usability issues. Here’s an example:

On a scale of 1 to 10, where 1 = Very Difficult, and 10 = Very Easy, how easy or difficult was it to understand the purpose of the site from the homepage?

Use a mix of open-ended and closed-ended tasks and questions to get a bigger bang for your buck with usability testing.

5. Let Participants Fill In the Blanks

Avoid referring to specific UI elements or marketing language. By using more generic language in your tasks and questions, you allow participants to fill in the blanks and uncover stumbling blocks in the user flows you’ve designed. If your tasks are overly specific (e.g., click on the blue gear icon in the top-right, next to “Your Account”), you’re probably missing out on valuable feedback.

6. Invite Questions

Get inside participants’ heads by inviting participants to ask questions during the test. I often do this with a task like, “Based on what you’ve seen, what questions do you have right now?”

7. Recruit Your Target Market

Unless you’re building something for an extremely niche audience (e.g., African pygmy hedgehog breeders with a degree in astrophysics), UserTesting makes it easy to recruit people from your target market. First, use the demographic options to specify general characteristics of your target market, like gender, age, income, and Internet aptitude. Then, tighten your net by adding specific requirements to your introduction. Lastly, include specific interview questions that will expose any participants who might have failed to read your requirements, so you can get those sessions replaced by UserTesting’s friendly support team.

If you have an Enterprise account with the ability to create a screener, avoid asking yes/no questions. Instead, use multiple choice questions with 5–7 response options. Even if you’re using a screener, I still suggest beginning every test with one or more interview questions to qualify participants and validate personas.

8. Get Your Study Peer-Reviewed

Ask a friend or colleague, preferably someone who’s not too close to your project, to check your test script for clarity, spelling errors, missing info., etc. Fixing sloppy mistakes is one of the easiest ways to set yourself up for a successful pilot test.

9. Run a Pilot Test

Even after your test has been peer-reviewed, it’s a good idea to run your test with a single participant to identify any other problems with the language, scope, or flow of your test.

After you address any significant issues with your pilot test, you’re ready to recruit a group of participants!

10. Recruit (at least) five participants per study

If you run a study with only one or two participants, research suggests that you’re missing out on ~30% of the usability issues that you’d expect to discover with 5 participants*. Moreover, with feedback from just a couple people, you run a greater risk of being distracted by accidental or outlier behavior.

However, that doesn’t necessarily mean you need to recruit all five users at once. Depending on your resources, you might consider employing the RITE method of prototyping practice where you iterate and make changes to your site or app between participant sessions.

* For more on this topic, read Jakob Nielsen’s article, Why You Only Need to Test with 5 Users.

I hope you’ve found this article to be clear, actionable, and helpful. If you have unanswered questions or feedback, please email me at adam@adamruf.com or tweet me @ux_adam.

Happy UserTesting! :-)