10 Usability Testing Pro-Tips

A few weeks ago, a colleague scheduled a meeting with me asking for usability testing pro-tips. I would not say I am a pro, but I did have some advice. Here are my “pro”-tips:

1. Define the research goals

In my opinion, usability testing is very good at answering questions, such as can a user do X, describing how they try to achieve X and the reasons why. It has been my experience that some people will approach a qualitative researcher and say, “we need to run a usability test on X”. Unfortunately, some view usability testing is synonymous with qualitative research. However, if you wished to understand teens attitudes to X or compare design A with design B an in-depth interview/focus group or lab-based/live experiment might be more appropriate. The thing to remember here is that the goals/questions define the method.

2. Do you even need to run a usability test?

One of the most important lessons I was taught by my mentors at Microsoft was when to say “no”. The determining factor of this question is if you can have impact. I have a few rules to determine if I can have impact:

• Is there time to make feature or product changes before launch? If the launch date is in a couple of weeks or less there is probably not time to make changes. Beware of the retaliatory response, “we will fix it in V2". In my experience, there will be other priorities; and after launch other feedback channels (e.g. app store reviews, in-app feedback, usage data, etc…) are often more effective.
• Do stakeholder just want to confirm what they already believe to be true? Usability testing should not be done to tell us something we already know or believe to be true. Similarly, usability testing should not be used to settle internal arguments. As such, make sure you are doing usability testing for the right reasons — ultimately, the good of the product and the people who use it.

3. Understand the feature or product that you’re testing

To do this, you need to embed yourself within the feature and product team and meet with:

• The product managers to understand the feature or product, as well as its purpose and the vision
• The engineers to understand bugs with the current build or prototype, as well as technical limitations
• The designer to understand design decisions, as well as differences between the design of the current build or prototype and the design vision.

When you are evaluating the feature or product you will need this information to “guide” the participants as appropriate and interpret the results effectively.

4. Think in terms of goals, not tasks

This is a subtle, but important difference. The people who use the features and products we develop think in terms of goals they wish to achieve, not the tasks to achieve them. For example, if evaluating a new search engine or a feature within, you would not ask someone to “search for the rules of chess”. Instead, you may ask them to “find a website that explained the rules of chess”. The difference here is the ends, rather than the means. While you may be interested in a particular sub-task, you should keep the participant focused on the end-goal. Hence, it is important for you to know the various tasks to achieve a goal.

5. Create a template for quick, structured note taking

When I have a script for my study in place I create a notes template based on the script. For this I use a spreadsheet where each column represents a task or question and each row represents a participant. Below is an example of such a template.

In the appropriate cell I mark either a S, P or U, which signifies successful, prompt and unsuccessful, respectfully — more on this later (see 8). This makes it easier to tally task success. I then include a short notation for actions taken. For example, selecting the Omnibox in Chrome, typing a search query and then selecting a link would be annotated as, SOB, TURL, SL. This makes it easy to understand what participants did. Finally, I include other notes and quotes. My personal notes start with a hyphen; and participant quotes are surrounded in quotation marks. If I need to follow-up on something I include actions in capital letters (e.g. SEE VIDEO).

6. Be friendly

Participating in a lab-based study can be intimidating — you are in a strange place, talking to someone you don't know and being asked to perform tasks while thinking aloud in some cases. As such, you have to make the participant as comfortable and relaxed as possible. To do this, when I meet a participant in the lobby I introduce myself and offer them a drink — better still, be proactive and give them a drink, such as a bottle of water. On the way to the lab, make small talk. Ask them if they have been to company X before, if they have participated in a research study before and if they have any plans after. It is also wise to ask if they need to use the restroom before the study starts, as it is best to avoid interrupts during the study.

Once in the lab you are ready to start. Over the years I have crafted an introduction that has served me well.

Welcome to <company />. Once again my name is <name /> and I am a researcher on the <team /> team. Part of my job is to understand more about the people who use our apps or may use our upcoming products and work with our product team to design better products for people such as yourself. To do this I will ask you to perform a number of tasks during which I will observe how you achieve these tasks and ask you questions about actions you take. It would also be great if you can think aloud as you are performing these tasks — any thoughts, areas of confusion, improvement, etc... that come to mind. I want to emphasize that this is not a test. If you are unable to do something, it is not your fault, it is ours. This is an opportunity for us to get a reality check on our thinking; and for you to have an impact on the product we may release and people like you will ultimately use. As such, any feedback you have whether positive, negative or perhaps more importantly areas for improvement are greatly appreciated.
Today I would like to talk you about <area /> and then show you something we have been working on and get your feedback.
As we are going to be getting lots of great feedback and insights from you today, is it OK if we record this session? We are capturing <cameras /> and the recording will only be accessible to employees at <company /> and will not be passed on to any third-party. I am also going to be taking some notes. Please bear with me if there is a pause in our conversation. I will either be typing notes or thinking of what to do next. Do you have any questions before we start?

7. This, that and what they say

When talking to participants you want to use their language, as they do not use the same jargon companies do internally. For example, the Omnibox in Chrome is often referred to as the address bar; and the Composer in Facebook is often referred to as the status bar. As such, point to things and use terms, such as this and that to reference. Once you understand how the participant refers to things you can start using their language.

8. Everyone deserves a second chance

Just because someone is not able to do something the first time, does not mean they are unable to do it at all. First you need to understand if they believe they have successfully completed the task. Upon establishing that they have not achieved the intended end-state, ask them what else they may do. If they are still unable to achieve the task or get stuck, give them a hint that will take them one step closer towards achieving their goal. For example, this might require selecting a button or menu — “what about if you were to select this button, what would you do from here?”. If the person is able to complete the task from here, I mark completion as prompted (see 5). If the person requires further prompts till completing the task I mark completion as unsuccessful. This gives you some indication as to the failure points in achieving some goal.

9. Learn to shut up and listen

If you are talking more than the participant, something is wrong. If you do need to talk, you should ask questions to get the participant talking, such as:

• “What are you thinking at the moment?”
• “How would you describe this?”
• “Compared to <X />, this is…”

Furthermore, be like child and ask “why” followed by more “why” questions when appropriate. Do not assume that you understand the participant’s point of view. Similarly, do not make inferences, such as, “Is this confusing?”. That is just leading.

10. Don't ask people if they like the feature or product

A lab-based study is not the appropriate place to ask people if they like a particular feature or product. In many cases they will be visiting company X developing feature or product Y, they will be meeting with a representative of that company and it is more than likely that you are paying them to be there. Of course they are going to like it. A better approach is to show alternative designs and ask participants as to state their preference. Most importantly, ask the reason for their preference. This will better get at the underlying reason for liking something.

I hope these “pro”-tips are applicable and insightful. If you wish to read tips from a real pro, I recommend reading Michael Margolis’s Fast Design Co. article, Google Ventures On 8 Shortcuts For Better, Faster Design Research. Happy researching!