7 important tips for moderating a usability test

Improve your moderation skills for better usability testing results

Ask somebody what the UX team at Govtech does and a likely answer will be usability testing (UT). At GovTech, we run a lot of usability testing sessions for partner agencies. I am fortunate to be given the chance to conduct many of these sessions and play a part in the creation of many citizen centric products.

There are various types of usability tests but I will be focusing on the in-person-moderated usability test in this article.

The moderator plays a pivotal role that involves building a partnership with the participant. Most research findings are derived by observing the participant’s actions and gathering verbal feedback. To be an effective moderator, you need to be able to make quick decisions while simultaneously overseeing various aspects of the research session.

Here are seven tips that I find useful when moderating a UT session.

1. Make the participant feel comfortable

An ideal UT session should replicate the exact environment in which the real user interacts with the product. However, this is rarely the case.

Participants often:

• Feel nervous
• Are conscious of how they are perceived
• Are aware that they are being tested and observed

As a moderator, you should be friendly, sincere, open-minded and flexible. Always be sensitive to how your participant is reacting.

Remember, a relaxed participant will give you more accurate results and the UT session will be more pleasant.

2. Profiling of participants

A nifty trick that I use whenever I begin a UT session is to match the participant against a list of commonly encountered participant personas. This helps me quickly determine how to interact with the participant.

My participant personas:

• The mouse: Very quiet and reluctant to share
• The talker: Very verbal and tends to go off topic
• The naysayer: Extremely negative and tears everything apart
• The know-it-all: Acts like the subject matter expert on everything
• The designer: Highly critical of colour, images and anything visual
• The yes man: Everything is great no matter how much he struggles to complete the tasks

As a moderator, you have to be aware that personas are just loose stereotypes and only serve as an initial point of reference. There is no one-size-fits-all classification for participants. You should constantly review and adjust the persona of the participant based on your own judgement, and alter the way you communicate with him/her accordingly.

3. Listen

The purpose of a UT session is to gather feedback from the participant. A good moderator should find the correct balance between listening and taking the lead.

Minimise interruptions by speaking only when the participant pauses so that you do not break the participant’s train of thought and miss out on valuable insights. Always give the participant the chance to internalise and think about your questions.

You can read more about being a good listener here.

4. Watch yourself

Know your moderation style

The Chinese philosopher Sun Tze wrote about the importance of knowing oneself. As a moderator, you should be aware of your own style of moderation, it’s limitations and how to compensate for them.

Your personality affects how you moderate a research session. Ask yourself if you have a tendency to be:

• Robotic vs animated (Personality)
• Transactional vs relational (Moderation method)

Try to avoid the extreme ends of each spectrum. If you are robotic, consciously make an effort to be more lively and engaging. If you are transactional, add a human touch. Simple gestures like words of affirmation can help the participant feel more comfortable.

Be aware of researcher bias

Researcher bias is a common pitfall of UT moderation. It happens so subtly that even experienced moderators are susceptible to it. You can identify and control bias by first being familiar with the different types of biases. This would allow you to be more mindful of the way you structure and deliver questions.

Common researcher biases:

• Question-order bias
• Confirmation bias
• Leading question and wording bias
• Halo effect

Find out more about the different types of biases here.

Watch your language

Besides phrasing questions in a way that minimises bias, you should also watch your own body language. Gestures can be picked up and easily misinterpreted by participants.

5. Watch the participant

Be aware of respondent bias

Respondents are also prone to bias. Once again, recognise the types of bias and take steps to reduce incidences that might affect the quality of your research.

Common respondent biases:

• Acquiescence bias
• Social desirability bias
• Demand characteristics

Read and respond to non-verbal cues

Non-verbal behaviours like body language and tone of voice are great indicators of a participant’s mood and comfort level. Such behaviours often accompany spoken words and can augment or contradict what is spoken.

Learn to identify such cues and adjust the way you interact with the participant to ensure that you are in the best position to interpret his/her responses. E.g. A fidgety participant might be anxious. Start with easier tasks and adopt a more reassuring demeanour. If this doesn’t work, the participant might actually be impatient or bored. Look for other cues, reassess your interpretation and react accordingly.

UT sessions can also be pretty draining. Increased restlessness and disinterestedness are telltale signs of respondent fatigue. Depending on the situation, you might want to pause for a short break or even consider simplifying or shortening the tasks.

Picking up subtle non-verbal behaviours often requires you to make inferences that take into account situational and sociocultural factors. It takes some experience to be able to notice and read these gestures accurately so don’t be too hard on yourself if you are just starting out.

6. Manage observers

A great way to get stakeholders involved is to allow them to observe a usability testing session. However, caution should be observed when dealing with observers because:

• Observers are usually not researchers and have a tendency to say or do things that may influence the outcome of a research session.
• Users feel more uncomfortable when they are overly outnumbered.

At Govtech, we have observation rooms that allow observers to remain ‘invisible’ and unobtrusive. If you do not have access to a dedicated research lab, an alternative method is to grant observers access to a livestream of the UT session. In the event that a stakeholder insists on sitting in for the session, brief them on a list of dos and don’ts and gently remind them if they overstep their boundaries.

7. Watch the time

Time management is a key but often overlooked aspect of a UT session. It is impossible to anticipate how much time individual participants require to complete each task. Watch the time, prioritise and modify the questions and tasks based on the situation.

In a nutshell

A big part of usability testing involves nuanced probing, ad-hoc questions and careful observation. You need to think quickly, maximise your time with participants while also being sensitive to their behaviour and reactions. In addition, moderators need to be conscious of their verbal and non-verbal interactions with the participants, and find the ideal balance between listening and taking the lead.