First-Time Usability: Moderating
Tips on moderating your first usability study
If you’ve ever moderated, or participated in, a user session, then you know it can sometimes feel as though you’re under a lot of pressure. Just before my first usability session, I was a mess. I was sweaty, nervous - yet handsome - mess. I’d completely convinced myself that I was going to blow it, embarrass everyone, and not get any usable information.
Thankfully, that didn’t happen — not totally anyway. I definitely made mistakes, but come with errors beforehand because they are going to happen. And that’s totally fine. Even with mistakes, nothing will explode and the world won’t end (probably). And you’ll still get lots of usable information.
As with most things, sessions get much, much easier to run with practice and time. You’ll find your own pace and rhythm, but, in the meantime, here are a few things that might help.
Before Arrival
Practice.
It’s really easy to skip this. You probably wrote the script and know the product really well. But the importance of practicing can’t be overstated. You’ll need to practice:
- The flows of your script, the tasks, and how they interact with the product.
- Good habits, such as asking open-ended follow-up questions
- Avoiding bad habits, such as accidentally helping the participant
So, grab a few coworkers run through that script!
Check all the equipment.
Things tend to malfunction at inopportune times — computers, cameras, microphones. Count on something being difficult or, at very the least, turned off. Addressing one of these situations can turn into a giant time suck and ruin the time frame for your session, so, double check your equipment!
This also applies to your product, particularly if testing a demo or beta site. It can be terribly frustrating to find that the site has been updated, looks different, or is down for maintenance.
Make the space comfortable.
Clean up any trash, adjust the temperature, close/open the blinds, and make sure it doesn’t smell weird. These are small adjustments, but a clean, comfortable space will make a positive impression and help ease the participant.
I also like to make sure the chairs are arranged so that the participant is facing away from others that may be in the room - no one likes to see people staring at them.
Prep with your partner, if you have one.
When moderating a session, it’s important to focus and concentrate on the participant and the tasks at hand. Having a partner to take notes and deal with unexpected problems during a session is a particular boon, especially if new to moderating. Take a few minutes and go over everything with him/her before each session, as you may want to make some adjustments to the script or notes.
Arrival
Introductions and housekeeping.
I like to meet participants at the door to start with a positive and welcoming tone. I take few minutes before going into the space to sign/complete any paperwork and lightly go over the general structure of the session. Hopefully, you’ll have covered this when recruiting, but it’s always very helpful to frame it again, such as discussing the type of session (usability test, interview, etc.) and the length.
I also discuss the concept that this is to gather insight and NOT A TEST and let them know their thoughts and opinions can’t be wrong. I often tell people that this is 45minutes of their life where they can’t be wrong.
Be considerate.
Successful sessions are deeply reliant on whether a participant is comfortable. If they feel comfortable, then it’s likely that they will share all their opinions, good and bad. If they’re uncomfortable, answers tend to be more vague and less helpful.
You want the participants to trust you so be polite, courteous, and, above all, actually curious. If you’re alert, aware, and asking pertinent follow-up questions, participants will know that you’re genuinely interested, which often leads to more detailed comments.
Give them the incentive.
While incentives aren’t the only reason people choose to participate, it can be a motivating factor. If you get it out of the way in the beginning, then they won’t be thinking about it and can focus on the session. To me, it reinforces the idea that they “can’t be wrong” and encourages people to answer honestly without a possible consequence of not being rewarded.
During the Session
Start the recording. Start the recording. Start the recording.
I’ve gotten into a few sessions only to realize later that I didn’t start the recording and was all:
Recording sessions is vital. While I mostly depend on my notes to guide me, recording each session allows you to relax a little. You can always go back and look at task times, double check interactions, or look for additional insights.
And while recordings won’t change, your memories will, especially after a few sessions, so it’s important to have the recording to double check your impressions and provide evidence of findings.
Don’t demo the product.
This can be difficult, particularly if you often show people how to use your product. It is a natural tendency to help people during a session, especially if they express frustration or ask if they are doing the task the “correct” way.
Remember, though, that the point of the session is to find both the strengths and weaknesses of the “correct” way. You should rarely, if ever, be using the mouse.
In times of confusion, force the participant to think through the issue aloud, which will help exposed his/her mental model of how they think product should work. This is the heart of testing and user sessions — uncovering the participant’s mental model that agrees/disagrees with the product.
Let the user do most of the talking.
Silence in sessions is sometimes hard to swallow. It’s very easy to keep talking to fill the silence that occurs when a participants pauses. Remember that silence in sessions is very important — it allows participants a moment to stop and gather their thoughts. Filling that space with talking might derail a thought instead of allowing them to work through it themselves.
When in doubt, repeat it back.
If you don’t understand something, ask the participant to explain it again. Then, in your own words, repeat it back to them to make sure you interpreted them correctly. I often say something like “Ok, I’m going to repeat back to you what I heard to make sure I’m getting this right…”. Don’t let a participant leave without fully understanding everything they expressed.
This is an important thing to do when a participant uses a buzzword, like “intuitive” or “a bad experience”. It’s very important to dig further here. Ask what they mean specifically and, possibly, ask them to pick two or three other words to describe the issue.
After the Session
Leave time for their questions.
Often, sessions will create a lot of questions for participants. Be sure to leave 5+ minutes to go through anything they may want to cover as well. It gives a nice segue out of the pointed questioning they just experienced.
Have a summary sheet for anyone else who was in the session.
It can be hard to take notes that others can understand, so as a fail safe, I often like to give a quick summary sheet to those in the room after the user leaves. This is basically a structured opportunity for people to record what they remember about high-level issues. For example, two questions I use frequently are “What 2 or 3 things seemed to be most difficult for this participant?” and “How would you change the product to address this specific participant’s problems?”
Leave a few minutes to talk about the session quickly with your partner. Review the issues when they’re fresh with your partner or, if unavailable, another person, such as a notetaker or stakeholder. Sometimes, a session might expose a very interesting aspect or problem, which you may want to focus on in other sessions. If you think something might change the script, you should confirm it with others before you make changes.
First-Time Usability:
1. Creating a Plan & Scoping and Planning Sessions
2. Creating a Test and Script
3. Recruiting Users
4. Moderating Sessions
5. Analyzing Results
