5 things you should know before running a usability test

So you’ve spent weeks, months, even years developing your product, and now need to show it to your users, run some usability testing and get some feedback. Daunting huh!

I’ve facilitated usability testing with hundreds of users, on a whole variety of different companies and brands, and it turns out that there’s a very simple process you can follow no matter what testing you’re doing. If you get the basics right, it’ll never be a waste of time.

We run user testing every few weeks, not only to understand how well our products work for them, but also to keep in perspective that our users are real people with real needs. The point of user testing is to get answers from you users. It’s not about your voice, it’s about theirs.

Usability testing measures how easy your product is to use, by the users you put in front of it. Before walking into the testing room, you should have already arranged 5 or 6 users to turn up throughout the day, and arranged for a meeting room or lab to sit in.

Assuming that’s all set up, let’s dive in.

#1 — Build trust

When you facilitate user tests a lot, you’ll find that users will respond very differently to you, and you may need to subtly try a variety of techniques in order to get them to trust you. If they don’t trust you, they won’t tell you what they really think.

I have a few ways of doing this, including telling the user that I was brought in to test the product, and that nothing they say they don’t like about it will offend me. I’ll talk more about this in point 4.

Recently, I was out testing with one of our incredible summer interns, and I suggested they should run the introduction to the testing with the participant. This included getting some background from the participant, using a survey to understand their digital confidence and also getting them to sign a consent form. I hadn’t defined an order in which the intern should give these forms to the users.

The first two users were asked to fill out the consent form, then the background questions, and finally the digital confidence survey. They were both quite concerned about the data privacy of the testing, and we had to assure them several times that their data was not being collected.

The intern realised this was odd, and not something we’d seen before with other users we’d run the test with, so they decided to swap the order of the forms around, and ask background questions first, then the digital confidence survey, and finally the consent form. We didn’t find any users having concerns about data privacy after that, because we had built up a relationship with them before asking them to give us something.

This is an interesting example of permission priming in the real world (google it if you’re super nerdy like me).

Whilst conducting testing for a government client I worked with a few years ago, I had just finished a testing session with a particularly quiet and timid user, so I had been adjusting my body language to be very unthreatening, and using a quiet, gentle voice. This technique worked very well for this user, however when the next user came in the room very shortly after (leaving me no time to re-set myself) it was a different story.

I continued using these techniques almost like a hangover from the previous user. This did not work well for the second user. They were very assertive and weren’t giving me any useful answers. I realised I was stuck in the gentler, non-threatening persona I had adopted for the first user, and hadn’t adjusted to the much more masculine interaction this user was trying to have with me.

By adjusting my body language, as well as the tone and volume of my voice their answers totally shifted and it turned into a much more productive session with many more findings. The user trusted me because I was speaking their language, and putting them at ease.

#2 — Ask open questions

This may seem like the most obvious one, but it’s the most important, and the one I see most people get wrong.

Asking open questions isn’t about just asking the question on the sheet, it’s about allowing the user to share the stories they think are important.

More than anything, asking open questions takes practice. Its not just the first open question

“How you feel about this screen?”

But all the follow up questions too, that you may not have written down on the sheet.

“̶D̶i̶d̶ ̶y̶o̶u̶ ̶s̶e̶e̶ ̶t̶h̶i̶s̶ ̶b̶u̶t̶t̶o̶n̶?̶”̶ “So what would you do next?”

“̶C̶a̶n̶ ̶y̶o̶u̶ ̶f̶i̶n̶d̶ ̶t̶h̶e̶ ̶”̶b̶u̶y̶”̶ ̶b̶u̶t̶t̶o̶n̶?̶”̶ “What does it feel like you can do here?”

“̶H̶o̶w̶ ̶e̶a̶s̶y̶ ̶w̶a̶s̶ ̶t̶h̶a̶t̶ ̶f̶l̶o̶w̶?̶”̶ “How did you find getting through that?”

It’s an art, and it takes time to perfect.

My favourite techniques here are

• To start wherever possible with ‘Can you tell me a bit about…” because then you’re not directing them towards an answer, just setting a topic and letting them run with it.
• Silence. Just smile and nod and listen, and they’ll try to fill the silence. They should do about 90% of the talking.

#3 — Don’t take notes

When you’re talking to someone and they’re looking at their phone, or their smartwatch, you can just tell they’re not paying attention. You want to give up talking because you feel like they don’t think you’re worth listening to.

That’s what it’s like when the facilitator takes notes.

That’s why we always take an extra person or two along to the testing sessions so they can be the ones frantically taking notes, recording quotes and counting task completion rates so the facilitator can focus on interacting with the user.

However, you might find yourself in a situation where you’ve got a “talker” (a user who is a verbal freight train and you can’t get a word in edgeways). These users are still great to get findings out of, but sometimes you need to store up questions to ask later in the testing session. I find it’s best to use the facilitator sheet with all the questions and simply place dots next to the topics or questions I need to do more digging on as a prompt to myself, rather than writing out the whole question.

Taking notes as a facilitator is a sure-fire way to lose that trust you’ve been building with the user since the beginning of the session.

#4 — Don’t test your own stuff

I am a designer by trade, and trust me when I say it’s impossible to be impartial when someone is tearing your design apart in front of you.

Some users point things out that are really helpful, things you’ve overlooked, and you’re grateful for them sharing their points of view. However you will eventually find yourself faced with a user who is on the warpath.

They may hate the company you’re working for, they may believe they could do a better job designing it, or they may just be plain mean about what you’ve designed because they’re having a bad day.

We recently tested a service I had designed for another government agency and our last user of the day arrived, announcing they knew everything there was to know about web development because they were developing their own app. They spent an hour explaining how everything I had designed was wrong. Unfortunately they forgot the number one rule of design:

YOU ARE NOT YOUR USER

So all the fixes they were throwing out were going to work very well for them, but not for anyone else we had tested with that day.

Now, you should never ever blame your user for not being able to understand what you’ve designed, however sometimes there are users who have ulterior motives and aren’t going to give you constructive feedback.

After that session, my colleague who was facilitating came back to the observation room completely drained and de-motivated after seeing a user tear apart the designs for no apparent reason.

Moral of the story — never test your own stuff!

#5 — Plan, plan and plan some more

User testing is tough, its tiring and it’s incredibly complex. There are so many moving pieces to get sorted before you can even begin.

In order to keep everything flowing, start with a good plan.

I normally have 3 different documents prepared each time we start a testing session:

1. Facilitator task sheet — This is a very detailed, step by step breakdown of the whole testing, from who is doing the intro and background questions, to the exact topics and areas the designers need answers for. This will take a long time to write at first, but you can re-use it in every session and just re-hash the content.
2. Participant task sheets — If you are testing anything that requires the user to have information, make sure it’s printed out, and that you have multiple copies. If you are testing a shipping tracking service, make sure you have the shipping information printed out for the user to work from, and make sure the information is realistic!
3. Introduction sheets — Background questions, digital confidence surveys, consent forms and anything else you need the user to fill in before they start the testing. Always print 2 extra copies as well, you’ll be amazed how many people will mess up a consent form!

I hope this has been helpful, and puts you in good stead before running your first user testing session.

There is a tonne of information out there about the finer details of user testing if you want to read more. My go to site is always Nielsen Norman Group https://www.nngroup.com/topic/user-testing/