Testing designs with humans

Tips and tricks on how to get started

Ross Chapman

Published in

Etch

7 min readApr 11, 2018

Will (left) testing an experience with Joe (right)

You’re not a user experience designer if…you don’t talk to users
— Whitney Hess

Background

When I started learning about user experience design, it was the usability testing that instantly drew me to it. No longer was the designer a magician who could miraculously deliver designs that were successful from launch. No, there was a science to it. Creating designs for people successfully now meant that designers needed to involve the user in the process in order to understand their behaviour.

Years ago it was hard to explain this to our partners. They had the assumption that only they would sign-off everything — they were footing the bill after-all!

Thankfully, the importance of UX grew and businesses grew wise to it. They started admitting that they didn’t know everything and actually their whole business was in the service of other people.

Why should businesses test with humans?

Applying UX up front saves money further down the line. Whether that be reduced operating costs in fewer support requests, increased sales by reducing friction to buy, but moreover — the reason to test for businesses is to get the maximum amount of value out of investing in UX.

It also reduces risk. Knowing whether your potential customers will want and use your new idea (and as you intend) before you release it to the world is invaluable. Isn’t it better to know after four days that your idea sucks, rather than four months?

Creating a prototype and testing it with the right people will also increase confidence that you’re building the right thing before investing in development.

There’s a lot of evidence online about why testing with people is vital. Here are a few related links:

That’s great Ross, but how do you run them?

I’m assuming that you have something to test out. That could be:

A high-fidelity prototype in InVision or Marvel
A front-end coded experience, maybe on a localhost or staging server
It could even be a paper prototype or low-fidelity experience

You’re allowed to do a bit of smoke-and-mirrors too. You shouldn’t be creating the product and then testing it because then you’ve spent all the budget only to find out at the end that the design might suck and people are getting confused (potentially).

It’s much better to do something small, test it with real people and iterate. Creating this feedback loop is something Jeff Gothelf talks a lot about in his book Sense and Respond. You need to create a two-way conversation between yourself and users and ask the right questions. Do you know what you are really testing? Some ideas:

Is the proposition right?
Is your assumption that they will click on a specific button correct?
Do they experience any “weirdness” — designs that don’t work as they imagine they should?
Are you explaining the concept in a way that people understand it?

The methods of running the test

There are many ways of running a test and often you’ll decide within the constraints of time, budget and the quality of the feedback you need:

Guerrilla user testing — essentially grabbing anyone you can for 20 minutes. Due to its informal nature, it is the quickest and easiest (£0)
Remote panel testing — through apps like What Users Do who have their own panel of users (£500+)
Moderated remote test — online, but with users you’ve recruited yourself. We use Lookback.io (Starting at $59 per month+ time to organise)
Ethnography — going to where users are. This maybe their place of work, where they spend time at the weekend — places they are familiar with (£0 + time to organise)
In-person tests — getting them in front of you where you work. Convenient for you, but maybe less convenient for them — depending on audience (£0 + time to organise)

There are pros and cons to all. Guerrilla testing with co-workers is easy but doesn’t give you that necessary outside view. Remote with general-interest folk is also easy to setup, but are they fully engaged?

From that list, the last three options are ones we typically go for.

Regardless of the type of test you are doing, try to keep these tips in mind:

Get participants to use their own equipment — the number of times I’ve winced when they get stuck using my Mac — not a great feeling!
Mention that you didn’t design it — It can be useful to mention that you didn’t create the design. Some participants like to people-please and you could get less useful results.
Believable scenario — Try and start with a scenario that’s believable. With a local council, we asked participants to find out when their recycling bin collections are because they heard from a neighbour that their’s wasn’t collected one week. The more believable and natural, the better
Double-up — I’ve found it best to have two people — one moderating the test and one to take notes. It’s a lengthy process to transcribe after the event and you may realise that you should’ve asked the vital follow on questions
Keep to consistent questions to gain trends — Ask the participants to follow the same test and ask the same questions. That way you can find trends. Saying 100% of people found “X” an issue is powerful. Sure, you’ll get extra feedback, so note that down too
Test with five — Use the Nielsen Norman Group formula and aim to get around five participants. It’s enough to spot trends and not too much that the research becomes unwieldy to use

Recruiting participants

Depending on the type of test you use, you’ll go about finding participants differently. When using What Users Do or Usertesting.com, they sort all that out for you (just bear in mind that it’s a general audience).

In most cases, our partners will help find participants. Some even have ‘customer panels’. Again, be wary that they won’t people-please.

We have found that social media is a valuable resource when recruiting participants for user tests. Both Facebook Ads and even just sending out a tweet are effective methods of finding humans to test your product.

There are recruitment companies that specialise in finding participants too, if you need further help.

The things others won’t tell you

We’ve done testing for a number of years and have found that while it is a manual process, it is a repeatable one. Here are some of the things we’ve learned over the years:

Most people will be working 9–5 — Consider testing at lunchtimes and providing snacks. Some users might be more available after work — can you be flexible with your testing?
Cancellations — You will often have people cancel on you — such is life, but offer…
Rewards! — Add incentives, such as £20 Amazon voucher — people are volunteering their time after all and it may help ensure that they show up
Use a screener — Send potential participants a screener questionnaire to whittle down a bigger list to more ideal candidates. Do they fit your persona/target user?
Test the technology — For remote moderated testing especially, it is worth testing that the real test will work before hand. Tools like Lookback require a Google Chrome plugin to be installed and screen sharing, webcam and mic to be enabled. Some companies will have pretty secure firewalls that might prevent that, so test the connection first!
Be careful — Some of our projects are in their early stages and partners may want to keep their new product a secret until launch. You might need a non-disclosure agreement signed by participants
Size it right— Where we’ve been testing a responsive web experience, we’ve created separate mobile and desktop tests. This may help to highlight unique issues

Here at Etch, we have learnt a lot about testing with humans, and we strive to continue that learning. These simple insights and considerations are a collection of lessons we have learnt to help a user test run smoothly.

Remember, test small and often. Above all don’t stop learning!

This post was originally featured on the Etch blog, check it out here.