The Biggest Misconception about User Testing

Get more accurate user feedback with less work by chilling out and taking it one step at a time.

The single biggest misconception I see about user testing is that your team needs a User Testing Plan. In other words, that there should be a roadmap of exactly what to build, how to test, and with many people.

Plan Only For the Next Step

In reality, it’s not only unnecessary to have a plan — it’s actually literally impossible to make an accurate one. Even more, creating and sticking to a testing plan at the beginning of the project is likely to set you up for active misinformation.

Here’s why:

The most important reason to user test at the beginning of a design project is to identify unknown unknowns.

In other words, we don’t just test when we have a pretty good educated idea of what’s going on and we want to confirm it. That’s great and all. But it’s even more essential to leverage interaction with users to see what large swatches of perspective we are completely missing in the first place.

The most critical first stage of design & testing isn’t about getting the answers — it’s about identifying the questions.

That means that, when done properly, our understanding of what the right questions to ask are should change every 2–5 user tests, aka every cycle of understanding. It should take up to half a dozen cycles to get a good enough sense of the problem and solution space to make highly educated projections.

That means that, for a long initial design, development & iteration period, it’s literally not possible to know what experiments we should run down the line — we can only project the next experiment we need to run now.

Salsa, Not Ballet

If it helps, think of user testing more like a salsa than a ballet. Mellow out, pay close attention to your partner (ie, your user), sashay your proverbial hips, and pivot your testing itself fluidly in response to a continuous stream of new input from the user.

Another metaphor: Imagine you’re just starting to meet someone and date them. Would you plan out your life with them in the first couple dates? No, you’re just starting to get to know them and understand what the mutual compatibility is.

Benchmark by Questions

Instead of managing your design workflow by having a roadmap of tests, think of your journey as a journey through questions.

You start with the big, existential questions, like:

• Is this even valuable for people?
• Is it possible for us to strike a unique new balance between A and B?
• What are the most important components for the core design?

Then you learn what a bunch of specific questions are through testing. Like

• It turns out that everybody just cares about X. Can we redesign it around X while still achieving Y?
• Nobody understands what’s going on because they think it’s this other thing. Can we connect it to something they’re more familiar with?
• We’re stuck between A demographic who trusts us with nothing and B demographic who trusts us with everything. Should we strike a balance or focus on one?

As the journey goes on, you keep testing, and you know you’re making progress because you’re biggest, messiest questions keep changing.

Suddenly, one day you look up and your questions look more like:

• We know everybody wants this list of features, but what timeline should we implement it on?
• We’ve gotten X market down, how can we scale to Y market?
• Is there a more efficient way to get people through this part?

And these feel like the most stressful, most fundamental questions in the world…. Until you look back and realize that 6 cycles of testing earlier, you were asking yourself whether anybody at all even wanted to use it.

What’s the Right Next Test for Me?

I was original going to call this article “The Single Biggest Mistake in User Testing”, but I realized that’s not true.

The single biggest mistake I’ve seen people make in user testing is just not getting around to doing it.

And most often, we don’t get around to doing it because we’re not sure if we’re ready to test, or what to test, or how, or if we’re doing it right. But that uncertain comes from a sense that there is a “correct approach” that we may be falling shy from.

Instead, just get moving by freeing yourself from the need for a correct plan, and just ask:

“What’s the most important thing I need to learn right now?”

“What’s the right next test for me? How might I gain insight into that in under 3 hrs of work within the next few days?”

As long as your next test gets you some insight into your most important question, you are moving forward along the correct plan.

There Was a Plan All Along!

Okay, yeah, there kind of was a plan all along.