Asking better questions (How I do UX research)

In every project I need to collect feedback. Sometimes I’m trying to define a new feature, sometimes I’m evaluating and old one. But gathering feedback from real users is crucial to any process. The more specific feedback I need, the harder it is to come by.

More and more misinformation creeps in from the well meaning and genuinely nice people in the test groups. If I don’t work hard at avoiding it suddenly the feedback is crap and I end up designing a monstrosity like Windows Fucking Vista.

What does user group X really think of feature Y?

Asking questions like these is actually quite difficult.

While asking questions we bump up against all sorts of biases and ways of thinking that can frame or even distort the answers we get. It’s simply easier to give quick answers than it is to digest a question, think about it for a while and then respond. Who wants to look slow in this quick moving world?

But, these answers have to be right. They don’t just affect my design process, but guide the entire product. I’m not doing my job if I report back the first things that came to mind.

To get better answers, we need to be smarter about asking questions. These are some of the 3 most common problems I struggle with daily:

Trying to excel.

Most people want to excel when they’re presented with a question. They’re perceiving the situation as a challenge, or an opportunity to be noticed. This means that they’re trying to outsmart the question and give the right answer.

This is a terribly unhelpful. Because they’re no longer sharing actual experiences, but post rationalisations about experiences. This information is not really about the product I’m working on.

Being friendly.

A lot of people really just want to be friendly. They get a question and remember how in school feedback was encouraged to be positive.

“If you don’t have anything nice to say, then don’t say anything at all.”

Bullshit.

The problem with this friendlyness is that positives don’t really tell us anything.

While designing or creating something I equate 1 negative to about 1000 positives. So many people will tell you the nicest thing they can it creates a terrible bias in your data.

In short, people saying your product is great is worth absolutely nothing if they’re answering you. Give me some frustrates annoyance any day of the week, I can learn from that.

Trying to hard.

A small group of people try to solve the problem for you, or win. This behavior is the worst.

When people really put in the effort, usually to help out, they’re working against the purpose of feedback. I want information about how someone would use my product in a normal, daily, situation. Not how they might work around the bugs if they really put their minds to it.

Give me the lazy, distracted, half-assed responses that pop up when you’re beating the laptop keys in frustration at 10pm on a friday night when all you really want to do is eat (and have a god damned drink).

How I avoid this… so far.

There’s no perfect answer. I don’t claim to get this right every time. These are just some ways I’m trying to avoid these pitfalls as of today.

Never ask about what you really want to know.

Focus on something that is in proximity to the object of study. Preferably do this in a relaxed setting, with as little information as possible.

Fake test

Maybe you can even do the real test while asking them to click around to get warmed up, and then fake a short test after that.

Alcohol

Sometimes beer googles really help. Just make sure they do the drinking, not you. Impromptu demos at the bar will give you feedback you’d never, ever, get in the testing room at the Google Labs.

Don’t trust answers. All the helpful social skills are working against brutal honesty, and we need proper feedback to build good products.

Jesper is a former Game Designer turned UX designer that designs and develops products in many different fields. Right now he might be cruising bars demoing the next version of BlankPage, the online service that helps writer finish that novel they’ve been dreaming about.