The Question Block: Level Up with Better Questions
While playing Super Mario Odyssey recently, I had a small epiphany. Those iconic question blocks that Mario constantly bashes hide a treasure trove of possible rewards. That wasn’t the revelation. But when thinking about design research and those question blocks, something clicked. In Mario, most question blocks reveal only a single coin. You jump, you break, you get a coin, and you’re done. It can feel the same way with research questions sometimes. You ask a question, you get an answer, and you’re done.
But occasionally, you stumble on a question that unlocks new ideas, new questions, and a veritable treasure trove of understanding. Going through the research process can feel like Mario bashing at every block in his path, hoping for something good. But there are strategies for pulling more value from the right questions, and they’re a lot like the strategies for finding better rewards in Mario.
Test every question
In a Mario game, every question block hides something of value. In research, each question holds a small nugget of possibility. The work (and the game) is figuring out which ones give what you need and ignoring the rest. When you’re first starting out, it’s a good idea to ask every question you can. Remember which questions give you the best insights, and make an effort to repeat and build on those. Test your questions by:
- Answering them yourself: If you can answer with a single word, the question is probably too closed. Are you able to answer the question honestly, or do you feel compelled to make something up on the spot? If the latter, the question may be too speculative. Can you learn something from your own answer? Read the questions closely. Is the answer you want embedded in the question? If so, it’s likely a leading question and needs to be reframed.
- Asking them of peers: Testing your questions with peers can unveil whether your questions are clear or muddy. Do your peers answer the question you expect, or do they hear something different? Does the question make them uncomfortable? Does it confuse them?
- Evaluating responses in aggregate: As you do more research, pay attention to which questions unlock better insights. Refine, adapt, and reuse the best questions. If you constantly get the same, repetitive answer to a question, consider whether that question is worth your time or whether it needs to be reframed a different way.
Not every question is worth the time
As you do more research, you’ll quickly learn that some questions just don’t yield the results you want. It’s okay to skip them in later sessions. Hitting the highlights and moving on will either grant more time for deeper responses or allow you to explore new and different questions. If the question is important enough, but you’re not getting what you expect from it, revisit the question. There’s a good chance you could get better results if you reword it, reframe it, or move it.
Be ready to pivot quickly
Sometimes, question blocks release a moving reward: a 1-up mushroom, an invincibility star, etc. The only way to capture these is to chase them down. In research, the right question might lead the discussion in a new direction. If you are prepared to follow the thread, you can reap the rewards. But to follow up and do it well, you’ll need to adapt and change tactics on the fly. Here are some tips to help you follow up:
- Repeat or paraphrase the participant’s answer back to them: Hearing their answer in your voice gives the participant a chance to evaluate whether it matches what they meant, whether they have more to say about it, and whether you understood it. It can prompt them to either confirm, refute, or build on what they said before.
- Answer questions with a question: You might feel like a psychologist, parroting questions back to your participant. Our instincts are to help people when they have questions or problems. Ignoring those natural instincts and putting the impetus back on the participant can nudge them to dive deeper into what’s truly driving them.
- Ask for clarification: It’s okay to admit if you didn’t understand or need more detail. Even if you think you fully understood something, you might find that asking for more detail completely changes how you see what the participant was saying before.
- Be quiet and let it happen: Silence is golden — and super awkward at the same time. Get comfortable letting the participant do most of the talking, and you’ll find that they fill the quiet subconsciously.
Keep pushing
Much like the repeating coin blocks in Mario, some questions just keep giving and giving. My 6-year-old can help me see things in a completely new light with a single word: “Why?” Sure, the word will drive most parents completely off their rocker at some point, but it’s also effective at cutting to the core. The Five Whys is a helpful technique for diving from a surface-level answer to a much deeper understanding with just a few simple queries.
Sometimes in research, a participant’s answer barely scratches the surface. When you’re just starting out as a researcher, it’s a good idea to ask “why?” more often as you try to suss out what may lie deeper. As you get more experienced, knowing when to probe and when to let it slide will come more naturally. One important key no matter how experienced you are is to present yourself as genuinely curious and not judgmental. Ask why gently and show continued interest with each successive question.
Use questions as platforms
As part of a typical research session, a mentor of mine liked to ask participants the standard net promoter score (NPS) question near the end of the session: “On a scale of 0 to 10, how likely would you be to recommend this product to a friend or colleague?” On its face, this seems like a terrible idea: NPS isn’t valuable as a survey question even in aggregate, let alone from a single person.
Then I noticed that he never recorded their rating. Instead, he used it to frame his next question. “Why did you give it such a high rating?” or “What would make you rate the product a 9 or 10?” The initial NPS question was just a throwaway. The answer only mattered insofar as it gave him a starting point to frame the real question (or questions).
It’s common advice to avoid asking closed questions: those that can be answered with a yes, no, or other single word. This is often the right approach. But sometimes, these questions can give you a new starting point to jump from.
In conclusion
You can’t control which blocks in Mario yield the best rewards. But as you play the game more, you’ll get better at focusing on the blocks most likely to yield better rewards. The same is true with research. By testing your questions, learning when to change direction and when to keep pushing, and using your questions as a platform for others, you can get pretty good at using the right questions to uncover better insights.
Want more help coming up with great questions? Check out these resources:
- Asking the right questions during user research, interviews and testing by Fabricio Teixeira
- Open-Ended vs. Closed-Ended Questions in User Research by Susan Farrell
- How to interview users by Ania Mastalerz
- Never Ask What They Want — 3 Better Questions to Ask in User Interviews by Charles Liu
- Interviewing Users by Jakob Nielsen
Originally published at uxcellence.com on February 17, 2018.
