Dig in the Right Spot
Without asking the right questions, you‘re just wasting time
“Forty-two,” said Deep Thought, with infinite majesty and calm.”
“Forty-two!” yelled Loonquawl. “Is that all you’ve got to show for seven and a half million years’ work?”
“I checked it very thoroughly,” said the computer, “and that quite definitely is the answer. I think the problem, to be quite honest with you, is that you’ve never actually known what the question is.”
—Douglas Adams, Hitchhiker’s Guide to the Galaxy
Debating the utility of various research methods and activities is a popular pastime among designers and product teams. Even among those who value evidence-based decisions, no one seems to be talking about what makes a good or appropriate guiding research question. A common assumption is that research questions are just what you ask in a survey or user interview, and that you can ask whatever you want to know directly. The truth is two counts of nope. If you don’t start from the right research question, you’re probably just kicking up a lot of dust and obscuring more than you’re clarifying.
Questions give data meaning
Without a good question, all the data in the world is as useful as a random page of Google results and no query. No one is that lucky. Staring at data until patterns emerge is just another—popular—way to set money on fire. Many well-meaning teams make a regular habit of talking to their customers and then don’t know exactly what to do with what they’ve found, except argue about it, or build features based on anecdotal requests. A good question is a tool that grows more valuable with use. A good question can help protect collaboration from a dominant personality.
You can ask “What’s the weather like?” every single morning without needing to change up the question. But imagine if you asked once, got an answer, and then dressed for rain for the next six months on the assumption that nothing had changed. A lot of organizations work like this with regard to learning.
Your research question is simply what you want to find out in order to make better evidence-based decisions. However, it takes some care to identify and articulate the most useful question.
For example, if you want to start yet another meal kit company, you might need to know how to get recent college graduates to subscribe to your service. But you can’t just go around to recent college graduates and ask “What would get you to subscribe to my meal kit service?” People are as good at fabricating plausible yet totally bogus explanations as they are bad at divulging their true motivations. Shorter: people lie.
A good research question is specific, actionable, and practical.
A good research question is specific, actionable, and practical. This means that it is possible to answer using the techniques available to you and that it’s possible to find an answer in which you can have sufficient confidence. (Certainty is impossible, so it’s all degrees of confidence.) If your question is too general or beyond your means to answer, it’s not a good question. “How’s the air on Mars?” may be a practical question for Elon Musk to answer to meet his goals, but it isn’t for me.
“How do we get Millennials to like us?” is the most popular worst question.
Does your team need help figuring out what you really need to know? Mule can help. Get in touch.
Research questions are not interview questions
Again, research questions are not interview questions. Your research question is what you need to know to make better product decisions. Your interview questions are what you ask other people in order to learn what you need to know. If you have a strong research question in mind, you might not even need specific interview questions.
Research question: How do recent college graduates decide what to have for dinner?
Interview question: What did you eat yesterday?
Interviewing may not even be the best way to answer your research questions. It might be better to read existing literature, or observe people out in the world, or do a competitive analysis. Because of the emphasis on a narrow definition of customer empathy, a lot of designers skip analogous research entirely when it could be the most fruitful. Maybe the best way to find out whether anyone will develop a habit around your app to learn what gets people to floss their teeth.
Yes, it takes more effort to identify what you really need to find out than it does to write a set of the wrong interview questions. It takes collaboration and a clear shared goal and a willingness to be proven wrong. The public uncertainty, it burns!
And, by the way, the best interview question is often just “Tell more more about that.”
Let’s review the process
The applied research process—using inquiry to solve a specific, practical problem—is not without nuance. However, there are three essential steps. 1) Form questions 2) Gather data 3) Analyze data to determine what it means.
You cannot skip any of these. Research for product design is too often treated like the inverse of the underpants gnome scenario. Everyone focuses on Step 2, the accumulation of data, without a clear question or sufficiently rigorous analysis. Given a pile of data without discipline, biased human reasoning sees patterns and draws conclusions that don’t reflect reality.
And the whole point of using research is to make sure your work comports with reality rather than reflecting wishful thinking.
Goals before questions
Before you can form good questions, you need to have a clear goal. Your goal could be something very general, like “make money solving a problem for affluent people”. Or, it could be much more specific, like “increase the number of subscribers by 10% in a 3-month period” or “decrease the amount of hate speech on the platform by 80%”.
Without a goal, it is impossible to identify the highest priority questions, or evaluate whether the research you did was useful and worthwhile. Did what you learn help you make money solving rich-people problems? Then, great. This is why a lot of organizations follow the process of 1) give new types of research a try 2) find it vaguely unsatisfactory, then 3) go back to making stuff without asking as many questions.
Focus on the highest priority questions
“There are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns — the ones we don’t know we don’t know.”—Donald Rumsfeld
People made fun of Rumsfeld for that statement, but, other examples of terrible judgment aside, it’s pretty astute. Just acknowledging that there are things you don’t know you don’t know puts you in a much stronger position vis a vis reality.
There are so many things you could ask. How to choose? The best question is the unknown that carries the most risk. Risks include things like:
- Your target customers don’t value what you are thinking of creating
- Solving this problem won’t create sufficient business value (empathy doesn’t fix this one)
- Solving this problem risks creating other larger problems
- Your organization doesn’t have the capability to solve the problem
- Someone or something else offers a better solution to the same problem
Laura Klein has a good presentation on Identifying and Validating Assumptions. My one note is that you shouldn’t rush to test hypotheses before you ask more open-ended questions. Otherwise you could miss the big picture. It’s tempting to narrow too quickly in the interests of being “lean”. Then you end up with the dreaded local maximum. The goal is learning, always learning.
Questions before methods
Research activities are simply ways to answer questions. There is no one best method, just the most appropriate for your goal, question, and available resources and expertise. I hate the term guerrilla research, because it implies that more resource-intensive processes yield better results. This is not the case at all. Only after you know your question can you pick the best way to answer it. And you can’t start from an answer and work backwards as though you are playing design Jeopardy.
“Guerrilla research” implies that more resource-intensive processes yield better results, which is often false
A lot of bad research results from a mismatch between question and method, usually because designers or researchers find a method they are comfortable with and try to use it for every question. Even more bad research is designed specifically to provide support for an existing solution. (That’s why it’s called “validation”.)
You can’t answer a qualitative question, that seeks a description, with a quantitative method, that provides a number. Is your question what problem you should solve, or how well your solution is working (for a particular population in a particular context of use)? If you want to know how people behave, you won’t learn by testing your product.
It takes a lot of discipline to remain goal-oriented and skeptical, even of the ways you are asking questions. But this discipline pays off over time. You can focus your efforts on asking and answering the questions that make a difference. This leads to better products, faster and more learning over time. And you won’t find yourself sitting with your team staring down the useless certainty of an answer like 42.
