Jan 10, 2019
Research Questions Are Not Interview Questions
(You can’t just ask people what you want to know. Sorry.)
The most significant source of confusion in design research is the difference between research questions and interview questions*. This confusion costs time and money and leads to a lot of managers saying that they tried doing research that one time and nothing useful emerged.
(It’s even worse than how the terms “user research” and “user testing” are thrown around interchangeably. They are not interchangeable. The former is the process of learning about the people who are the intended users of a system, and the latter is evaluating the system itself against established criteria by having people interact with it. And for the record “user testing” isn’t really a thing because you aren’t testing users, you are testing the usability, or other qualities, of a system. Intentional phrasing matters.)
Your research question and how you phrase it determines the success and utility of everything that follows. If you start with a bad question, or the wrong question, you won’t end up with a useful answer. We understand this in daily life, but talking about research in a business context seems to short-circuit common sense. Everyone is too worried about looking smart in front of each other.
Research Questions
Your research question is simply what you want to find out in order to make better evidence-based decisions. A good research question is specific, actionable, and practical. This means:
- It is possible to answer the question using the techniques and methods available to you
- It’s possible (but not guaranteed) that you can arrive at an answer with a sufficient degree of confidence to base decisions on what you’ve learned
If your question is too general or just beyond your means to answer, it is not a good research question. “How’s the air on Mars?” may be a practical question for Elon Musk to answer to meet his goals, but it isn’t for most of us.
Research Question Examples
- How do recent college graduates decide what and when to eat?
- What are the greatest barriers to saving money for Canadians in their 30s?
- In what US cities do residents spend the highest proportion of their income on vacation travel?
- How well does the latest version of the marketing website answer prospective customers’ questions?
- How does Company X create their product roadmaps?
Only after you have identified your research questions can you select the best way to answer the question. A lot of people get this wrong and pick the research activity first.
NO: “We’re going to run a survey. What should we ask?”
NO: “We’re testing a prototype with non-customers on Friday. What do we need to know?”
YES: “We need to understand the factors that influence how our current customers select and view television programming, so we are going to recruit and interview current customers in their homes.”
You won’t even know whether doing interviews is the right thing until after you’ve written your research question. It might be more effective to read existing literature, or observe people out in the world, or do a competitive analysis instead.
Maybe the best way to find out whether anyone will develop a habit around your is app to learn what gets people to floss their teeth. Maybe the best way to understand your prospective customers is to usability test a competitive service. Resist the urge to narrow your research focus too quickly or you will miss opportunities to learn what you didn’t think to ask. Resist the temptation to ask your questions directly or you will be basing decisions on total fabrications.
Special comment about organizational research
Many product teams assume all the research they need to do is user or customer research. And then they are confused when their findings are being ignored, disputed, or distorted by leadership, or engineering, or even the members of their own team.
In addition to the wider business landscape and social context, your research questions must include the organization in which you’re working. Organizations are the social context in which design and product decision-making happens. If you don’t understand how people make decisions in your organization, you will never be able to influence them.
Example organizational research questions
NO: How shiny does our report need to be to impress the leadership?
YES: Who makes product decisions, what incentives influence them, and what sources of information do they trust?
Facts don’t change minds. Reports are easy to ignore. Reality often contradicts the wishes of authority. It doesn’t matter how good your data is if you haven’t done the groundwork to ensure an evidence-based framework for making decisions is in place before doing the work of gathering other evidence.
Don’t be afraid to step back from your comfortable assumptions and start at a high level. When you consider what you need to know to solve the whole design or business problem, it probably includes far more than user behavior. You don’t need that much more time, but you will need a lot of courage.
Quantitative vs Qualitative
There are only two kinds of data: descriptions and measurements. If your question is about what happens or why something happens, you are asking a qualitative question and need a qualitative method. If you want to know how much something happens, or how many of something there are, you need a quantitative method. And without getting too much into statistics, you need a large enough sample size to do quantitative research. I’ve seen a lot of executives who freak out that a round of interviews only included 12 people, and yet are willing to base major decisions on analytics from a few hundred active users.
A lot of bad research results from a mismatch between question and method, usually because people spend more time worrying about the activity (surveys, testing, interviews) than about forming a good question. Even more bad research is designed specifically to provide support for an existing solution. (That’s why it’s called “validation”.)
Ethnographic interviews are a qualitative method.
Interview Questions
Once you have identified a good research question and decided that interviewing people is the best way to answer it, you need to figure out what to ask the people you are interviewing. Again, research questions are not interview questions. If you have a strong research question in mind, you might only need a couple of prepared interview questions.
Research Question Example
YES: How do families with school-age children decide how to spend money on vacations?
Interview Question Example
YES: “Walk me through your last vacation from planning it until when you arrived back home.”
That might be all you have to say to learn about how people choose a destination, how they budget and spend their money, and how they share information and make recommendations to their friends.
Some research questions are completely unanswerable if you pose them directly, especially if they are about motivation or sensitive topics such as health and money.
You cannot ask people to predict anything or to remember something that was too far in the past. No one can tell you how likely they are to do anything in the future. And humans are way better at fabricating answers than at remembering past events accurately.
When surveys are written without regard to context and with untested questions, it’s really easy to gather a bunch of garbage data based on unanswerable questions.
Bad Interview/Survey Question Example
NO: “On a scale of 1–10 how likely are you to buy a new smartphone in the next 6 months?”
No one can answer that. Do not ask anything like that of anyone ever. This is an extreme example, but I’ve seen similar in the wild many times.
Go forth with rigor
It takes discipline to remain goal-oriented and skeptical, even of the ways you are asking questions. But this discipline pays off over time. You can focus your efforts on asking and answering the questions that make a difference.
*You can replace “interview question” with “survey question.”
Does your team have more data than understanding? Need help creating a learning culture? Get in touch!
