How Leading Questions Distort Reality
Sustaining Unbiased Questions During User Interviews
Summary: This article explores why leading questions impact the answers people give, and how to craft the perfect question to derive hidden insights from users.
The Leading Question
Taylor asks, “How much better is this coffee?” You take another sip, nod in agreement, and say, “It’s so much better than that other hipster coffee shop.”
The phrasing of Taylor’s question created a barrier for disagreement, and encouraged the respondent to provide a favorable opinion of the coffee. “How much better” framed the answer.
Leading questions are found in everyday conversations, marketing material, and even inside the courtroom. Watch TNT or NBC on any given weekend and you’re bound to run into a Law & Order marathon. Watch how both the prosecutor and the defendant’s lawyer will call out leading questions, assumptions, or statements.
In Practice
Leading questions produce disingenuous results, which corrupt product design decisions. This culminates in shipping poor solutions that do not address core problems.
“How much easier is it to upload your file now?”
This questions assumes that this new design is easier to use.
“How confused are you in this workflow?”
Positioning the workflow as confusing anchors users to respond in kind.
“How upset do you get when the software produces an error?”
This question assumes the user is upset or frustrated.
How to Avoid Leading Questions
Writing unbiased questions is challenging. Flexport’s design team uses a three column method to help identify and reframe questions before diving into a user interview. Hypotheses & Goals, Draft Questions, and Sanitized Questions make up the 3 columns.
Hypotheses & Goals
Phrase questions as variables you are testing and controlling for.
Examples
- By simplifying the upload process, users can get more work done.
- This feature will decrease the drop-off-rate during onboarding by 15%
Draft Questions
Start by writing questions with no concern for bias. Address each goal or hypothesis in the first column before continuing.
Examples
- “How much easier is this design to use than the current system?”
- “How will this app decrease the time you spend on uploading documents?”
- “Why is the application process so painful?”
Sanitized Questions
In this last column go question by question to remove bias and focus on impartiality. Make sure questions are open ended and will continue to address what you are hoping to learn by doing this research.
Examples
- “What changes to the document uploader are most noticeable? Why?”
- “How would these changes impact your workflow?”
Impact
Leading questions result in inaccurate feedback. Participants may hope to please you or may assume you are more of an expert than you are, and may avoid pushing back against biased questions.
Avoiding leading questions will help you understand core user problems and make accurate design decisions. You won’t have to wait until the design validation phase to learn your design is ineffective or off course. You’ll hear more of the truth, not just what you want to hear.
Next time you’re in a research session and a colleague asks a leading question, calmly shout “Objection! Leading the witness….. sustained!”
Quintin Carlson researches and designs for Flexport — the first internet powered freight forwarder and supply chain logistics provider. Every Wednesday, the Flexport design team breaks down user experience research techniques so you can apply them in your own design process.
Want to tackle design problems that impact the world economy? Join our team.
If you found this article of value please click the ♡ below so others can find this post and follow Flexport Design.
