Avoiding bias in your user interviews
What not to ask when conducting user research.
User research can be used effectively to understand user needs and behaviours in the product design process. The actionable insights from User research helps us build successful products. In a country like India, where diversity is the driving force — it is imperative to conduct in-depth research to understand our users.
At Flipkart, we regularly conduct generative and evaluative studies. These researches follow the traditional method of interviewing with users. It was during one such interview that I realised how leading questions can affect the insights that we get. Simple things like hand gestures, body language, a word here and a smile there, may affect the user from being entirely honest with us, leading to corrupted insights. Hence, I’d like to shed some light on what may create biases in such interviews which might have expensive consequences for the business.
Leading questions are those which create a bias in the users’ mindset while answering. Hence, they should be avoided while conducting interviews.
Here are some of the insights I’ve gathered through my experience with research moderators and classified leading questions –
1. Choice of words
In a conversation, use words that do not lead to a biased response.
Incorrect : “I am going to show you an app. Hope you will like it.”
Words such as ‘like’ lead the user to think in a specific direction. In this instance, the user would only think about the generic likability of the product in her viewpoint.
Correct : “I am going to show you an app. Tell me what you think about it.”
2. Options
Initiate with an open-ended question and then deep dive into specific themes or options. Don’t start asking questions directly by giving options. If forced to choose, people will choose one option , even if none of the options match their opinion. This might leave out the important information which you never thought could help with your product.
Incorrect: Which feature is most important to you: X or Y?
The two options given will make the user pick one and it might lead to missing the most important feature. This will mislead the analysis.
Correct: What features do you use? Which feature is most important to you?
3. References
Keep the interview to the context of the person you are talking to. Questions with a reference to another participant’s interview often leads to binary responses, creating a bias in the participant’s mind. The response is likely to be similar to that of the other participants.
Incorrect: I am talking to other participants as well and they said this is a good product. Do you also agree with that?
Correct: What, if anything, you like about this product?
4. Follow-up modals
Follow up questions are important if you want to know why users are behaving in a particular manner. Ask the user to show and describe the process instead of introducing new words. Make sure you use the terminology used by the participants and frame a question accordingly.
Q : How do you buy clothes?
Reply : I click on this (pointing to one icon), see tops and whichever top I like, I buy.
Incorrect [Follow up] : So you browse the product and add to the cart. What do you do in cart?
Introducing words without knowing the user’s terminology might confuse the user or it might lead to the user using that word. You will miss the true terminology used by the user.
Correct [Follow up]: Could you show me how do you shop online?
5. Additional information
Never try to educate or inform the user about a feature of your/the product or which the user is not aware of while interviewing. Treat the participants as people who have some awareness about the subject and try to gather what information the user has currently about the product and its features.
Incorrect: Do you know this feature exists?
Correct: How would you go about it (flow)? Follow up What do you think of this (feature)?
6. Tone & Body language
Maintain a neutral tone while having a conversation. Users participating in any interview should not feel that you are expecting a particular answer or their answer is wrong. Don’t point fingers, laugh or try to reinforce your viewpoint.
Incorrect: Don’t you think that this would be better if it was available on smartphones?
This implies that the person asking the question will disapprove with the user if they hear otherwise.
Correct: Is there any other way you might use this feature? [Follow up] What do you think of the feature availability on a smartphone?
7. Positive affirmation
Use a neutral tone like ‘Ok’, ‘tell me more’.
In the beginning of the interview, the researcher informs the user that there are no right and wrong answers and feel free to express their opinions. In that case, never use a biased filler or tone with leading and positive affirming terms like “correct”, “Well done”, “awesome”. It will give a perception that the researcher approves their answer and they may not say the contrary response.
It is difficult to retain non-leading question within a conversation. It requires constant vigilance on the part of the person asking the questions. Being aware will eventually help in getting non-biased insights which will help the product and business.
References
Interviewing users by Steve Portigal (Rosenfeld Media)
