5 Research Biases: How to Identify and Avoid Them in Your Research
Research bias is a critical consideration in the interpretation of market research data. Without it, businesses run the risk of making decisions using imperfect or incorrect information. In business today, there are several reasons why bias is often ignored: Short timelines, unique market conditions, no budget, new areas of study and other reasons tend to discourage taking steps to understand outcomes without bias.
This article seeks to re-examine some basic types of bias encountered when conducting market research and briefly explore some ways to avoid making biased decisions.
A general definition of bias is:
A particular tendency, trend, inclination, feeling, or opinion, especially one that is preconceived or unreasoned. Dictionary.com
Note the lack of objective terms to describe this concept. Tendency, trend, inclination and preconceived are all forms of imprecise guessing. Put simply, bias is human error. As we construct market research studies or interpret research data, it can be easy to unknowingly include biases. Make no mistake; data on its own is meaningless without vigorous analysis and thoughtful interpretation. All the more reason to guard against disaster: Budgets poorly allocated, branding misdirected, product features ignored, irrelevant advertising campaigns, and lost revenue.
Measuring the effects of bias is challenging and more art than science. So, it is worth examining some biases and identifying ways improve the quality of the data and our insights.
Social Desirability bias is present whenever we make decisions to put ourselves in the best possible light and make socially acceptable choices. Quantitative surveys face a challenge when asking personal/sensitive questions. Simple questions like “how much weight have you lost in the past month?” will elicit the respondent’s personal feelings about their weight. Typically, respondents will over report weight loss on a question like this and thus results are often skewed. Questions should be written to minimize bias and focus on unconditional positive regard (i.e., there are no “wrong” answers). By modifying the question above to ask the respondent’s agreement on weight loss statements and capture results using a neutral scale: “I think I lose more weight per month than others” (agree/disagree) social desireability bias is minimized.
As with other biases, control of social desirability bias is often managed in real time in qualitative studies. To be considered acceptable, participants in a focus group may answer based on the comments of another member or may respond in a way that protects their feelings or beliefs. Reframing questions indirectly, such as asking participants how a third party feels allows them to project their own feelings onto another and promotes a more truthful answer. Using unconditional positive regard by Informing the subjects that there are no wrong answers can also increase a participant’s likelihood to answer according to their real thoughts and feelings.
Confirmation bias is one of the most common forms of research bias. It happens when information is interpreted using a previous assumption or hypothesis rather than letting the research results drive conclusions, that is, letting the data speak for itself. This is a common error that occurs among researchers and their audiences. Unfortunately, it is harder to combat this bias in data interpretation. Internal politics, personal goals or simply lack of knowledge can turn research users into cherry pickers who will consider certain results and ignore others. It is not uncommon for a research analyst to present findings and conclusions to a client who has a preconceived notion about what the results means. Relying on expertise or past research studies may affect how he processes the new results.
Qualitative research has its own set of challenges. In qualitative research, confirmation bias can take place at the moment of execution and can extend to focus group observers and into the analysis. Quantitative data gives you the opportunity to step back, reassess and reinterpret considering confirmation bias. Qualitative research should evaluate participants’ impressions, attitudes and beliefs in real time. When clients are observing a focus group, promoting their own desired hypotheses, it can be daunting to maintain good moderation practices that dissuade the biases of the observers.
Irrational escalation motivates us to ignore new research results if they override or undermine research decisions already made. Sometimes, senior management, brand managers or product managers get so invested in a product idea that regardless of what the research says, they still will do what they intended to do. They often dismiss the research and try to poke holes at the methodology. Objectivity and a willingness to listen to the results is the key to promoting non-biased interpretation. Quantitative research can guard against irrational escalation by ensuring questionnaires are written without leading, vague or poorly worded questions. An audit of previous research studies and published data may provide another check for misdirected assumptions.
Irrational escalation can manifest in qualitative research with a discussion guide that prevents respondents from expressing their own conclusions about hypotheses or key topics. In practice, researchers often summarize conclusions or opinions on behalf of the respondents. During a focus group, moderators should take precautions not to put words in the respondents’ mouths and encourage them to express their own opinions. In many cases, we need to loosen our hold on confirming every hypothesis in every focus group, and instead, build rapport to summarize understanding using the participant’s voice.
Cognitive framing occurs when outcomes are different when presented in different ways. Question order bias and leading questions are variants of this bias and provide a helpful example. In our quantitative surveys when we present a list of question responses in the same order for all respondents to see, we know that the static order will impact how a respondent answers that question. Items always listed first and last will receive more attention than other items on the list. Such bias, if not accounted for, can skew question results. Today, well-designed survey instruments typically have automated randomization built into their platforms to guard against this hazard.
In an example of cognitive framing, patients awaiting surgery are asked a leading question offering two options.
-An operation that has a 5% mortality rate.
-An operation in which 90% of the patients will survive.
Patients may choose the second option when they see or hear the words “90%” and “survive,” but in fact a 90% survival rate (or 10% mortality) is worse than a 5% mortality.
When the research is qualitative, another set of safeguards should be put in place. Asking leading questions during the discussion will shape a participant’s response. So, using the respondent’s own words in asking questions or probes can help offset the tendency to use leading questions. For example, respondents will reply with a different set of brands if we ask them which clothing retailer or which online clothing retailer comes to mind. Much as we like to make our discussion guides “airtight,” ensuring all key points are covered in each group, reordering and rewording topics or dropping/adding topics across groups will preserve findings and limit bias.
Knowledge bias is common among research subjects already aware of products’ or services’ strengths and weaknesses. They are often very familiar with features of an old product and may not favor a product with new and better features. Put another way, people often won’t change their views even when it’s obvious the newer product is better. This bias can turn up whenever new products, services or messages are compared to previous versions. A question that asks respondents their preference for new PC features may generate knowledge bias from those respondents who already own a PC and who may not objectively assess the more advanced features of a newer PC. For example:
How a respondent answers the first question may affect their response with subsequent questions because of their previous experience with computer brand X. To counteract this, researchers should probe into respondents’ motivations behind likelihood to buy or switch brands.
Qualitative researchers can run afoul of this bias when introducing products, services, advertising and other stimuli. Moderators risk bringing their knowledge bias into the meeting room if they focus on satisfying the needs of the client at the expense of objective participant feedback. If a qualitative study is testing advertising relevance, a moderator may unintentionally focus on the positive statements of a current ad, biasing reaction to a new ad, one that has a higher degree of relevance.
Given the thousands of studies market research professionals conduct each year, it’s understandable how diligence in avoiding bias can be pushed aside. Streamlining questionnaires for a quick turnaround should not be done at the expense of longer, well-written questions that account for bias. Similarly, poorly prepared focus group moderators may be unable to manage biases as they occur. Successful group moderation requires a thorough knowledge of the subject matter and a tactical plan for how to redirect or reorder the flow of the group to avoid bias. Perhaps the best insurance against a costly bias decision is to leverage research professionals who can deliver carefully designed projects that are expertly executed.