Question order bias exists. Use it.

Arthur Grau IV
Jan 28, 2018 · 10 min read

What if I told you that by just asking you questions in a different order I could influence your survey response by nearly 40 percentage points. As a logical and intelligent person, you might laugh this off as trickery. There is plenty of research, however, to prove otherwise…sort of. Question Order Bias is well documented as related to surveys and questionnaires. (1,2) There is little written, however, regarding question order bias during the qualitative personal interview, a staple design research method.

Quantitative researchers agree that order effect bias, as it’s also known, should be avoided or controlled when possible, and accounted for when not. But, I see question sequence and the bias it promotes, as an essential building block of the personal interview, as long as I know how to use it.

The ethnographic interview is a foundation to today’s design practice. (3) Proponents claim that ethnography offers a richer understanding of consumers, users, or stakeholders than does traditional research like surveys, P.O.S. statistics or other quantitative methods. (4) While question order bias is seen to be avoided in the context of the survey, let’s see how it can be used to help us learn what we need to know while helping our informant to feel at home with us. First let’s understand this bias in surveys and questionnaires.

A Few Shades of Question Order Bias

Question framing is one way that order can influence outcome. For example, imagine I am asking you to rate your interest in sports first on a scale from one to ten. What would your rating would be? Now, imagine I ask you a series of twelve specific questions about teams, scores and players of various sports activities during the current season. Now after this question series, I ask you to rate your interest in sports on a scale from one to ten. Do you think your rating would have changed? Research shows that you would rate your interest far lower once the rating question was framed against the series of specific subject area questions. This may happen because you might question your own ‘interest’ after knowing what depth the query was asking for. (Lavrakas, P. 2008) Framing may also be seen in a binary effect when one question is posed alongside another to establish a norm.

The order effect bias of establishing norms or reciprocity is cited in our opening as the effect that could be influenced by almost 40 percent. During the cold war, Americans were asked whether or not journalists should be allowed to travel in both directions across the “iron curtain” (between the US and the USSR) to bring back news stories to their home countries. When respondents answered the question first regarding American journalists traveling to Russia, they were 37 percent more likely to support the reciprocal travel of Russians also traveling to America for the same purpose of bringing news stories home. (5) When the question regarding Russians traveling to the US was asked first, American respondents tended to disapprove more, and subsequently answered for less American travel reciprocally as well by a factor of 24 percent.

Though this early documentation of the order bias remains widely cited, the research field now overflows with viewpoints and nuance on the subject. At Pew Research they are respectably transparent about their discovery of bias in their own research. According to Pew, open-ended questions that follow closed-ended questions tend to bias toward concepts or considerations already offered in the closed-ended questions. For this reason, in their survey work they most often place open ended questions first. (6) They group the effects of this bias into the categories ‘contrast’ and ‘assimilation’.

These contrast and assimilation effects depend on question order. They are closely related to reciprocity effect above and manifest when respondents contrast their answers based on what they were “primed” with from a previous question as in this example from Pew Trust in the Notes section of this article.

What Pew calls assimilation and contrast, are also subtly redefined by Gallup as ‘additive’ and ‘subtractive’ order biases. (8) This may also be closely related to controversial order bias. At the Centre for Biomedical Ethics in Oslo Norway, researchers identified up to a 20% question order bias on questions related to physician assisted suicide. Respondents who were given more controversially worded questions first were generally more in agreement with subsequent questions that were less controversial. (9)

The Many Shades of the Ethnographic Interview

There are other identifiable and subtle order bias effects in surveys and questionnaires. (10) But what does this mean for the in situ, personal ethnographic interview? Based on research, discussion with practitioners and conducting many primary interviews myself, I propose that the question order bias is an essential component in the interview toolkit, if used with awareness.

In a personal interview there is more nuance and mutability. You can change the order of questions on the fly and even respond with new impromptu questions based on your informant’s feedback. According to the textbook by Perreault et al, “The researcher must be careful to place questions so that [ze] evokes and maintains the respondent’s interest, stimulates [zer] attention, and in some cases even overcomes resistance to answering questions.” (11)

In living design research, maintaining interest and attention is only the starting line. In every case, working for the first time as a stranger to the interviewee, I surely come up against some resistance. In the short span of a single interview I may probe into personal subjects and aim to get at core beliefs. (12, 13) I may also be asking my interviewee to perform tasks, or to share artifacts from their environment. In this setting, we can leverage question order to help the respondent become familiar with us and to help us move them toward the kinds of sharing that we seek.

Leading with general questions, and unlike during a survey, I respond with pointed and guided follow up to my informant’s answers. This is the art of the interview that far extends the scope of questioning beyond what a survey could ever offer. The detail and insight available as a result of this kind of research is precisely why organizations are willing to invest much time and resource in this type of ethnographic research. (14)

Now, let’s suppose I ask you to choose first what you like out of a series of choices, and then I ask you why you like it and to help me understand it. You are more likely to positively associate your knowledge of the topic and your interest in explaining it, than if I had asked you the questions in the other order. In this case, liking promotes understanding. There is a liking-driven order effect when related to understanding and the way question is asked. (15)

This is where it could get sticky. I have to be very aware of question order, and other biases as I proceed. I want to avoid steering you into a confirmation bias, or leading you with loaded questions to some hypothesis I’ve already formed in my mind. Instead, I should try to find order in our questions that will first allow you to form some sort of hedonic response. I should be able to see your emotion and attachment or detachment to a subject, question or series of questions. It is exactly this kind of response that will help me to get to an understanding of your needs, jobs to be done, and beliefs. These same attributes of your answer will help me to design something for you or the persona that you come to represent in my research.

The framing bias may be terrible for surveys, but it is essential for the interview. Given our short visit together, I need to actively frame the questions I am hoping you will answer. I am not looking for ratings on a scale or numbers to aggregate among hundreds of other respondents. I am also not looking for you to endlessly ramble in friendly conversation. I am looking for triggers and associations, for reasons why. I will later review and contextualize your responses and combine them with those of other interviewees to arrive at themes and insights. If I can manage to similarly, and carefully frame my discussion questions across all interviews, I may arrive at some valuable information. And unlike a survey, I will actively frame my question, because my observation of your response becomes part of the data set.

Norms and reciprocity may also be useful in the living interview. For one thing, I can repeat and reframe a question over again…and sometimes, over again. Meaning that if I detect a reciprocal bias in a previous answer that you gave, I may go back and re-ask the question from another point of view. I may refer to your previous answer and echo it back to you so that you can hear what you said. I can ask why, and ask why again.

Lastly, regarding shocking or controversial questions, I may have to use these in the deep interior of the interview in order to get a reaction from you. This is where I can get down to your thoughts, core beliefs and values from your words, decisions and actions and really see into what you may mean.

When to use it.

To sum this all up, I do see order effect bias as a real and somewhat controllable research effect that should be considered for surveys and questionnaires. In the ethnographic process it is a useful, if subtle tool for the personal interview. Much more research and practice is needed to hone my own skill in the personal and ethnographic interview. There are plenty of recommendations on controlling effect order bias for researchers and designers as referred to in the appendix.

So the next time someone asks you a series of questions, watch closely the order and observe how it may be influencing your response. This may help inform your own design practice while creating new research.

References

  1. Lavrakas, Paul J. (2008). Encyclopedia of Survey Research Methods. Thousand Oaks: SAGE Publications, Inc. pp. 664–665. ISBN 9781412918084.
  2. Blankenship, A. . Journal Of Marketing, 6(4), 66–75., Albert (April 1942). “PSYCHOLOGICAL DIFFICULTIES IN MEASURING CONSUMER PREFERENCE”. Journal of Marketing. Vol. 6 Issue 4, p66–75: 9 — via Business Source Complete
  3. Salvador, Tony; Genevieve Bell; and Ken Anderson (1999) “Design Ethnography,” Design Management Journal (pp. 35–41). p.37
  4. https://www.bloomberg.com/news/articles/2006-06-04/the-science-of-desire
  5. Hyman and Sheatsley, Herbert Hiram and P. B. (1950). “Current status of American Public Opinion”. National Council for the Social Studies Yearbook. Vol. 1950, p11–34: 24 — via Education Source
  6. Questionnaire design”. Pew Research Center. 2015–01–29. Retrieved 2017–11–18.
  7. See Appendix
  8. Moore, David W. Measuring New Types of Question Order Effects- The Gallup Organization: Public Opinion Quarterly Volume 66:80–91 2002
  9. Magelssen et al. BMC Medical Ethics (2016) 17:24 DOI 10.1186/s12910–016–0107–3 Attitudes towards assisted dying are influenced by question wording and order: a survey experiment
  10. See Appendix
  11. Perreault, W. D., Cannon, J. P., & McCarthy, E. J. (2010). Essentials of marketing: A marketing strategy planning approach.
  12. Personal interviews 2012–2017
  13. Benjamin Little, MassArt MDes coursework and lectures, 2017
  14. https://hbr.org/2015/09/design-as-strategy
  15. Pelowski, Matthew et al (2016). Understand after like, viewer’s delight? A fNIRS study of order-effect in combined hedonic and cognitive appraisal of art. Acta Psychologica 170 (2016) 127–138

Appendix

Resources on order effect bias

qualtrics.com/blog/biased-data-is-bad-data-how-to-think-about-question-order/

surveymonkey.com/curiosity/question-order-matters/

www.pewresearch.org/methodology/u-s-survey-research/questionnaire-design/

researchaccess.com/2014/08/order-bias-is-a-larger-source-of-error-than-you-think/

en.wikipedia.org/wiki/Response_bias

rmsresults.com/2011/10/31/question-order-bias-in-customer-satisfaction-surveys-syracuse-ny/

surveygizmo.com/survey-blog/survey-question-order/

fluidsurveys.com/university/survey-priming-question-order-giving-survey-bias/

smashingmagazine.com/2016/08/user-memory-design-how-to-design-for-experiences-that-last/

Pew Research on Contrast Effect

“An example of a contrast effect can be seen in a Pew Research Center poll conducted in October 2003 that found that people were more likely to favor allowing gays and lesbians to enter into legal agreements that give them the same rights as married couples when this question was asked after one about whether they favored or opposed allowing gays and lesbians to marry (45% favored legal agreements when asked after the marriage question, but 37% favored legal agreements without the immediate preceding context of a question about gay marriage). Responses to the question about gay marriage, meanwhile, were not significantly affected by its placement before or after the legal agreements question.”

Questionnaire design”. Pew Research Center. 2015–01–29. Retrieved 2017–11–18.

Question order bias as part of Response Bias on Wikipedia (edited contribution by author, 19 Nov 2017)

Question order bias, or “Order effects bias”, is a type of response bias where a respondent may react differently to questions based on the order in which questions appear in a survey or interview.[22] Question order bias should not be confused with response order bias that addresses specifically the order of the set of responses within a survey question.[23] There are many ways that questionnaire items that appear earlier in a survey can affect responses to later questions. One way is when a question creates a norm of reciprocity or fairness as identified in the 1950 work of Herbert Hyman and Paul Sheatsley.[24] In their research they asked two questions. One was asked on whether the United States should allow reporters from communist countries to come to the U.S. and send back news as they saw it; and another question was asked on whether a communist country like Russia should let American newspaper reporters come in and send back news as they saw it to America. In the study, the percentage of “yes” responses to the question allowing communist reporters increased by 37 percentage points depending on the order. Similarly results for the American reporters item increased by 24 percentage points. When either of the items was asked second, the context for the item was changed as a result of the answer to the first, and the responses to the second were more in line with what would be considered “fair,” based on the previous response.[25] Another way to alter the response towards questions based on order depends on the framing of the question. If a respondent is first asked about their general interest in a subject their response interest may be higher than if they are first posed technical or knowledge based questions about a subject.[25] Part-whole contrast effect is yet another ordering effect. When general and specific questions are asked in different orders, results for the specific item are generally unaffected, whereas those for the general item change significantly.[25] Question order biases occur primarily in survey or questionnaire settings. Some strategies to limit the effects of question order bias include randomization, grouping questions by topic to unfold in a logical order.[26]

Image Credits:

“Order” Image courtesy of https://www.flickr.com/photos/svenwerk/ cc license

“Conversation”: Image courtesy of https://www.flickr.com/photos/p_valdivieso/ cc license

MassArt Innovation

Stories and opinions from the students, faculty, and friends of MassArt’s Design Innovation program.

Arthur Grau IV

Written by

Living in a better world arthurgrau.com

MassArt Innovation

Stories and opinions from the students, faculty, and friends of MassArt’s Design Innovation program.