Are you asking the right questions?

Questionnaire administration is an invaluable way to collect information for a number of interests. Whether your research interests are clinical, academic, or in marketing, a well-designed questionnaire can provide a lot of information about your population of interest. Designing (and administering!) a questionnaire should involve a rigorous thought process — one that considers the biases of both the researcher and the participant.

If you do not know how to ask the right question, you discover nothing.” — W. Edward Deming

Let’s talk bias.

As a researcher, you need to be sensitive to the way the participant interprets the items on your questionnaire. Consider the following four problems.

You sit down to take a survey on attitudes toward recreational drug use and come across a question that reads:

“Do you agree that marijuana is not a dangerous drug and that it should be decriminalised or regulated?”

If you answer “no,” are you disagreeing with the statement that marijuana is not a dangerous drug, that it should be decriminalised, or that it should be regulated? This is the sort of issue caused by “double-barreled phrasing.”

Now imagine that you are filling out a questionnaire on study habits and you arrive at the following question:

“Do you often do poorly on exams because you tend to spend more time procrastinating than studying?”

In this case, a “yes” or “no” answer could be agreeing with one or both of the statements in the question — doing poorly on exams or spending more time procrastinating than studying (or both)! To collect info on study habits, and not test performance, this question needs to be rephrased to avoid the issue of ambiguity.

In our next scenario, you are taking a survey on risk behaviours and attitudes toward gambling:

“You are given $20 to play the game of your choice. You can choose to play Game A or Game B (both cost the same to play). Which do you choose?”

Game A: You have a 90% chance of winning $200.

Game B: You have a 5% chance of losing $20.

If you’re a discerning reader, you probably picked Game B, since your chance of winning is 5% higher than your chance of winning Game A (remember, both cost $20 to play)! But what if the survey were timed? Under a time constraint, you might feel pressured to make a quick decision, and without thinking carefully, Game A definitely sounds more attractive than Game B (a “90% chance of winning $200” as opposed to a ‘10% chance of losing $20’). Framing of individual questions can have a major impact on the decision-making process.

To wrap up our discussion of bias:

Questionnaires are a bit like conversations — if they drag on and become one-sided, it’s difficult to avoid passively nodding until there’s a break in the dialogue. Similarly, if all of the items on your questionnaire can be answered with ‘yes’ or ‘no,’ it’s likely that you’ll observe the phenomenon of “nay” / “yea” -saying (also known as “response bias”). If you have a questionnaire with items that can all be checked ‘yes’ or ‘no,’ you may want to include some negative and positive statements to avoid having the respondent exclusively tick “yes” or “no.”