What to watch out for when creating a survey or questionnaire

Bias is a very common occurrence as mentioned in one of our previous posts, and it is very easy for it to creep its way into your survey questions. Have you ever sent out a questionnaire and found that you did not receive the type of answers you wanted, or that people completely misunderstood a question? Don’t worry, you’re not alone. If you want to collect the most accurate data from respondents you must understand and be able to prevent (or at least minimise) bias in the design of your questionnaires.

10 types of bias in question design:

1. Ambiguous questions

If you’ve ever taken a survey and wondered what exactly they were asking you, it was most likely an ambiguous question. The reason why ambiguous questions are problematic in surveys is because they allow respondents to interpret the meaning of the question however they like.

An example of this is: Where do you like to shop?

When writing a question, think about what your answer might be? How may others respond? In this ambiguous question it is not exactly clear as to what kind of shopping the question is referring to. Is it asking you where you like to do your regular clothes shopping? Or maybe it’s asking you do you prefer to shop online or offline? What kind of shop? This question has done a poor job of explaining the criteria for ‘like’. What exactly does ‘like’ mean in this context? Perhaps you are asking which store has the friendliest staff or the best selection or value for money.

When writing satisfaction survey questions, here are four tips to avoid ambiguous questions:

• Avoid words or phrases with multiple meanings
• Specify the context of the question
• Watch for similar spellings or pronunciations of key words
• Be direct about what you’re asking

2. Complex Questions

Complex questions should always be avoided in questionnaires.

Here’s an example for you:

At first glance, you may not consider this a complex question. But on closer examination, you can see that it is vague and lacks context. Whose point of view is this from? Not all of your survey respondents will understand what you mean by ‘soft drink distribution’. Like most other common survey pitfalls, this is an easy fix! Start by asking yourself what you’re really trying to understand or uncover, then focus on using ordinary words whenever possible and write survey questions at the appropriate reading level for your survey respondents.

An easier way you can ask this is: “Are soft drinks easy to find whenever you want to buy them?”

3. Double-barrelled questions

A double-barrelled question is when two questions are combined into one. Survey creators do this in an effort to reduce the number of questions in their survey. Unfortunately, doing this drastically reduces the reliability of the collected data and the types of conclusions that can be drawn from survey data analysis.

The problem with double-barrelled questions are there are no way to tell what the survey respondent was rating. Was the survey respondent very happy with the phone company rates, so they said they were somewhat happy? Did they give an average as to how they felt about the rates and the customer service? Did they only answer one of the questions? There’s no way for you to know if you keep the two attributes together in one question.

4. Technical jargon

Regardless of who’s taking your survey, use clear, concise, and uncomplicated language while trying to avoid acronyms, technical terms or jargon that may confuse your respondents. Make sure to provide definitions or examples if you need to include tricky terms or concepts, that way you can be certain that almost anybody can answer your questions easily and that they’ll be more inclined to complete your survey.

Example:

You should strive to write questions using language that is easily understood. Certain sample groups, however, may have a knowledge base that can make the use of more difficult terms and ideas.

5. Uncommon words

To avoid misunderstanding, common words should always be used in questionnaires. If you can, have someone unrelated to the creation of the survey read over the questions to see if they are able to quickly understand the questions or not.

Here are some examples for you to consider:

6. Vague words

By using vague words in survey questions, you will find that this will encourage vague answers in response.

Example:

How would you answer this question? What type of graduate is the question referring to?

Try to avoid using vague quantifiers such as ‘regularly’, ‘often,’ ‘rarely,’ etc. These quantifiers don’t convey the same meaning for all respondents. Depending on the question asked, one respondent may interpret “regularly” as once or twice a week and to another it means five to six times a week.

7. Belief vs. behaviour (also known as hypothetical question or personalised question)

Hypothetical questions are based on speculation and fantasy.

Example:

Questions such as these force respondents to give their ideas on a particular subject. Generally the data collected is inconsistent and unclear. As the investigator, you must determine whether the purpose of the question is to collect data regarding a belief or a behaviour and design the question accordingly.

8. Insensitive measure

This occurs when your outcome critera make it impossible to detect significant changes or differences.

Example:

The issue with the first question is that it may not have sufficient discriminating power to differentiate the respondents because of the limited categories. The second question is better as it allows for more accurate and telling results, but often a limit of 5 answers is ideal depending on the question.

9. Forced choice

Questions that provide too few categories can force your respondents to choose imprecisely among limited options.

Example:

The first question which does not have a “I don’t know” category, may produce a bias because your respondents who have no opinion are forced to select an answer that may or may not reflect their true feelings. The second question is recommended as it prevents this.

10. Framing

Some questions may be framed in such a manner that lead your respondents to choose an inaccurate answer.

Example:

Patients scheduled for surgery may choose the second option when they see or hear the words “90%” and “survive,” as these are positive associations, but in fact a 90% survival rate (or 10% mortality) is worse than a 5% mortality rate.

When you write your survey questions, you’ll find it’s the small nuances that make a big difference in the context of the question. Subtle changes make a big difference in how your survey respondents read and interpret your question. If you keep these recommendations in mind when writing your survey questionnaires, you shouldn’t have any problem avoiding these common survey question pitfalls.

If you’re crafting a survey and need help avoiding these common pitfalls, get in touch as we would love to assist!