A well-designed survey can provide insight into behaviour, sentiment and needs that are otherwise impossible to ascertain. A badly designed survey can provide, at best, confusion and, at worst, actively incorrect information. This can lead to wrong decisions and wasted time and money.
Understanding the common pitfalls of surveys is essential to avoid them. This article illustrates the fives ways things can go wrong if you ask the wrong questions.
1. Double-barrelled questions
Questions should only contain one concept to avoid confusion. A double-barrelled question is a question with more than one concept meaning that the respondent is forced to answer two questions in one. In the example below, the respondent can only agree with the statement if they found the training both informative and easy to follow. However, this doesn’t tell us whether the training was informative, but hard to follow, or whether the training was not very informative, but easy to follow and so on.
A double-barrelled question with multiple concepts to answer
Question asking about one concept only
This is one of the most common mistakes people make when writing surveys and is an easy one to fix!
2. Ambiguous questions
Questions should be formulated in a way that the respondents know exactly what is meant. It is best to avoid vague language such as “regularly”, “often”, “very much” as all of these are subject to interpretation. “Often” for one person might mean once a week, while for another, it might mean once a day. By using precise language, the respondent is much more likely to answer the question the survey intended to ask.
Question using ambiguous language that can be interpreted in different ways
Question using precise language
This one can be harder to spot and fix than the previous pitfall as we are so used to using these kinds of words in our daily lives. Sometimes, the presence of these words also indicates that a different type of question might be more useful here, for example a single choice question instead of a Likert scale question:
3. Leading Questions
It is very easy to accidentally write leading questions as we like to use adverbs and adjectives to make our sentences more descriptive. However, they should be used with caution in survey questions as they can inadvertently bias the respondent in answering a certain way. In the example below for the loaded question, we are leading the respondent to a more positive review of the training by qualifying it as “excellent”.
The way the question is formulated is not the only way to accidentally write a leading question. The answer scale can influence respondents’ answers as well. In the example below of a question with a biased scale, you can notice that there is no true neutral option like in the Good example. Instead, there are more positive options than negative ones, potentially making the outcome of the training appear more positive than it truly was. Balanced scales with a true neutral option are essential for good survey design.
Question with a biased scale
Question made leading through wording choice
Question formulated more neutrally, with a balanced answer scale
t is easy to check if your answer scales are balanced, but it might be hard to spot if a question is leading as this might be more subtle.
4. Loaded Questions
A concept related to leading questions are loaded questions. Loaded questions make assumptions about the respondent, which biases the question. In the example given below, we are assuming that respondents committed safety violations before doing the safety training. However, there is no basis for us to assume that this is what was happening. If we want to ask about the impact of the safety training, then we want to formulate a question that is free of assumptions and tests the outcomes of the training without making assumptions about how well respondents followed safety procedures before the training.
Loaded question: making assumptions about respondents committing violations before training
Question formulated more neutrally, without assumptions
5. Relying on questions to get information that can be obtained from more reliable sources
Surveys are inherently subjective as they rely on respondents’ self-reporting of their experiences, perceptions, attitudes, and feelings rather than observing them in a more objective way through their behaviour. While this is one of the limitations of surveys, it can be interesting to contrast the perception of respondents with the actual reality.
For example, we might want to know how often employees receive safety trainings. We could do this by asking the following question in the survey:
However, it would be much more reliable to ask the company for the safety training schedule and attendance records to understand what safety trainings are like at the company.
Some value can be derived from this question, if both the actual data (training schedule, attendance) is collected and the question is asked in the survey. It helps us build a picture of how people perceive the safety trainings, and how big the discrepancy between their perception and reality is, if at all.
In general, we do not want to waste too much valuable survey time on these “factual” questions that can easily be answered through different, and often more accurate, means. Instead, we would want to ask respondents about their subjective experience of the trainings and their frequency. For example, by asking the following question instead:
In Short
Survey writing can sometimes feel more like an art than a science, although it is a highly methodology-driven field with many other aspects to consider, such as survey length or passive versus active voice, which we have not addressed (yet!). It is also important to not be too dogmatic about any of the advice presented here, as sometimes preserving the flow of a survey or question is equally important.
In case you’re wondering, these screenshots are from our survey tool, Sense. Are you interested in creating your own survey? Contact us here — info@empirsys.io. We hope this post gave you a few ideas on what to look out for when writing surveys. Happy questioning!
If you found this useful, please let us know by getting in touch, give us a like or follow. You can find out more about us at empirisys.io or on Twitter @empirisys. And you can drop us an e-mail at info@empirsys.io or directly to the author of this article, Nyala@empirisys.io
