Asking the right questions

We’ve all done it. We’re working on a project and need to understand a problem better.

So we design a survey, and try to get as many responses as we can.

Because the more data we have, the more we will know, right?

Not always.

Asking 30 people 30 questions doesn’t sound like that much, but it will generate 900 answers.

And without careful survey design, that could mean that firstly, you need to review 900 individual pieces of data, but that secondly, the value of your data is diluted as people cannot answer the questions accurately.

Mobile Data Collection in Uganda (Source: IFRC)

Here’s a few examples

You are conducting a survey around community health in a rural part of Kenya, and need to know the demographics of the population. You ask a free text question ‘age or year of birth’.

• Your first answer is ‘59’. So is this person 59 years old, or are they born in 1959?
• One answer is a 3 digit entry of 133 — is that a typo?
• Someone has written ‘thirty’ in letters. Now you need to convert that to numbers.
• Someone else’s answer states ‘the year of the locusts’ — hmmm, you’ll need some local knowledge to understand this one.
• And as you only needed to know if the respondent is a child, working age adult, or older person, you now have some work to do to group the individual answers.

You make a survey to try and understand barriers to gender equality in the UK. Your compulsory question about to what degree menstruation impacts upon a typical working day, is asked to everyone, including those who do not menstruate. The result?

• An older person gives an answer based on their experience from half a century ago. Does this help you know more about the impacts today?
• Someone who does not menstruate, gives an answer based on their perception of their partner’s experience.
• Others select the middle-ground option of 3 on a scale of 1–5, because they do not know, but must answer the question to move on.
• Others close their browser, because they think the survey is not relevant to them.

You are reviewing hygiene traditions and behaviours following a local cholera outbreak in a small town in Cameroon. Your question about when people think they should wash their hands does not allow for multiple selections. Most people select ‘To prevent Covid-19’.

• This may be linked to a community engagement session about this topic held last week.
• This may be because of the high levels of media coverage around Covid-19.
• This may be because it is the first answer on the list.

You want to measure levels of child malnutrition in different areas of sub-Saharan Africa, to decide where to base a new child health programme. You decide to use a tried-and-tested metric to be able to compare the areas.

• You need to ask questions around 7 food groups eaten in the last 24 hours.
• But because you’d like to know more, you break these groups down further, and add in others that you think are missing.
• However due to food group overlap, and the use of colloquial language that not everyone understands, respondents are confused about which options to select.
• Also, one of the 7 original groups gets accidentally missed. Now you cannot calculate the metric you originally needed to enable the comparison of areas.

All of these are real examples of survey responses I’ve worked with at the British Red Cross and elsewhere, where additional efforts were then needed to make sense of the responses.

Sometimes the data could be rescued and was still useful. Other times, the only option was to start again.

The key to getting the best out of your survey lies in 3 steps,

1. Identify the problem you are trying to solve, and ensure your survey questions reflect only this. Don’t try to understand or fix the world with one survey.
2. Less can be more. Keep it short, and use simple language.
3. Design your survey journey to ensure the flow of questions make sense for all. Where possible, avoid free text answers.

By spending a little time carefully designing and testing your survey, the data you gather will be useful and informative, and able to tell a story. It will be ready for your analysis without the need for extensive data cleaning blocking your progress. Your future self will be very grateful!