Writing Good Surveys Is Harder Than You Think Pt 1
Part 1 of 4: Making a plan
Writing a good survey is easy, right?
Actually… no, not necessarily. But you’d be forgiven for thinking that it is. The proliferation of surveys out there has meant that almost everyone has had a go at writing a survey.
Like many research methods, writing surveys is thought to be something anyone can do, but there’s more to it than meets the eye.
We’ve all had to endure bad surveys… so what makes a survey a good one?
From a respondent’s perspective, a good survey:
- doesn’t disrupt what I’m doing;
- is short, or else incentivises me for my time;
- lets me know up front how long it’s going to take;
- is designed to fit the device I’m completing it on;
- uses language I can understand and relate to;
- asks me questions I can actually answer;
- doesn’t put undue demands on my memory;
- lets me know where I am in the process at all times;
- lets me have my say;
- protects my privacy;
- thanks me for my contribution; and
- doesn’t reappear if I've dismissed it or done it already.
And from the survey writer’s perspective, a good survey:
- measures what it claims to measure;
- targets the appropriate respondent population;
- is served through a medium appropriate to the questions being asked; and
- enables straightforward data analysis and interpretation.
Let’s explore these in more detail…
Before you start…
There are five key things you need to consider before you begin writing a survey.
- What are you trying to find out?
It’s important to first define what you are trying to find out. This will help you identify who you are targeting, and might also help you decide on the best delivery method.
It will also help you structure the survey and decide on the best types of questions to get the answers you want. Write a plan of the things you want to learn, the goals of the exercise, and the overall broad-brush questions you want to answer.
For example, are you trying to get preferences on specific items? Do you want reactions to stimulus materials? Do you want respondents to compare things, or describe something? Do you want them to report back on their behaviours? Do you just want to know how they feel?
2. Who are you targeting?
When considering what respondents you are trying to target, ask yourself if you want to survey a sample of the entire population, or just users of your product or service. What demographic criteria should they meet, if any? What other variables are relevant to the objective of the survey and what you’re trying to learn?
Knowing who you are targeting will help you ascertain the best way to deliver the survey to them. It will also help you determine the language and tone of voice to use in the survey.
3. How do you plan to deliver the survey?
How you deliver the survey is a strong influence over the survey questions. For example, if you are targeting users on-site, then it’s important the survey does not disrupt their tasks and is as short as feasible.
If you’re sending an email survey and asking respondents about something they’re not currently using or doing, you may need to provide images or other prompts.
Typical survey delivery options are via email, on-site (or site intercept), on mobile, over the phone, or in person.
4. How are you going to analyse and interpret the data?
These days, most survey tools are sophisticated enough to do the analysis for you, however you will still need to interpret the data and derive insights.
You also need to think about what questions will enable the analysis you want. For example, you may need to think about whether you want to compare responses between different customer or demographic segments, in which case you will need to ask segmentation questions. You may want to contrast responses between desktop and mobile users, in which case you will need to ensure your survey tool naturally splits these out and enables comparisons, or else plan to brush up on your Excel skills.
It really helps to do a ‘pilot’ analysis with data from colleagues or friends — this can help you identify missing data from questions you may not have thought about.
5. What quota(s) do you need to meet?
Be aware of response rates when delivering surveys… if you want a certain number of responses to give you/your stakeholders confidence and/or to achieve statistical significance, you will need to ensure that you target a much larger number of respondents.
Response rates can vary by delivery method, audience, survey length and whether an incentive is offered or not.
Part 2 talks about how to structure a survey.
