Surveys: when should you use them, and when to pass
“We want to get to know you better”, “What did you think about …?”, “Share your experience with us”, “We are curious about your opinion”… You see online surveys everywhere, and they are often top-of-mind if you want to learn something about your (potential) customers. When is it a good idea to use a survey for customer research?
An online survey quickly comes to mind if you want to find out which customer segments use your product the most or want to know how satisfied they are with it. It’s fast, cheap, and you get quantifiable results and statistics. Your decisions are therefore crystal-clear and evidence-based. Isn’t it fantastic?
A survey is very easy to set up. But it is even easier to set up a bad survey.
A survey is very easy to set up. But it is even easier to set up a bad survey. The fact that your customers answer a number of questions does not mean that these answers will help you make the right choices. On top of that, questionnaires with unclear or leading questions also provide incorrect information. Before you know it, you will be making decisions based on misleading data. I’d rather know that I don’t know something, than believe that I do, and be wrong.
For what kinds of questions should you use a survey?
With quantitative research (such as certain types of surveys) you attempt to learn numbers and facts, and gain statistical insight. The results are therefore often expressed in graphs and figures. With qualitative research (think of interviews or contextual inquiry) you look for experiences, reasoning and meaning. The results are usually descriptive and cannot be quantified.
Christian Rohrer, User Experience Design and Research Executive at companies such as SurveyMonkey, Intel Security and Ebay has provided insight into how different quantitative and qualitative research methods relate to the questions you can answer with them:
All methods can be placed on two axes: what people do and what people say that they do; and how qualitative or quantitative a method is.
Surveys are great for collecting and categorising the opinions, attitudes, demographics, self-reported data on how often / when something happened. Surveys can help you find out what demographic characteristics the target audience has, in what context they use your product, and what opinions and attitudes they have on different topics.
But is it that bad if you try to collect other information via survey, for example insight into the ambitions or motivations of your target group? Unfortunately, this can lead to results that do not represent reality. Even if you are aware that the results may be not representative of the truth, it becomes difficult to take nice-looking statistics with a grain of salt. Especially when someone summarises it so nicely: “70% of our customers say that …”. So better don’t do it in the first place.
For what kind of goals is survey a good method to use?
Do you have specific questions with quantifiable answers and hypotheses that you can confirm or disprove with numbers? If you need precision and statistical insights, you can certainly use a survey as a means. With it you can learn more about:
- who your users are, what kind of people use this product,
- when they use it,
- which customer segments there are and how big they are in relation to each other,
- what their top tasks are,
- how expectations relate to satisfaction, etc.
When do you use an online survey?
Before you launch a product, you can enrich the qualitative insights with knowledge about how often a specific problem occurs, who the target group is or make a baseline measurement of the current customer experience. In a continuous agile workflow you can use surveys for questions such as:
- find out which problems cause the most irritation (perception)
- get a better idea of the priorities of the possible improvements or features
- after a release of a new version, gauge how users experience the changes and where unexpected pain points may be.
What you can and cannot achieve with a survey
A survey can help you with the ‘what’: you can discover that in recent months an ever-growing number of your customers have become dissatisfied with your product, or that the target group that uses your app is slightly older than what you originally estimated. With surveys you can quickly gage where the points for attention lie for the product.
With a survey it is hard to get a deep and truthful insight into the “why”: why has the satisfaction dropped? What makes the app more attractive to this age group, and why does it appeal less to the other age group? This can be done with the help of qualitative methods, such as interviews, observation and usability tests.
In general, it is better to observe how customers use your site and perform tasks than ask their opinion about it. This is because all people have cognitive biases in predicting or explaining their own behaviour.
In my own experience I saw the potentially misleading effect of surveys. During a usability test, one of the participants completed the task effortlessly. Another one had to make several attempts to complete the assignment. In the end of the test we asked some general questions about how they experienced the usability research. Both gave the same score for ‘ease of use’, average. I was mind-blown. I assume they both gave the same score because the second participant was less technically savvy, and using technology in general is a bit slower and error-ridden; while for the first participant, who is at ease with technology, even the smallest hiccups in usability amount to a lot of irritation.
Through observing what they did I knew what the problems the interface had. Those were pretty actionable insights for me. However, if I were to base my next steps on rating that they gave, without any other context (just as you would get it from a survey) you cannot find out exactly what can be improved, only that it should be done.
Surveys are awesome, but not for every single purpose. Proceed with care :)
This post was originally published on oneshoe.nl (in Dutch)
