Hacking insights using survey

Now, you have interviewed a couple of respondents. You get a lot of insights and identified several key of improvements. Then, you share your learning to key people in the company who can make those improvements happen. But guess what, they question about the reliability of your entire findings because you only interviewed how many? 5? 9? 12? Out of millions of your users.

The same concern that we-UX researchers- often have to face again and again is that: We cannot base our company decision solely from talking only to a bunch of people. An inspiring friend once wrote that we can only choose 2 out of 3 things: cost, quality, or time. We do not want to argue with quality. Time is also the luxury that we don’t have, so we know where we’re standing: we choose quality and time to keep.

For me, in terms of research, the options shift a bit where you can only choose 2:

a. In-depth understanding

b. numbers, or

c. time.

Which one is the priority? Again, for an agile company like GO-JEK, time is the luxury that we don’t have, so we have to either choose between an in-depth understanding we get from several people or a surface learning we get from thousands of user. It is a tricky option because most of the time the case is: we are expected to get both in a short given of time. But guess what? We can cheat the situation by doing it two rounds: First, do in-depth interview with several people, get the deep understanding, absorb the stories. Next, after identifying the pattern of your answers, then validate the result with quant survey to get the numbers.

Of course, doing it in two rounds won’t be as quick as just conducting a single survey only. However it is much more effective than doing interview to hundreds and thousands of user. Doing it two rounds is absolutely an option you should consider if you are aiming for both in-depth understanding and also statistical power from your research.

Okay, it’s pretty convincing. Now, how are we going to do this?

1. Define your survey objective. First thing first: you have to clearly identify the goal of your survey. What are you going to do once you get the data. Are you going to validate your findings from previous research?Or do you simply want to know your customer segmentation? The objective that you want to reach will affect the rest of the step of this survey.
2. Identify segmentation for the survey. From whom are you going to collect responses from? Do you have any specific groups that you wish to get responses from? For example: My aim for the survey is to validate previous finding. To do so, i have to get responses from the same segments as previous study: power user and non-power user. You also need to identify how to reach them. If you wish to send them email, you need to contact BI to get the email data accordingly.
3. Craft the survey. a) Introduction: Tell the respondent the objective of this survey in general. Also specify approximately how long it takes to finish the survey. This may help respondents to allocate time to answers all of the questions. It is also desirable to offer rewards to respondents because we appreciate their time and willingness to participate in the survey. If giving all respondents reward is not feasible, alternatively, you can limit your reward to only first 100 responder or you can also provide a big reward and draw them. Another thing is that you should always give instruction to your respondents on how to do the survey. b)Questions: Make the questions shortly-worded, straight-forward, and simple. A question that is too long will make it harder for respondents to understand. Avoid the word “not” and try to find synonym instead. Next, strictly limit your survey question so that respondents would only need maximum 10 minutes to finish the questions. We do not want to get invalid data because the survey is just too long and boring. Also, specify the response options for each questions, would you want a likert scale, multiple answers, or a forced choice response.
4. Try-out question to several people. After crafting the survey (and get reviewed by your team) you want to pre-test you survey questions to three or five people who are not familiar with the context but have similar characteristic with your segmentation. Ask them to give inputs on the survey. By so doing, you can identify which questions need revision or even need to omit because it just doesn’t make any sense to other people!
5. Send out the survey. You want to send survey when it is the best time to get highest responses. On what day you send the survey affects the overall numbers you’d get at the end. Survey monkey found that sending survey on Monday results to highest response as people don’t feel like working. In contrast, Friday leads to lowest response as people do. Access the full findings here
6. Wrap up the survey. When is the best time to start analysing? Another finding from survey monkey show that 40% of responses are obtained on the first day but we want to wait until a week to get 80% of the responses. After that, the result will not increase significantly. Access the full article here. So after seven days, we should start analysing the results.
7. Do stats thing. We can choose several statistical techniques according to your objective but let’s just discuss about margin of error and confidence interval here. Basically confidence interval tell us “how much faith we can have in our sample estimates”. The bigger the interval number, the more we are uncertain about the actual score in the population. We can specify confidence interval by calculating margin of error first. To do so, we need average score, standard deviation, and the sample size (you can get all this instantly from survey monkey analysis). For a more thorough yet practical reading, let’s go here.
8. Voila! Present the results. Once you get the result, make sure to build a detailed report about all of your findings there but do not go too detailed when you go through with others. Only highlight key findings and do give simple explanation about what you did (analysis and stuff) but do not go with detailed actual numbers, trust me they don’t really care. Just make sure that you know how to interpret them in a simplest way possible and what are actionable items you can execute out of that.

So, it’s possible to hack insights and get rich understanding and numbers in a short given of time. Qualitative method gives richness in our understanding about behaviour, it’s a good initial move for you to take. However, it’s should not be your last.