Should I Use Surveys? 7 Ways to Know For Sure — The Complete Guide to Writing and Using Surveys that Don’t Suck
If you want the “too long, did not read” summary of this extremely long post, here it is: if you can’t tell me exactly why you’ve decided to use a survey as a research tool and which specific audience/s your targeting, you shouldn’t be using a survey. Full stop. End of story.
Surveys should not be your default research tool because they’re easy to design (they’re not) or because it’s difficult to talk to customers (it might be, but that’s no excuse).
They aren’t a replacement for interviews or conversations.
And, you certainly shouldn’t be quantifying things that have no business being quantified. Seriously, stop asking me how happy I am on a scale of 1–5 — it makes no sense. Just by asking me that question, you reduced my happiness to Level 4. You owe me 2,000 energon cubes.
So, What IS a Survey?
Surveys are a quantitative research tool for gathering information about a group of people. At their core, surveys are used to make inferences about larger populations using samples of a smaller number of people — this is because it’s typically really difficult or expensive to perform research on a complete population. A population is simply the entire group of people that shares a set of characteristics, e.g., women in California between 18 and 35 or all Facebook customers who own a house. Theoretically, you could list out every single member of a population because they actually exist in the real world.
For example: say you want to know how many iPhone customers want covers for their phones, so you can quantify the market size for your awesome new iPhone cover company. In an ideal world, you’d be able to ask every single iPhone user if they want to use a cover for their phone. Then, you could be absolutely certain that your iPhone cover data is 100% accurate.
But, as you can imagine, it would be really difficult, likely impossible, to talk to every single iPhone user on the planet (note: this is probably a bad population to sample because it’s too broad, and you should better segment your market). How would you get the names and contact information of every iPhone user on the planet? Even Apple doesn’t have that information, and the NSA probably won’t take your call. Even if you were a nearly-omniscient wizard with the best address book ever, there’s no way everyone would be willing to respond, for a myriad of reasonable reasons ranging from time limitations to privacy concerns.
Surveys are Statistical
This is where the survey comes in and saves the day. Using the power of statistics, you can instead randomly choose a certain number of iPhone customers (a process known as sampling), ask them if they want a cover for their phone, and then be pretty darn certain that your data reflects the opinion of everyone in the world — not 100% certain, but pretty damn close if you sample enough people. I may eventually write a post about populations, sample size and how to sample, so stay tuned if statistics and probability get you out of bed in the morning.
In summary, surveys are at their most powerful when you want to understand something about a large group of people. This leads me to reason #1…
Reason #1: You care about quantitative data and generalizations.
Surveys are about understanding larger groups of people using statistics. If you don’t want to make some sort of generalizations about a large population, another research tool would likely be more useful to you.
That doesn’t mean you can’t use a “survey-ish” form in other situations to collect data about people, but you shouldn’t be making generalizations from this data. This is an incredibly key point. Small sample sizes are fine — the vast majority of my own work is squarely in the “small, qualitative samples” realm. But, this data doesn’t have the same statistical power that larger sample sizes would — you can’t and shouldn’t make inferences about a larger population.
Example: If your product is for data scientists working at large enterprises and you survey 15 of them, and all of them tell you that tree maps are the most important graph they use, that’s incredibly valuable information you can absolutely use to build your product. You cannot, however, say anything conclusive about how important tree maps are to enterprise data scientists, in general. Your sample is too small. There’s a very likely chance that your random sample isn’t representative of the opinions of all data scientists. You may have gotten the super-weird tree map fetishists or something.
If you don’t want to make statistical conclusions about a larger population, but you still want to use a survey, I’d encourage you to really think about why. You’re in the realm of small samples and there are awesome, much simpler methods that are designed specifically around collecting data from a limited number of people. Personally, I will almost always default to an interview or simple conversation over a survey for small samples.
Reason #2: You want to understand a large population.
First order of business: what counts as “large”? Generally speaking, you get more bang for your buck the larger the group of people you’re trying to understand. Personally, I wouldn’t put the work in of designing a survey for a population of less than 200 people. That doesn’t mean you need to have 200 people take your survey, it just means there need to be 200 people that you could theoretically talk to if you had all the time and money in the world.
Here’s why I don’t recommend surveys for small populations: as you can see in the table below, you can sample 96 people out of a population of either 10,000 and 100,000 and get the same 10% margin of error. For example, to get the same margin of error for a population of 100 people, you still need to sample 50 of them, which is half the population — it’s really challenging to get 50% of a population to take a survey.
It’s not terribly important to understand the statistical details of these numbers quite yet, but it shows the power of large numbers and how surveys can be particularly useful for making inferences about big groups of people.
Conversely, if your population is relatively small (less than ~100) — say, owners of vegetarian restaurants in Berkeley — a survey might not be the best tool for you. You will have to sample a relatively high percentage of them to make any inferences with statistical significance, which means you’ll will have to ensure that you can reach and get responses from a majority of these small business owners. For context, 10% is considered a pretty decent response rate. If you do the math, that means if you have 100 owners, you’ll only get 10 survey respondents — nowhere near enough to make any conclusions with statistical power.
Reason #3: You have a specific group you want to survey and you know how to reach them.
If you’ve been following along, it should be fairly clear that populations are fairly central to survey design. This means you need to know exactly which group of people you want to understand, so you can design your sampling to capture them and your survey to appeal to them. A survey is no different than any product — you can’t design your survey if you don’t know who you’re targeting.
Think critically about the goals of your survey and consider the types of people you want to hear from — are they people with certain habits? From certain locations? Particular age groups? Are they people who consider themselves self-starters? You can segment your population by an infinite set of qualities and attributes, both qualitative and quantitative. Once again, you just need to know why you want to engage with a particular audience.
Hint: your audience isn’t and never should be “everyone.” If your population is very broad, think carefully about what the goals of your survey are: they should be specific, so if your population isn’t equally specific, there might be a problem.
Equally important? Your ability to find your target participants. It’s great if you have very specific and narrow criteria, but will you be able to identify these people and reach out to them? Are they likely to respond?
Or, maybe your population doesn’t have the resources to participate—literacy, time, and technology are all big issues for disadvantaged populations. For example, if you want to survey the homeless, your ability to sample and find participants will be very limited. I’m guessing you’d have to partner with local organizations who have existing relationships.
Reason #4: You’re willing to put in the time and effort to critically analyze every detail, get feedback, and iterate, iterate, iterate.
It should be clear by now that surveys aren’t quick and easy to design if you want good results. There’s a reason why people spend years getting graduate degrees and founding entire companies around designing surveys — it’s kinda complicated.
Even if you’re good at writing surveys, your first draft will suck. Your questions will be biased or unclear or be in the wrong format or be screwy in one of 50 common ways. Your questions almost certainly won’t be in the best order. You need to think carefully about how you’re finding participants, and if your sampling technique is missing any important segments.
The solution is easy: like any good designer or entrepreneur, you need to iterate, iterate, iterate. This means you have to calculate the time and resources to revise and test your survey. Have everyone on your team take the survey and critique everything: your language, your formatting, your instructions, your font — everything. Remember, you should empathize with your participants, so anything that affects their experience is fair game.
Ideally, you’d also get feedback from someone who has survey design expertise or is a scrappy social scientist/statistician. If you need help in this area, send me an email, and I’ll see what I can do about helping.
If you don’t have the time to critically analyze your survey — that’s OK! Look into other leaner, quicker tools that you can easily use.
Reason #5: You aren’t asking lots of open-ended questions.
I don’t know about you, but I hate open-ended questions on surveys. Surveys are painful enough to take, but then they want me to sit and write them an essay? Argh. I honestly ignore open-ended questions, as do many, many survey respondents. If you’ve completed many surveys, you know that much of the time, people write half-assed answers, or worse, abandon the survey entirely if they get overwhelmed. Use them with great trepidation — 2 or 3 open-ended questions, at most.
They can be used well — as a place to start exploration on a particular topic or to get your participants’ unfiltered thoughts. But, they’re asking a lot of people. It’s cognitively much more difficult to come up with responses to open-ended questions than closed-ended ones (see Recognition vs Recall). Be especially thoughtful of how you’re asking an open-ended question to make it as easy as possible for people to respond.
On the other side of things? I also know people struggle with how to use the open-ended response data, since they can’t be analyzed using traditional statistics, so these responses are glanced over and sometimes ignored. This is bad. You should never be collecting data that you will not use.
Also, if you’re primarily using open-ended questions, this is evidence that you haven’t converged to understanding your problems quite yet. Use interviews or other qualitative methods to get to the core of the questions you need to ask and the data you need to collect. Remember, with open-ended questions on surveys, you can’t ask follow-up questions. You should always be asking “Why?” of people, which is just not possible with a static survey.
Reason #6: You can tell people how your survey will benefit them.
Surveys are an exchange in social capital — you’re asking someone, often a complete stranger, to take time out of their busy lives to provide you with valuable information; information that (hopefully) has a direct impact on your business. If you design your survey well, you will getting incredible value from the input people provide. It’s only fair that you consider what the survey participant gets out of this exchange. Is it compensation? Will you improve technology that they use every day? Will the survey provide them a voice and representation? You should be able to articulate exactly why a person should bother to take your survey — they’re doing you a favor, not the other way around.
Empathize with the participants — I’m guessing you’ve taken a number of surveys in your time and have found them variously annoying and tedious. Be better than other survey designers and think about your participants’ experiences. Survey participants should get as much respect, consideration, and courtesy as any of your valued customers. Design the survey with them in mind, prioritize their needs, and listen to what they tell you. Don’t waste their time or input by not knowing how you’ll use the results of the survey. Make the survey easy and pleasurable to take, and don’t ask any more of the participants than you need to achieve your goals.
This leads me to the final guideline.
Reason #7: You know the goal of your survey, and how you’re going to analyze the data.
Hopefully you’re convinced by now that surveys aren’t exploratory data collection tools. You should have a clear, concrete reason for doing a survey and for the type of data you are collecting. This allows you to evaluate the survey and make sure that everything you’re doing puts you closer to this goal.
You also need to know exactly how you will analyze the data after you’re done collecting the surveys. You will be making a lot of choices about question format, sampling, etc. that greatly influence how you can look at results. Plan ahead, or you’ll be really bummed when you can’t used that logistic regression you’ve always dreamed of.
