You see it all the time on social media. Someone’s doing a poll and they add on “RT for a bigger sample size/more accuracy” at the end of it. Someone else argues with an opinion poll because “they only asked 1,000 people and no one I know has ever been itterviewed”. Finally, someone explains that while they might have only asked a few people for their research report, it’s still a bigger ratio than polling companies manage for the whole country, so therefore it must be accurate.
And in response to all three examples, people who actually understand how statistics and polling work wince at seeing an idea being so badly misunderstood.
Now, I appreciate on a surface level that opinion polling and surveying can seem like they’re just guesswork. The idea that you could ask a thousand or so people and get a broadly accurate picture of the way the nation from questioning them does seem absurd on its own. If you can tell the views of millions from a sample of a thousand, then why not a hundred, or ten, or even just one, if you could find the right person to represent the nation as a whole?
This misses that the important part about sampling is that it’s not just grabbing a thousand or so people, asking them their opinions and then totalling them up at the end. It’s not just any sample, it’s one that’s both random and representative.
The ideal way to get a random sample is to have a list of the entire population you want to survey and then pluck your sample from that. So, if you had a city with a population of a million, you could go through the list of all of them, take the name of every thousandth person, and end up with your sample, who’d you then go an interview. As this technique isn’t easy to do in the real world, polling and research companies have found all sorts of other methods to attempt something similar. These methods also attempt to counter the problem that many people don’t respond very well to being asked their opinion by a stranger — in order to get a thousand responses, you’ll need to ask a lot more than a thousand people.
Solving this problem is something polling and research companies spend a lot of time and effort on. The questions of how to get a random sample of the population, and then how to get them to answer your questions when you’ve identified them is the key to being able to do their jobs and the quicker and cheaper they can do that, the better for them. Every company, and every academic polling research team has their own way of solving these issues. It’s also worth remembering that for many polling companies, political polling is a sideline (and even a loss-leader) but they do it because it’s a very good way of advertising your accuracy.
Another thing to remember with a random sample is that it’s the responsibility of those doing the survey to find their sample, not to expect their subjects to come to them. As an example, a large part of the British general election exit poll is still done as a face-to-face survey outside polling stations, but the interviewer doesn’t stand under a sign saying “please come speak to me about your vote”. To get a random sample, they approach a certain proportion of the people who vote and ask them to take part. By approaching every tenth person to come out a polling station, they’re getting a random sample of the people who vote there, not a self-selecting sample of those who’d choose to come to them.
The second key part is making sure that your sample is representative of the population as a whole. In an ideal world, your random sample will have been a generally accurate representation of the demographics of your wider population, but even if that is the case, different response rates from different parts of the population might mean that your data is skewed because you’ve ended up speaking to too many people from one segment of the population and not enough from another.
The question then is whether the data you have can be weighted to attempt to balance out that skew, or if you need to scrap it and start again. As an example, if your survey of a thousand has responses from 550 women and 450 men, you can probably weight the responses to make it closer to the 510 and 490 a fully representative sample would have. If you’ve got 999 of one and 1 of the other, then you really need to start again (and ask some serious about your sampling methodology).
The key thing about weighting is that it’s being done on a random sample of the whole population in line with known demographics. You can’t do it on a sample that’s neither random or representative to fix it. That’s why you can’t make a self-selecting poll accurate by weighting the responses because there’s a bias at the heart of the data towards those who choose to respond and those who’ve heard of your survey.
I’m not going to go heavily into the maths of sample size and accuracy here because they’re complex and I don’t have any of the key texts to hand so I’d likely make a mistake in explaining them. The maths of sampling originate in mathematical probability, but the key thing to understand is that the relationship between sample size and accuracy is not a straight line. Having twice as many responses does not make a survey twice as accurate and even if your sample is sufficiently random and representative it requires a drastic increase in the amount of data to make a small increase in the level of accuracy.
If your sample isn’t properly random or representative to begin with, however, then it doesn’t matter how large your sample is, it’s only ever going to be accurate by accident. This is not new information, it’s something we’ve known since the 1930s and the dawn of opinion polling as we know it, most famously demonstrated in the 1936 US Presidential election, when George Gallup’s poll of thousands of people proved much more accurate than the Literary Digest’s poll of millions because he used proper sampling and they didn’t.
The important thing to remember is that the sample size of a poll is important in determining how accurate it is, but only if that sample has been obtained in a way that ensures it’s random and representative. The number of people who’ve responded is not important if they’ve not been accurately selected to participate, no matter how much you might like the results they generate.