America Does Not Understand Polling

Arjun Bhattacharya
Aug 8, 2017 · 3 min read

Whether it’s a clear lack of statistical knowledge or a distinct fear of science and mathematics, it’s actually an epidemic that’s hurting our society.

Last night, senior media correspondent for CNN and host of Reliable Sources tweeted this:

It’s a poll, and it’s reliable. Scroll down and you get anecdotal evidence to Americans both disbelieving and misunderstanding how polling actually works.

We’ve got this:

And this:

And then, we had this pleasantly worded mockery:

Yes, yes, I know. I’m making a grand statement about the ability of using statistics to draw conclusions about a population and then claiming that America has an overall problem with statistical knowledge by citing a few examples of idiocy on Twitter.

But it’s a recurrent problem that we’ve seen over and over again. In 2016, FiveThirtyEight predicted one election incorrectly, the national polls wrongly predicted the presidential election, and hell, we can stick Cleveland coming back from down 3–1 to the list. And after a few wrong predictions, America thinks that all statistical prediction and polling is under the shadowy stranglehold of fake news.


I’m a graduate student in statistics, and though I’m not going to claim I’m an expert, on the spectrum of novices to experts, I’m on the right half. So, let me try to quickly explain how polling works.

Let’s make this simple: the goal of polling is to accurately measure the sentiments of the country on the whole by measuring the same sentiments from a representative random sample. The idea is that if this sample is selected randomly enough and the sample size is big enough, even a sample of maybe two thousand Americans accurately measures the sentiments of the country. And yes, there’s error in drawing inference from samples, but the error is more likely than not small enough to justify sampling. And how else are you going to do this measurement on all of America?

So let’s address these asinine accusations:

“Fake polling”

If you really think that CNN or the Washington Post or the Pew Center is literally making numbers up to justify some kind of agenda, then there’s no point in talking about statistics. I refer you to this.

The conflation of prediction and polling

National polling on presidential voting tries to capture the popular vote. State-wide polls aggregated to reflect electoral voting gave a less overwhelming probability of Clinton winning the election, as in FiveThirtyEight’s predictions.

Just look at that graph of predicted electoral college votes over time by FiveThirtyEight. If the lines fall in the purple shaded region, then those numbers are statistically insignificantly different, meaning FiveThirtyEight isn’t very sure about the difference. They’re saying that Trump was more likely than not to lose, but they weren’t 100% sure of Clinton’s win.

There’s also another issue with this conflation: people are starting to hold a few incorrect statistical predictions against every kind of statistics. And then that’s used as justification for trusting all the flaming piles of shit that exit the mouths of Breitbart, InfoWars, and (I can’t fucking believe I’m writing this) Trump TV.

Also, statistical predictions of the probability of the outcome of a one single binary random variable (as in Trump wins or loses) are just that — statistical predictions of a probability. Just because something has a 99.99% chance of happening, there’s still a non-zero probability that this is that 1 out of a 10,000 times. The New York Times above gave Clinton a 92% chance of winning. That means, based on whatever methods they used, if you ran the election over 100 times, the NYT saw 8 scenarios in which Trump wins the election.

Low chance events occur. That’s life.

Polls don’t really measure anything.

Sure, I can’t argue with that. “Trust most of what you hear from the White House?” is probably a vague question. It also probably a snippet of a single question that yielded the most interesting results from a long survey that CNN sent out. That doesn’t discredit the poll.


America’s collectively an idiot. That’s my bottom line. Pick up Freedman, Pisani, and Purves’s Statistics for God’s sake.

The Crevice

Sports and pop culture with absolutely no filter

Arjun Bhattacharya

Written by

Editor and writer for The Crevice

The Crevice

Sports and pop culture with absolutely no filter

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade