Were the Polls in 2016 Wrong?

No, but your understanding of them may have been!

Joe Ramirez
Sep 2, 2020 · 6 min read
Image for post
Image for post
Nate Silver of FiveThirtyEight.com Source

hen Nate Silver declared on November 1st, 2016 that “Donald Trump Has a Path Victory,” many political analysts were quickly incensed at the notion. Donald Trump was rarely viewed by the media as a serious political opponent as he cruised to the Republican nomination. The prevailing notion at the time was not if Hillary Clinton would win the presidency, but rather, by how much? Of the nearly 50 national polls conducted in the month of October, only 6 had Donald Trump in the lead. How could he possibly have a path to victory given those polling numbers? Well as we well know now, he did indeed have a path to victory

What Went Wrong?

For one, the media became overfocused on the national polling numbers while virtually ignoring the emerging story in the Midwestern battleground states (Pennsylvania, Michigan, and Wisconsin). Even though the national numbers had tightened during the month of October, Hillary Clinton’s lead in the polling aggregate on RealClearPolitics was still outside of the margin of error.

Image for post
Image for post
RealClearPolitics General Election 2016 Source

The Margin of Error?

The margin of error is a way to determine precisely how much random sampling error there is in a given poll. Why do pollsters get random samples in the first place? Because it’s not economically or practically feasible for any pollster to reach out to every single registered voter to perform a survey. As a result, these pollsters take random samples of the registered voting population. However, to account for the fact that the poll was performed on a random sample of the population and not the population itself, the pollster always provides the margin of error for the poll. Take this quote from the pollster Selzer & Company from a poll they conducted on behalf of Bloomberg Politics in the final days before the election:

Percentages based on the subsample of 799 likely voters in the 2016 general election may have a maximum margin of error of plus or minus 3.5 percentage points. This means that if this survey were repeated using the same questions and the same methodology, 19 times out of 20 times, the findings would not vary from the percentages shown here by more than plus or minus 3.5 percentage points. Results based on smaller samples of respondents — such as by gender or age — have a larger margin of error.

Selzer and Company tucks away at the very bottom of their poll a nugget of information that is lost on the general public when the media reports about polls. Even though a pollster reports a certain result from their poll, they are not at all guaranteeing that their result is exact. In fact, they are saying for 95% of samples taken just like theirs was, their results will fall within plus or minus 3.5 percentage points of their results.

Why Is This Important?

While the margin of error might seem like a minor detail to some, the concept is absolutely crucial to interpreting poll results. Take our Selzer & Co poll for example: the poll’s findings were that Hillary Clinton was leading by 3 points in the final run-up to the election, but the margin of error in this poll was a sizable 3.5 percentage points! To many Americans, that seemed like a significant lead, but in reality, Selzer & Co had declared that any result within the +0.5 Trump (-3.5 percentage points) to +6.5 Clinton (+3.5 percentage points) range was valid for their poll. Suddenly her lead no longer seemed that large, right? Her lead of +3.2 percentage points (in the RealClearPolitics average of polls) before the election seemed impenetrable at the time, but with an understanding of how margin of error works, we can now determines that it was most certainly not. Of course, that’s not even accounting for the fact that she ended up winning the popular vote, but losing the electoral college. The RealClearPolitics polling average of +3.2 percentage points for Clinton actually ended up being quite close to her final margin of victory in the popular vote of +2.1 points. But can our newfound understanding of the margin of error help explain why she lost the electoral college? Yes, indeed!

Image for post
Image for post
The supposed “blue wall” of Wisconsin, Pennsylvania, and Michigan Source

The Forgotten States

Prior to 2016, the last time a Republican had won the states of Michigan or Pennsylvania was 1988 and the last time a Republican had won the state of Wisconsin was 1984. These states were considered part of the Democratic Party’s “blue wall,” states that they had won year after year since at least 1992. Much has been made of Clinton ignoring the three rustbelt states that ended up costing her the election, but for the purposes of our analysis, we want to understand if a better understanding of margin of error might have helped us foresee this outcome.

Image for post
Image for post
RealClearPolitics Pennsylvania 2016 Source

As we can see in the above chart, Clinton’s lead in Pennsylvania was fragile at best. Political analysts around the country, as well as the Clinton campaign, took Pennsylvania as a given for Clinton. In reality, every single poll conducted in the final week of the election pointed to a possible Trump victory in the state. Clinton did not not have even one poll in the state that gave her a lead outside of the margin of error.

As for Michigan and Wisconsin, Clinton did end up having a lead just outside of the margin of error in both states. However, an understanding of the margin of error would certainly have helped better inform the popular opinion at the time that Clinton’s lead was commanding. In addition, the polls in these states had other issues outside of the scope of this analysis. As Nate Silver said in his analysis from November 1st, 2016 that was linked to earlier:

This time around, we haven’t seen too many of those polls in Clinton’s firewall states, such as Colorado, Pennsylvania, Wisconsin and Michigan. But that’s misleading, because we haven’t seen many high-quality polls from those states, period! We have seen lots of polls from North Carolina and Florida — for some reason, they get polled far more than any other states — and plenty of them have shown Trump gaining ground, to the point that both states are pure toss-ups right now.

What are the Final Takeaways from 2016?

While there have been countless post-mortems written about the 2016 election, there are without a doubt lessons to be learned for future elections. For the 2020 election and beyond, political analysts would better serve the public if they make sure to mention margin of error whenever they report on all political polls and clearly explain it each time (not just a footnote at the bottom of the article). This will ensure a better informed public as well as reduce the distrust in political polls that 2016 caused in the public’s eye. Finally, we must be wary of the polling aggregate of a particular state when the numbers of polls run in that state are comparatively low to other states.

Digital Diplomacy

Technology, digital, and innovation, at the intersection with government and foreign policy

Sign up for We Are Digital Diplomacy

By Digital Diplomacy

Focus on technology, government, foreign policy and anything in between. Take a look.

By signing up, you will create a Medium account if you don’t already have one. Review our Privacy Policy for more information about our privacy practices.

Check your inbox
Medium sent you an email at to complete your subscription.

Joe Ramirez

Written by

Data Science Student at Flatiron School

Digital Diplomacy

Technology, digital, and innovation, at the intersection with government and foreign policy

Joe Ramirez

Written by

Data Science Student at Flatiron School

Digital Diplomacy

Technology, digital, and innovation, at the intersection with government and foreign policy

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store