3 Simple Reasons the 2016 Election Polls Got it So Wrong

Since the blind-siding victory of Donald Trump, much as been written about how exactly the polls turned out to be so inaccurate. It may intuitively seem like a very complex problem. A person could truly spend a lifetime diving into every minutia of the polls. But the underlying trends which caused the polls to be so inaccurate are deceptively simple and have been going on for years.

At their core, the polls leading up to the 2016 election were not accurate because they were not well supported by the three, non-negotiable pillars of a high-quality poll. In order to collect truly high-quality data, one must:

1. Identify a random, representative sample of humans within a given geographic area.
2. Find a way to initiate contact with those humans.
3. Provide a survey engaging enough that an adequate number of the humans contacted choose to voluntarily complete it.

As an industry, we’ve seen a steady decrease in the accuracy of political polls in each of the past few election cycles. But it became glaringly noticeable to the public in 2016 with the blind-siding victory of Donald Trump.

When we in the industry spend all of our time poring over every little detail of our studies trying to find the specific reasons why we make it difficult to step back and see how all three polling pillars have been weakened and worn down at a consistent rate over at least the past decade. Some may soon fall down altogether!

The three pillars only make sense if you examine them in conjunction with the survey mode setting their foundation. When I analyzed the methodology of the polling organizations included in poll aggregators such as FiveThirtyEight, HuffPost, and NYT, I found every one falling into one or both of the following two groups: Phone or Web.

So let’s examine each pillar within the context of the modes used during the 2016 election.

Pillar #1 — Identify a random, representative sample of humans within a given geographic area.

Phone — When everyone was using landline telephones, this was relatively simple. But now that a plurality of people (47%) live in cellphone-only households, this is becoming an increasingly difficult and expensive task.

Web — No help at all. Everything on the web is either not tied to a person’s physical location or is not consistent across every member of the target population.

Pillar #2: Find a way to initiate contact with those humans.

Phone — This is difficult for the same reasons identifying humans by telephone number is. It is relatively easy to get the landline phone numbers for a list of people in a given area, but if you are omitting 47% of the people you identified from taking the survey, the second pillar will crumble.

Web — Still no help here either.

Pillar #3: Provide a survey engaging enough that an adequate number of the humans contacted choose to voluntarily complete it.

Phone — Response rates have plummeted over the past 10 years. Fewer and fewer humans are engaging back with pollsters by phone each election cycle, with Pew Research showing their average drop in response rates at slightly over 7 percentage points every four years in their phone surveys between 1997 and 2012 (Figure 1).

Web: Web surveys can actually be very helpful in this respect if the respondent is comfortable using technology. The problem is there is a split between younger people who do better with web surveys and much older people who simply cannot or will not complete a survey via web.

But even if web surveys solved every engagement problem, you’d still have to find a way to contact your random sample of humans to let them know about it in an enticing way. If you are using phone and/or web exclusively, it doesn’t matter how good your web surveys are. You’ll never get all 3 pillars in line!

Pew Research Center, “Assessing the Representativeness of Public Opinion Surveys,” May 15, 2012

By this point you might be wondering why having a true random sample is so gosh darn important in the first place. We’ll get to that next.

But first, try to answer this question; Who would likely select a more representative sample of humans within a given geographic area?

• An office of fully-trained survey methodologist armed to the teeth with all the latest Census Bureau data? Or
• A cup of Yahtzee dice with the same?

Why is a Random Sample Important?

If a survey is using a random sample, it means every individual in the population has an equal chance of being identified and contacted. If you randomly identify a large enough sample of people, it is likely to be a much closer representation of the actual population than anything humans could design themselves, no matter how hard they try to mimic the exact demographic characteristics of their out-of-date Census Bureau data (Yahtzee dice 1, Humans 0).

If you want a fun, interactive lesson showing why this is the case, click here to go to the PBS website for a short demonstration.

What is the Margin of Error?

In the aftermath of the 2016 election, many commentators have pointed out that the polls were within the margin of error, but what does that mean? What good are polls if they are technically within the margin of error, but still predict things incorrectly?

When you hear someone talk about the margin of error, they typically mean “I am 95% sure the actual, real-world number for the thing I am measuring in this population is somewhere between X percentage points above or X percentages points below the number I just told you was my measurement.”

Imagine a target with the real, honest-to-God value for the population in the center. Let’s say your archery skills make you 95% sure you can hit somewhere in the target, making the target your margin of error. You still want to get as close to the middle of the target as you can, even if hitting the side of it is still “technically” the target.

What Made 2016 So Different?

In elections past, polls were pretty good at hitting the center of the target. Things definitely got shaky by 2012, but people like Nate Silver at FiveThirtyEight were still able to use the average location of all those arrows to figure out where the center should be.

This election was different because when we averaged all the arrows, they came to a spot near the edge of the target rather than the center. This fooled us all into thinking that was where the center of the target really was. In reality, everyone shooting just had similar, bad aim.

But why did everyone have bad aim? We’ll never be completely sure about which of the minutia within each type of poll had the largest negative impact. And that’s okay. Because trying to figure it out is distracting us from the fact that the tremendous amount of energy we spent as an industry trying to aim just right was rendered futile as the platform upon which we stood continued to shift beneath our feet as we loosed.