Why traditional polling keeps getting it wrong
by Dr. Don Vaughn, Ph.D.
How is it that once again, nearly every poll vastly underestimated Trump’s support — a near repeat of 2016 — yet Invisibly’s novel polling technology stood alone in detecting Trump’s true support? In fact, to the best of our knowledge, our presidential predictions were the most accurate of any poll, as measured by either popular vote or electoral college (Figure 1).
I offer an explanation for the persistently poor performance of mainstream polls: traditional polling methods are plagued by a systematic psychological bias that favors socially-acceptable candidates.
Figure 1: Invisibly’s polling accuracy. A. Invisibly’s polling technology predicted the national popular vote for Trump far more accurately than poll aggregator FiveThirtyEight. B. On average, Invisibly’s polling technology also predicted battleground states more accurately than FiveThirtyEight. C. We directly translated state predictions into electoral college predictions, and our results predicted the national electoral college count to within 4 votes. Note: all data are normalized after removing 3rd party candidates.
————————————————————————————————
The psychological principle is rather simple: people are less likely to share something that makes them look bad. For example, most people would be hesitant to admit that they are an atheist when attending a church gathering, or that they are truly happy in their marriage after a friend laments their painful divorce. This reticence to share depends on the social circumstances, and it likely is exacerbated in emotionally-charged environments, where saying something controversial might have more severe consequences than just “looking bad.”
Given the currently polarized political climate, voicing support for Trump can carry significant implications. If the listener is also a Trump supporter, one might find some camaraderie. If the listener is not, however, the Trump supporter may suffer social consequences. While individuals on both sides of the political aisle clearly can experience these consequences, data from recent years suggest that conservatives feel less secure than liberals in sharing their opinions.¹ ²
Thus, when people are asked about their voting intention in an identifiable, personal manner, some either will not respond or will publicly voice support for one candidate and then will vote for the other. Unfortunately, traditional polls collect data by calling people on their personal phones — so the respondents know they are identifiable, and the conversations feel personal. In 2020 (as in 2016), I argue that Trump supporters were less likely than Biden (or Clinton) supporters to volunteer their actual intentions and to respond at all, and voilà — biased polling.
Polling in the presence of bias
How did Invisibly predict the outcome of the presidential election more accurately than traditional polls? Our technology minimized this “shy Trump” bias by creating psychological distance through anonymity.
Our Realtime Research surveys are displayed to people on webpages (Figure 2A). This type of interaction seems more anonymous than a phone call, and I believe respondents feel much more comfortable voicing their true intentions. Additionally, our surveys are fairly nondescript and completely voluntary, so people see a casual opportunity to voice their opinion, and some of them opt in. This method has proven successful in many of the previous races we’ve predicted (Figure 2B/C). To be clear: I do not claim that our methodology is bias-free — just that it’s less biased than traditional polling.
Figure 2: Survey examples and past political performances. A. Questions are shown to the user on webpages. Prior to the 2020 presidential election, Invisibly surveys predicted several political outcomes, including B. Iowa’s 4th District 2020 Republican primary and C. Texas’s 13th District 2020 Republican primary.————————————————————————————————
Now, traditional pollsters happily let us know that standard practice in the field is absolutely not to allow anyone to respond voluntarily (i.e., a “convenience sample”), but rather to dial random phone numbers. I teach statistics, so I understand that this is an optimal approach… in theory. But outside the ivory tower of well-controlled lab experiments, this method focuses far too much on finding a sample with near-perfect representation and then employing statistical corrections to make the data 100% representative across multiple demographic variables. It ignores the potential value in sampling only those who want to participate. After all, people choose to vote; they are not chosen to vote by the government randomly dialing phone numbers.
Our technology also increases precision by collecting larger samples — often at least an order of magnitude larger than traditional phone polling methods. I don’t doubt that this focus on scale, rather than the purity of the representative sample, also underlies our success. I’ll trust 35,000 interested and willing respondents over 800 robocalls any day.
While the biased polling phenomenon has the most obvious effects on political campaign strategies, it has secondary effects on the rest of us. The media displays long lists of polls, almost all biased in the same direction, and this exposure divorces our expectations from reality. When CNN consistently showed Biden up in the months preceding the election — by as much as 16 points — we are all shocked on election night when the race is actually very close. This discrepancy can undermine our trust in the results and may even affect the election’s outcome if enough people mistakenly believe that their vote isn’t “needed” because their candidate is so far ahead. More succinctly: bad polling is bad for democracy.
Dr. Don Vaughn, Ph.D.
Head of Product
don.vaughn@invisibly.com
*Results as of November 8, 2020; electoral vote count assumes North Carolina goes to Trump and Georgia to Biden.
[1] “Majorities of Democrats (52%), independents (59%) and Republicans (77%) all agree they have political opinions they are afraid to share.” https://www.cato.org/publications/survey-reports/poll-62-americans-say-they-have-political-views-theyre-afraid-share#liberals-are-divided-political-expression
[2] “Nearly two-thirds of Republicans and Republican-leaning independents (64%) think that ‘Democrats in this country are very comfortable to freely and openly express their political views,’ but only about a quarter (26%) think Republicans around the nation experience that same level of comfort.” https://www.pewresearch.org/fact-tank/2019/06/24/republicans-see-a-national-political-climate-comfortable-for-democrats-but-less-so-for-gop/
