Reliably Wrong — The Michigan Reminder
On the oft-chance that you have been following this year’s most watched reality-TV show south of the border “The Amazing (Presidential) Race”, then last week’s episode will probably have been quite the shocker for you.
For those those not in the know, right now the US is in the auditioning phases (a.k.a. primaries) for who will represent the Democratic and Republican party in the general elections later this year. The focus of my post today are the two contenders for the democratic spot, Senator Bernie Sanders and former secretary of state Hillary Clinton, and the upset I’m going to talk about are the results of the Michigan Primaries.
As in Canada, political organizations, news corporations and the parties themselves will poll constituents of a state to see who is the likely winner. On the night before the Michigan Primaries, the aggregate results from polls were predicting Hillary had over 99% chance of winning by a margin of 21%. So when Sen. Sanders wiggled himself a 1.5% win over Hillary, people went out calling this “One of the Biggest Upsets in US Primary History”. The website FiveThirtyEight has a great visualization of what the polls looked like entering the election.
Now why does this matter?
Well, the truth is, we can’t quite know if this win really mattered. That is, we can’t tell if this is just a case of low probability coming true despite good poll methodology, or if there are some real improvements that need to be made to the system.
That being said, even if it ends up being a one-in-a-hundred thing, there were some interesting trend that was brought to my attention and that suggest how polling could have gotten this so wrong.
Trend 1 (The Demographics):
Why is it that the polls seem to be more accurate on the GOP (Republican) side compared to the polling done with the Democrats? Well, if we rely on pundits like Cenk Uygur or Nate Silver, a lot of it might come down to how reachable are the potential voters. Under American Federal law, automated polling can only be done through phone lines, which severely disadvantages young voters. Exit polls for the State of Michigan seem to corroborate this story with a whopping 81% of 18–29 year olds voting for Sanders. Though this is unlikely to be the only cause for what happened, it might be unwise to ignore it.
Another methodological point regarding demographics are the assumptions being made is how the sampling is being made. No poll leading up to the Michigan vote seemed to use stratified sampling that matched the demographic make-up of the state. This in itself might not be seen as a big issue since minorities don’t matter so much (quite contrarily to research you’d do in clinical psychology) since they are few in number. However, in at least one poll, I saw another kind of stratification technique that had me raise an eyebrow. I’ll let you read the quote and decide for yourself:
“Because likely Primary voters are older, 54% are 60 or older and 86% are older than 50, we believe there are sufficient land line voters to get an accurate sample. We do not have to make any assumptions of likely voter turnout.”
This was part of the polling report that was commissioned by Fox 2 Detroit and what it shows is a polling methodology that weighs a lot heavier on the the older folks. And though it’s not hard to believe that the old might go out in greater number to vote, when that doesn’t happen the currents can quickly shift in favour. Case in point, the CNN exit polls recorded 45% of the electorate being under 45, a long shot from the 86% of likely voters that were supposed to be over 50. This discrepancy is nowhere more obvious than in the final predictions offered by Fox (29% Sanders vs. 66% Clinton) and the real outcome (49.8% Sanders vs. 48.3% Clinton)!
Of course, Michigan was the exception and not the rule, so perhaps stratifying the votes in terms of likelyness and not demographics is still best practice. However, I believe that a caveat should be made when introducing those polls.
Trend 2 (The Motivations):
SO WHAT did motivate so many young people to go vote, or conversely, why did relatively few old people vote? This is where things get murky and where speculations abound.
What’s important to understand is that polls don’t happen in a vacuum.
When the numbers come out, it is reasonable to think that people react to them, though how might be unclear. Returning to Michigan, most polls predicted an easy win for Hillary with a lead ranging from 7 points to 37 points over Sanders. This fact, coupled with people being forced to choose which primary to vote in, might have led Hillary supporters to instead use their vote to elect a more moderate GOP candidate. Though most polls already account for likelyness to vote in the democratic primary by weeding out those who aren’t sure where they’re voting.
This leaves the possibility that Clinton supporters simply got complacent and didn’t vote while Sanders supporters did. This is a real possibility if you measure the probability of voting in different ways. It might be that answering “likely going to vote” in your run-of-the-mill poll implicates very strong ceiling effects, clumping both voters and non-voters. Indeed, Gallup polls (which didn’t poll in Michigan), had developed a sophisticated and validated scale for predicting who is most likely to vote. In it, they ask for prior voting history, political versedness and how likely they are to vote. This is something that lacked in the polls I saw and that very likely played an effect. A simple testament to that is the fact that Sanders broke the US record on individual donation contribution; a fact that is very telling of the motivations in his camp, but that wasn’t reflected in the polls.
Trend 3 (The Debates):
Last but not least, the debates. Most of the debates are conducted 1 to 6 days before the election is set to begin. This is important because the aggregate poll scores that are used to predict outcomes take into consideration pre-debate polls (although granted they are weighed less) as well as the post debate polls. In the case of Michigan, only one poll was taken post debate, thus preventing us from grasping its impact on the electorate. To understand just what kind of impact we’re talking of, we only need to look back a couple of month ago to Canadian politics where campaign front-runner Thomas Mulcair lost around 20 points (and his lead) in polls 48 hours after espousing an unpopular view during a televised debate.
The lesson here isn’t for the pollsters themselves, but rather another potential explanation for the Michigan fail. By having a debate right before an election, we’re cut off from the possible fallout that might have occurred.
So to sum up
The case of Michigan was either a fluke, or a call back to the drawing boards. Though polls coming into the election had high consistency, they ultimately lacked any validity.
If there is one take-away from this post, it’s that polls aren’t infallible. Though they can offer a good estimate of actual results, methodological issues such as demographic selection, prediction of voting behaviour and the wildcards that are debates can begin to explain why every now and then, we will get results that are wildly different from what’s anticipated.
