Everyone is forecasting the election wrong

Let’s say you’re a stats whiz who wants to forecast the election. What do you do?

Usually, you build a model. You try to think of all the information that could be correlated with election results, and then you calculate those correlations for all the elections in the recent past. Next, you look at the same information for this year’s election and use the correlations to predict the result. Because you’re a stats whiz, you know how to convert your prediction into probabilities that reflect the uncertainty in your model. Candidate A has an X% chance of winning, while Candidate B has Y%.

And everything you’ve just done is totally wrong.

Forecasting an election isn’t about calibrating one model. It’s about figuring out which of the infinite possible models of elections is the true model—the one that perfectly describes why people vote for one candidate or another. In the true model, there is no uncertainty. Either one candidate is going to win, or the other one is. If we can’t figure out the true model, then at least we have to estimate what share of the models that could be the true model would make one candidate the winner.

Think about it this way. If we had no idea how elections worked, how would we estimate the chances of Candidate A beating Candidate B? We might be tempted to give each one a 50% chance of winning. But in that case, we’d just be saying that each candidate was the winner in 50% of all the equally likely models of reality. Yet because we had no idea how elections worked, there would be an infinite number of these equally likely possibilities for the true model of elections—and probably an infinite number in which each candidate won. We’d be just as well off putting all our money on one of them.

By making an election forecast, we’re saying that the true model is most likely to be one where a specific candidate wins. But we can’t say this if we’re relying only on information from the past. Nor can we say it if we’ve only looked at one model, like the one our stats whiz designed. And there are many, many possible models.

This realization may make forecasting elections seem hopeless, but it’s very important. The stats whiz’s model generated probabilities of winning, rather than a 100% prediction, because some of the information collected in the past failed to fit the model. There was an underlying assumption that the information which didn’t fit was somehow random; that’s where the uncertainty came from. But what if it wasn’t random? What if it reflected another dynamic that the model failed to encompass? The model itself doesn’t give us any way of finding out whether this was the case. Instead, it just tells us, “I’m the true model, but, because of randomness, I may not always call the election correctly.”

If that sounds like a stretch, it should. Compared to processes like horse races and football games, there’s relatively little randomness in elections. Sure, weather and other unrelated events may affect some people’s likelihood of going to the polls. But with tens of millions voting all over the country, it seems pretty likely that their votes will truly reflect the will of the people on Election Day. We need an approach that says, “I can’t give you a 100% prediction for the election, because I don’t know which model of election results is the true model. But of all the models that could be the true model, X% call it for Candidate A.”

Even if the stats whiz’s model had given one candidate a 100% chance of winning—and had been correct—it might just have been a coincidence. There would still have been a possibility that the model wasn’t the true model, especially since there were so few elections in our dataset. It’s also worth noting that this election has shown signs of being different from any one in the past, so models that have been 100% reliable so far may soon be revealed as not the true model.

So what’s a stats whiz to do? A good first step would be to ignore the polls, except to the degree that they tell us who’s going to vote. No one’s vote is determined by what they tell a pollster. Rather, voting decisions depend on preferences, context, and the choice to show up on Election Day.

Next, we should collect as many models as we can that rely on preferences, context, and voting behavior. The actual number of models may well be infinite, but we can try to put together a representative sample. We may want to weight the predictions of these models by how well they’ve performed in the past—but only if we sincerely believe that we can evaluate how close they are to the true model. And then we can make a meta-prediction of the election based on how many models choose Candidate A over Candidate B.

In addition to being more rigorous than a single model, the meta-predictive approach has one other big advantage: stability. That’s because preferences, context, and voting behavior don’t change much from day to day or month to month.

Why is stability important? Over the past few months, some forecasters have gone from giving one candidate a 90% chance of winning, back down to the 70s, and then back up as high as 93%. These kinds of predictions have virtually no practical use. Could anyone make household, business, or government decisions based on anything so volatile? The predictions are more for attracting pageviews than anything else.

Another advantage of the meta-predictive approach is that we can narrow down the pool of possible true models after each election. Any model that was wrong clearly isn’t the true model. We can stick with the ones that were right and, if we think of any new ones, add them to the pool.

This approach should work as long as human beings are the only ones voting in our elections. If the raw materials are the same, then the true model should always work. Of course, we may never figure out the true model, but we’ll always be getting closer—and maybe a bit closer than the stats whiz who simply tears up one model and creates another every four years.