A follow-up from yesterday’s post about using betting markets to evaluate prediction models; today’s post will be using the Brier Score calculation on a state-by-state basis. It’s used very often in weather forecasting to see measure how often it predicts rain and scores it based on confidence. A perfect score is zero, a perfect fail is one.
If the model thinks a state goes red 99% of the time and it actually goes blue, that result in a poor score of 0.98. Some models very confident on a lot of states, so if there’s a miss, it’ll hurt. On the flip side, D.C. will very, very likely go for Clinton. Calling it with a 100% certainty rewards the model with a 0.0. It’s like golf, you want to have the lowest possible score!
I’ll do this for every state and every model. I’ll also be using a weighted Brier score, which rewards calling a large state like Florida correctly with confidence (up to 29 points!), while that correct DC call is worth only up to 3 points. This, of course, wants to have the highest score. The max possible score will be 538, meaning the model called every state correctly with extreme confidence. (The 2012 Map might do well here if the 2016 Map doesn’t change much.)
Lastly, I’ll be using the models in aggregate to create a timeline of Optimistic, Pessimistic, and Mean expected electoral votes. As of right now:
- Optimistic Clinton prediction is 341.17–196.83
- Mean: 313.81–224.19
- Minimum: 280.68–257.32
As states are called, the bounds of the prediction will narrow toward the eventual number. An example of what that graph will look like: