Evaluating Prediction Models — Part 2: Brier Score

Danny Page
Nov 8, 2016 · 2 min read

A follow-up from yesterday’s post about using betting markets to evaluate prediction models; today’s post will be using the Brier Score calculation on a state-by-state basis. It’s used very often in weather forecasting to see measure how often it predicts rain and scores it based on confidence. A perfect score is zero, a perfect fail is one.

If the model thinks a state goes red 99% of the time and it actually goes blue, that result in a poor score of 0.98. Some models very confident on a lot of states, so if there’s a miss, it’ll hurt. On the flip side, D.C. will very, very likely go for Clinton. Calling it with a 100% certainty rewards the model with a 0.0. It’s like golf, you want to have the lowest possible score!

I’ll do this for every state and every model. I’ll also be using a weighted Brier score, which rewards calling a large state like Florida correctly with confidence (up to 29 points!), while that correct DC call is worth only up to 3 points. This, of course, wants to have the highest score. The max possible score will be 538, meaning the model called every state correctly with extreme confidence. (The 2012 Map might do well here if the 2016 Map doesn’t change much.)

Lastly, I’ll be using the models in aggregate to create a timeline of Optimistic, Pessimistic, and Mean expected electoral votes. As of right now:

  • Optimistic Clinton prediction is 341.17–196.83
  • Mean: 313.81–224.19
  • Minimum: 280.68–257.32

As states are called, the bounds of the prediction will narrow toward the eventual number. An example of what that graph will look like:

Just an example, but wouldn’t it be nice if it was over that quickly?

Wrap Up

You can see all the data here. The graph will update in the Google Spreadsheet, and also on my personal website. I’ll also include updates on my Twitter account: @dannypage — Hope you’ll follow along!

Danny Page

Written by

Sports analytics, software development, data analyst. Online portfolio: http://dannypage.github.io

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade