Can prediction markets predict the Oscars?

Predicting the Oscars is a fun challenge. They are a good example of non-cumulative information markets, where no information is revealed until the last minute, when the markets are resolved. And they are a clearly defined, popular set of markets, so are useful for comparisons. Following our recent analysis of political predictions, where Futuur outperformed several other prediction markets, we wanted to change the subject and see how we are doing in entertainment. So, here we do some analysis of our Oscars results, and comparison with other platforms as well as with our own results from past editions of the Awards. Taking the opportunity to use a well-defined set of results to measure the accuracy of forecasts without the potentially confounding variable of cumulative information. These were our results:

We created 20 markets for the Oscars 2024. Out of those, Futuur forecasters accurately predicted 15 markets, or 75%, where the highest-percentage forecast was the eventual winner. But predicting correctly is not the only thing that matters, especially in the Bayesian universe in which forecasts live. An event having a chance greater than random guessing indicates that the market forecast is aggregating at least some useful information. If you include all cases where the forecast for the eventual outcome was above 20% (the equivalent to random guessing amongst the five possible outcomes) Futuur’s predictions had a 95% success ratio! That’s 38 of 40 markets — counting 20 for real money and 20 for play money.

Notably, real-money markets consistently outperformed play-money markets. There were only two instances where play-money markets surpassed real-money counterparts: the Actor and Director categories — interestingly enough, two of the most active markets. In terms of volume, this Oscars edition averaged a relatively modest 45 trades per market. Despite this, we were able to outperform other prediction markets on accuracy as measured by Brier Scores.

Brier Scores

Our average Brier Score for this Oscar edition were: 0.0723 for play money and 0.0501 for real money. Let’s evaluate what those numbers mean and how they compare with others (spoiler alert: these are really good brier scores!).

In evaluating our performance, we compared our results to the Brier Score analysis conducted by Mike Saint-Antoine (highlighted in Astral Codex Ten’s Mantic Monday 3/11/24) albeit indirectly as we couldn’t access the complete results he presented. Also, it was not explicit which markets he selected to compare for each of the permutations he did. So, instead of doing a direct comparison, we used the average of his results as a benchmark to compare our results to. We included only the three prediction markets in his analysis (Polymarket, Manifold and Kalshi) and calculated the average of the brier scores he arrived at.

As a reference point, Manifold’s Average Brier Score for all the markets ever resolved is around 0.16, which as it happens is exactly the same number for Futuur’s play money markets (for the more than 12,000 markets ever resolved). The coincidence seems intriguing… perhaps all prediction markets’ brier scores will tend to converge to a similar number over time?

In addition to comparing Futuur against other platforms, we also thought it would be interesting to look into Futuur’s historical performance. To do that, we took the average Brier Score for the previous 3 years of measuring the Oscars.

Compared to our past 3-year average, in 2024 we improved our accuracy for predicting the Oscars by 15.54% for Play Money and 37.14% for Real Money. In the following chart, when plotting the results for the last 4 years, we can see a slight trend towards lower Brier Scores overall (more accurate forecasts).

Overall, we see a positive trend where Futuur’s community of forecasters is producing more accurate forecasts for both play money and real money on a defined set of non-cumulative information markets, the Oscars.

So, on the question of whether prediction markets can predict the Oscars, it appears that the answer is yes, and that Futuur is performing quite well relative to both peers and internal benchmarks. But we want to do better. We’ll keep honing our market mechanisms and building our community of forecasters, in order to produce even more accurate forecasts for everyone. We predict a bright Futuur/Future ahead!