Expanding the Forecast Horizon: Incorporating Futuur into First Sigma’s Political Prediction Analysis
In the ever-evolving landscape of prediction markets and other forecasting methods, a key question persists: which approach is the most accurate? A recent post on Astral Codex Ten (“Do Prediction Markets Have an Election Problem) addresses this issue, building from a comparison of forecasts made by First Sigma for the 2022 US midterm elections. In this article, we extend First Sigma’s analysis to include data from Futuur.
… And we got some surprising results.
Enter Futuur
Futuur’s approach to prediction markets is structurally distinct and stands out from its peers by offering both real-money and play-money betting on the same markets, thus generating two parallel forecasts. Another point of differentiation is that it leverages an innovative automated market maker (AMM), Abraham Othman et al’s Liquidity-Sensitive Automated Market Maker, which builds upon Robin Hanson’s original LMSR market maker, adding, amongst other things, the feature of increasing market liquidity as more bets are made, making prices lower and less volatile.
Futuur is committed to fostering a community characterized by rationality. By emphasizing quality over quantity, predictions tend not to be just aggregated lightly worked opinions — highly prone to biases — but rather informed forecasts shaped by insightful deliberation. We believe that this stance enhances the accuracy and reliability of our forecasts. But who are we to tell? Let results speak for themselves.
Methodological Continuity
Adhering to the meticulous methodology of First Sigma’s initial analysis, we incorporated Futuur’s predictions. We gathered the data from our platform’s openly accessible odds as of the night of November 7th, 2022, following the same framework. We used the same logarithmic approach to determine the accuracy score. This method ensures a seamless integration of Futuur’s data into the existing analysis, allowing for a direct comparison across all platforms for the US 2022 midterm elections.
Comparative Analysis
The original results from First Sigma’ analysis (excluding Futuur) are summarized within the following chart:
With Futuur’s inclusion, our analysis now presents a more comprehensive landscape of prediction platforms. The comparative analysis reveals significant variation in accuracy. While Metaculus and 538, platforms for expert analysis, aggregation and statistical modeling, lead the pack with their methodologies, Futuur has emerged as the most accurate among the dedicated prediction markets.
Futuur introduces a new dimension of analysis with its unique dual-market structure. By offering both real-money and play-money betting, side by side for the same questions, the format provides an excellent opportunity to compare the results of real money vs play money predictions on an apples-to-apples basis, with no other confounding factors. As such, it helps to address a longstanding topic of debate in the world of prediction markets, as to whether real money betting is more accurate than play-money betting.
The scores illustrate the surprising result that, at least in this case, play-money markets generated more accurate forecasts than real-money markets. We’ll explore possible explanations for this below.
Amongst the prediction markets analyzed, Futuur’s play money market emerged as the top performer, with an average accuracy score of 3.37, well ahead of its real money counterpart, which scored 2.80. This places Futuur’s play-money market at the head of the pack of prediction markets. Futuur ranked #1 for play-money markets, outperforming Manifold, and also #1 in real-money markets, surpassing Polymarket.
Particularly notable is that the two play-money platforms (Futuur and Manifold) were the top performers amongst the prediction markets, suggesting that non-monetary motivations can drive high-quality predictions.
The unexpected lead of play money markets over real money markets
Play-money prediction markets like those offered on Futuur and Manifold function similarly to real-money markets. However, the wagered currency does not have any monetary value; rather, it provides reputational and other incentives for accuracy.
On the one hand, it would be reasonable to expect that cold, hard cash will provide a stronger incentive for accurate predictions than a gamified, points-based system. Still, it is worth considering a few advantages of play-money when it comes to accuracy. First, play-money systems allow for anybody to participate, including bettors who are not able to participate in real-money markets due to regulatory obstacles (Futuur does not allow real-money gambling in certain countries, including the US). Therefore, it is often the case that play-money markets include a broader and more diverse group of forecasters than real-money markets, which correlates (in theory) to a higher degree of accuracy.
Secondly, real-money markets allow those with extra cash to have a disproportionate impact on forecasts. Being rich does not necessarily correlate with forecasting accuracy, lack of bias, or sound judgment, and could in fact be negatively correlated with having the spare time to deeply research the events that are being wagered on.
Thirdly, the absence of financial risk allows participants to focus on honing their forecasting skills over time, promoting a systematic approach to understanding market dynamics and prediction strategies. In that sense, play-money markets often function as platforms for skill development, encouraging participants to engage with genuine interest in improving forecasting abilities, detached from the immediate pressures and potential biases associated with financial gain or loss.
Finally, in the specific case of Futuur’s play-money structure, forecasters start on an equal footing, with everyone receiving the same starting amount of currency. From there, accurate forecasts are rewarded, building the fortunes of those who have consistently demonstrated their ability to predict. It’s worth mentioning that other prediction markets allow users to buy play money, effectively creating a hybrid real/play money, and thus removing the forecast accuracy reward. That’s not the case at Futuur, our play-money market design offers a more meritocratic approach, allowing those with a strong track record to have more of an impact on the collective forecasts — as opposed to those with more cash to spare.
The inherent value of prediction markets in general — be it play or real money — is the aggregation of diverse viewpoints from individuals with varied backgrounds, knowledge, and analytical skills. Play-money markets provide a more meritocratic and inclusive method to accomplish this aggregation, along with the engagement considerations. While the jury is still out regarding accuracy versus real-money markets, it is clear that play-money markets have an important role to play.
Reflections and Forward Look
The inclusion of Futuur in the analysis provides a more complete understanding of the prediction ecosystem. The observed variance between Futuur’s real and play money markets adds a new dimension to our appreciation of how different incentives shape forecasting behaviors and outcomes, and opens up new avenues for analysis.
Futuur’s dual system includes fiat currency payment options alongside crypto options and caters to a diverse spectrum of users — from those looking to test their predictive skills without financial risk to serious bettors keen on leveraging their insights for real-world gains.
Given the sheer breadth of markets Futuur offers, with over 800 markets currently open for betting in both real-money and play-money, and thousands of others having been resolved over the past years, we are excited to go deeper in this analysis. Our plan is to do a comprehensive review across all of Futuur’s markets that had both real and play money wagering, to provide a data-driven answer to the question of which approach is most accurate.
More and more, all eyes are on this year’s US election. We’ll continue to explore the dynamics of election forecasting against this background in our next article, adding Futuur contribution to the analyses and discussions around the power of prediction market platforms. Stay tuned for more!
