What We Can Learn from Superforecasters
We all try to predict the future. For example, I wonder: how many children will my newly married daughter and her husband have?
When we can’t figure it out ourselves, we consult everything from intelligence agencies to bookies, media pundits to palm readers.
But it turns out that the average expert is “roughly as accurate as a dart-throwing chimpanzee.”
Why can’t expert forecasters seem to figure it out? It turns out that making a good forecast (or a robust hypothesis) isn’t that easy.
Take Microsoft CEO Steve Ballmer’s 2007 forecast that, “There’s no chance that the iPhone is going to get any significant market share. No chance.” Balmer’s forecast is infamous as an example of a failed forecast. Ballmer’s forecast looks wrong, but is it wrong? There are many unaddressed questions within his forecast: What qualifies as significant market share? What market geography is he talking about? Is it the smartphone or general mobile phone market? And what is his timeframe? With so many unanswered questions, judging his forecast is actually impossible.
If we want to become better at forecasting, we can’t have ambiguity about whether or not a forecast is accurate.
Philip Tetlock shared a refreshing change in this pattern of failed forecasting in his book “Superforecasting.” He tells the story of a research team called the Good Judgment Project (GJP) which beat out four other groups predicting global events in a four-year U.S. government-funded forecasting tournament beginning in 2011.
Critical to the contest and to these forecasters learning to become better was creating forecasts that were testable. Forecasts should have:
1) A clear timeframe,
2) Clearly defined terms when describing what will happen, and
3) Numerical probability.
During the tournament, a small group of high performing forecasters began to emerge. Tetlock calls these people superforecasters.
These men and women were able to make surprisingly accurate predictions, consistently surpassing predictions by everyone from financial experts to trained intelligence analysts using nothing more than an internet connection and their own brains.
The superforecasters were incredibly diverse, and they didn’t have any special training. They were housewives, unemployed factory workers, lawyers, and math professors. However, superforecasters did have some common traits:
- They were clever on average, but were by no means geniuses.
- They believed the world is too complicated to boil down into a single slogan.
- They were humble in the face of complexity.
- They were comfortable with numbers and basic statistical concepts.
- They had a healthy appetite for information.
- They were willing to revisit their predictions in light of new data.
Most importantly, they had a “growth mindset,” a mix of determination, self-reflection, and willingness to learn from one’s mistakes. The best forecasters were less interested in whether they were right or wrong than in why they were right or wrong. They were always looking for ways to improve their performance.
The superforecasters’ diverse backgrounds and mental attitudes demonstrated that prediction is not only possible, it is teachable.
At Omidyar Network, we’re in the business of creating social impact in complex and dynamic environments. As part of our evolution to being a more effective learning organization, we use a rigorous hypothesis-based approach (essentially a forecast with an explanation) in our strategies and investments. We aim to adopt a growth mindset like the superforecasters’ as we make hypotheses, observe and measure our results, and reflect on our learnings.
While we may never fully know in the present what will happen in the future, it is worth asking:
How can you adopt the growth mindset of a superforecaster and become better at making accurate predictions?
#AlwaysLearning
Roy
Our Friday Learning Notes series is designed to share insights from Omidyar Network’s journey to become a best-in-class learning organization. Grab a cup of coffee and start your own Friday morning learning journey! *warning: side effects of regular reading may include improved mood, upswing in dinner party conversation, and/or increased desire to cultivate learning for social impact
