How to Make Better Predictions: Superforecasting [Book Summary]
Will either the French or Swiss inquiries find elevated levels of Polonium in the remains of Yasser Arafat’s body?
It is 2012, and after Polonium has been found in the late Yasser Arafat’s clothing, an investigation is opened. The goal is to figure out whether Mr. Arafat may have been poisoned.
How would you go about predicting the result to that question?
Superforecasting
In his book Superforecasting, Philip Tetlock describes how some of the world’s most accurate predictors go about doing just that. He and his co-investigators ran the original version of Good Judgement Project (GJP) from 2011 to 2015. In this prediction tournament, they recruited thousands of people who made 100,000s of thousands of predictions on questions just like the one above.
They used a numerical approach using a Brier score to measure the accuracy of predictions.
The very best 2%, they started calling “Superforecasters”. These people were able to beat US Intelligence officials.
Now were these all supergeniuses?
Turns out that general intelligence does matter, but you don’t need to be in the top 2% of IQ in order to be in the top 2% of forecasting. This suggests that a lot of what led to top forecasting performance can be learned.
Did they have access to superior information?
No. They were well informed, but mostly because they knew how to use Google. The extra information US Intelligence officials had wasn’t enough to beat the superforecasters.
How to forecast like a superforecaster
Let’s look at how Bill Flack, one of the people who consistently outperformed his peers in the GJP, approached this question. It’s important to note that Bill lives in Nebraska and had no special knowledge of the Isreali-Palestinian conflict.
Now, did Israel poison Arafat?
Step 1: Fermi-ize the Question
First, Bill realized the first step in his analysis had nothing to do with politics.
In fact, if you carefully reread the question at the beginning of the article, it asks whether elevated levels of Polonium will be found. Not at all whether Israel poisoned him. This is a “bait-and-switch” our System 1 pulls on us, replacing a question with one we might have a strong feeling about. “Of course Israel did it!” or “No, Israel would never do that!”.
So Bill asked himself: “What needs to be true in order for the answer to be yes? What needs to be true for the answer to be no?”
This is where you break down an intractable problem into more tractable sub-problems, also dubbed Fermi-izing.
One thing that became clear to Bill is that he first needed to understand if elevated levels of polonium could even be found eight years after someone’s death. (Arafat had died in 2004.) So he familiarized himself with how polonium decays over time and how the science of testing works. Only once he was satisfied they still could measure it, did he move on.
He also realized that only either the Swiss or French team had to find elevated levels, raising the probability of the answer being yes.
The next step was to list all the possible ways elevated levels of polonium could end up in Arafat’s remains.
Of course “Israel poisoned Arafat” is one possibility. But the leader also had many Palestinian enemies. They could have poisoned him. Or Palestinian special forces could have planted the polonium there, wanting to make it look like “Israel pulled a Litvinenko”.
This is important because they raise the probability that Polonium could be found.
Breaking down a question into subquestions is something dubbed Fermi-izing, after Enrico Fermi, a physicist who loved to answer difficult questions by breaking them down into more easily answerable subquestions.
Step 2: Outside View / Base Rate
So the next step is to delve into Middle-Eastern politics and answer the subquestions, right?
Not yet.
For example, let’s assume you are asked to solve this question:
Julia lives in Zurich and about 8 km away from her work. She’s 25, single and lives with two other roommates. She’s got a background in mechanical engineering and now works in management consulting.
How likely is it that she bikes to work?
Many people, including very smart ones, have the tendency to immediately focus on the details of the question such as “Ok she’s young and single, so she likely doesn’t have a car”, and so forth.
While this makes for a nice story, it can lead to predictions that are way off.
What superforecasters like Bill do, is that they first establish how many people in Zurich commute by bike in general. A few minutes on Google tell us that it is about 10% of all commutes.
Now we can adjust this 10% with some of the details of the story and tweak the 10% number up or down.
So the next step after fermi-izing is to find the base rate of similar events.
Turns out the best forecasters first find a way to anchor their predictions in a way that minimizes hunches and feelings about any particular details of the story.
In the Arafat question, this is quite hard. It’s not like leaders get killed each day and we have a base rate for this. So how would we go about it in this case?
We can make some further fermi-izing here.
What is the likelihood of authorities hiring a renowned Swiss research institute to conduct a study in the first place?
The current level of evidence has to be inconclusive. There can’t be sufficient evidence that polonium is there, otherwise, the test wouldn’t make sense. So maybe an upper bound would be 80% probability. On the lower end, if there was no suspicion at all, they also wouldn’t start an investigation. So maybe the lower bound is 20% likelihood. The mean of those two bounds ends us at a 50% likelihood. While far from perfect, it does give us an anchor more detached from any individual details of the story.
Step 3: Inside View
Now that we have a base rate probability of 50% established, we can start looking at more details for the individual questions to adjust this crude initial estimate up or down.
Let’s look at the first hypothesis from Step 1 “Israel poisoned Arafat”. What needs to be true in order for this to be true?
Fermi-izing again, we get to something like
- Israel wanted Arafat dead badly enough to go through with such a plan.
- Israel had or could obtain polonium.
- Israel had the means to poison Arafat with polonium.
Then the research can start. This is where you read intensely on the internet. Getting good information requires intense detective work. You use what you learn here to adjust your base rate estimate up or down.
You repeat this for the other hypotheses you’ve laid out.
Step 4: Share your view and have it challenged
After your done with your analysis, the best thing you can do is to share it with others and have it challenged. In the GJP, the forecasters who worked in teams outperformed the ones who didn’t. The more different perspectives you have access to and can synthesize into your final view, the more likely you are to be correct.
Step 5: Update your Views
When the Swiss research team announced it would be late in releasing its test results because it needed to do more — unspecified — testing, Bill Flack perked up.
What did that mean?
It could be just that a researcher had gotten ill for a few days and that some of the results weren’t verified yet.
But Bill also knew that polonium could also occur when naturally present lead in the body decays into polonium. In order to determine the source of polonium, researchers can remove all the polonium and wait to see whether polonium reappears. This would indicate lead was the source.
This could be the reason for the delays. The Swiss team had actually found polonium and needed to run another test to exclude its source was lead.
But this was only one possible explanation.
So Bill cautiously updated his forecast to 65%.
The Swiss ended up finding elevated levels of polonium.
In fact, superforecasters are very frequent to adjust their predictions. The research by Tetlock’s co-investigator Mellers shows in Mellers et al. 2015, that the number of predictions per question had the largest correlation with low Brier Scores of all the variables studied.
Summary
- Fermi-ize the question
- Start with outside view
- Adjust with the inside view
- Challenge prediction with others
- Update your views frequently
I should also mention that you can get better over time and that getting feedback on your predictions from others will improve your abilities over the long run. As with everything, this requires grit and discipline to stay on track.
Further Resources
There are a ton more nuances on how to make accurate predictions. Here is what you can do next:
Read the Book by Tetlock
Go through the free training of the Good Judgement Project
Join the Good Judgement Project and start practicing and learning from others.
Also, what is a prediction about your life you can get started with practicing!?
Subscribe below to get the next article straight to your inbox.
Flavio Rump is an entrepreneur and investor. He shares decision-making models from the world’s best decision makers. You can read his articles, watch his YouTube Videos or join his free newsletter to learn how to make better decisions.
