Member preview

Forecasting is not about choosing the right door (Pexels.com)

All of Life is a Wager

Overcoming Cognitive Biases and Becoming a Super Forecaster

If you want to learn about cognitive biases all humans share, you might start with Daniel Kahneman’s ‘Thinking, Fast and Slow’, or the paper he wrote in 1974 with friend and colleague Amos Tversky, ‘Judgment under Uncertainty: Heuristics and Biases’. There are solid summaries here on Medium, too.

But actively applying your newfound knowledge may prove difficult. After all, just because you know what biases you have doesn’t mean you can notice them at all times (nor do you need to).

Our System 1 takes care of most decisions while System 2 is only brought to bear occasionally. But, if we can get beyond what Kahneman called ‘What You See Is All There Is (WYSIATI), or what Tetlock calls ‘tip of the nose thinking’, then we can begin to employ strategies in overcoming cognitive biases when and where they matter.

Philip E. Tetlock and Dan Gardner have given us an interesting way to practice doing so in their book, Super Forecasting. In their story about a relatively small group of super forecasters who beat IARPA’s own government-backed researchers in a tournament of geopolitical forecasting, Tetlock and Gardner provide a blueprint for beginning one’s own journey of forecasting better than the average human (or at least better than you yourself do now).

Their results were impressive and their knowledge sharing is generous.

We are all forecasters

One of my favorite writers and thinkers is Christopher Hitchens, who once said:

All of life is a wager.

(This aphoristic phrase was ironically uttered when asked about his smoking and drinking habits, which might have contributed to his cancer of the throat.)

Nevertheless, the sentiment strikes me as true and powerful.

Even if I do not take on the challenge of becoming a super forecaster in the sense of Doug Lorch or Bill Flack (two of several super forecasters described in the text), or find myself at a conference table being presented with probabilities that Osama bin Laden is located at a specific compound in Abbottabad, then I can still apply an improved process of forecasting (and the informed decision-making that follows) to my own life.

Gaining comfort with probabilities (and uncertainty)

Some parts of this book were review for me. When a meteorologist reports there is a 70% chance of rain I am not surprised when 30% of the time it doesn’t rain, since this is precisely what the forecaster predicted. Where this can trip people up is in the prediction of important events, where a 30% chance, say, of a political candidate winning might have considerable consequence. Unlike the weather, which is forecast daily, political elections are one-time events.

I’ve played around 3,000 hours of poker at a competitive level (often against pros who made a living doing so), and in order to compete I needed to acquire more than mere heuristics for survival and an occasional flurry of profit.

For tough or close decisions, I would consult a legal heads-up display which monitored my opponent’s betting patterns. If, based on a reasonable sample size, I could gather from the data that my opponent was, say, 70% likely to fold to a 3rd raise before the flop, and his hand range for continuing with a 2nd raise was greater than 30% of hands, then (accounting for the pot-size), this 3rd raise I was about to make was easily profitable in the long run. I could have no buyer’s remorse if I ran into a big hand and had to fold to my opponent’s 4th raise. I routinely made decisions like this in a matter of seconds across 4–8 tables.

This repetition quickly inculcated me to profitable behaviors in the context of no limit texas hold’em.

The heads-up display was a reminder staring me in the face to make the correct decision, even if various mental roadblocks conflicted my conscious decision-making, either from distraction, fatigue, or anger at an unfair bad beat a few seconds prior at another table.

In short, I was able to overcome some of my cognitive biases in the narrow context of the digital felt.

Questions that affect my life

I like to get value from my reading through immediate application. The early chapters prompted me to consider questions I’d love to forecast, mainly existential questions, musings about human progress, and my own place in society:

  • Will we have an artificial general intelligence by 2042?
  • How long before my novel is accepted by a top-tier agent (my top 20)?
    Will it ever happen?
  • Will creative storytellers be in high demand in 10 years?

As I read on, I found some of my questions too difficult to answer according to Tetlock, mostly on account of their time horizons. Instead, I should look for Goldilocks Zone questions: not too easy, not too hard.

Clarity matters too. Tetlock and Gardner use Steve Ballmer’s prediction that Apple’s iPhone wouldn’t achieve significant market share to illustrate why the language matters:

First, Ballmer’s statement from 2007:

There’s no chance that the iPhone is going to get any significant market share. No chance.

Tetlock explains that this prediction lacks a time horizon and clarity. What does significant mean? What does market share mean?

In the extended interview of Ballmer, he also asserts:

There’s no chance that the iPhone is going to get any significant market share. No chance. It’s a $500 subsidized item. They may make a lot of money. But if you actually look at the 1.3 billion phones that get sold, I’d prefer to have our software in 60% or 70% or 80% of them, than I would to have 2% or 3%, which is what Apple might get.

The authors used data from the Gartner IT consulting group to show that in the third quarter or 2013 iPhone sales made up 6% of global mobile phone sales. The only question remaining is does the iPhone’s presence at 6% represent significant market share.

From this perspective, Ballmer’s statement isn’t as outlandish as it was made out in the press.

Better questions

I narrowed the field of questions. If I made forecasts I could then place more informed wagers in my writing projects.

  • Which book genre and sub-genre on Amazon will underperform beginning-of-year sales expectations by year end?
  • In the next 6 months, how likely am I to land a second ghost-writing job for a business-related book, having already achieved the first?

In the first case, prediction would allow me to take an extra writing project in an area where demand is underserved.

In the second, I would better be able to understand my near-term income sources and take on various smaller projects if I decided the probability was low, say under 20%.

But first I’d have to make a forecast

In the next section of ‘Super Forecasting’, I learned about creating a thesis, antithesis, and synthesis to form preliminary forecasts. I was particularly drawn to the concept of aggregation, which I feel serves to check the limits of one’s own information and knowledge against others’ thinking on the same question.

How we ask questions matters (Fermi-ize questioning)

Before the internet, Italian American physicist Enrico Fermi famously asked his students ‘how many piano tuners are there in Chicago?’

Tetlock came up with these four questions to help him make an educated guess.

  1. The number of pianos in Chicago
  2. How many pianos are tuned each year
  3. How long it takes to tune a piano
  4. How many hours a year the average piano tuner works

Answering this question without any researched information requires the ability to make estimates from prior knowledge about Chicago’s population, the number of households and businesses which may own a piano, and how often pianos are tuned and how long it takes a piano tuner to do the work.

For my own question, I would need to apply similar logic. Again, here’s my question:

  • In the next 6 months, how likely am I to land a second ghost-writing job for a business-related book, having already achieved the first?

Questions to ask might be:

  1. How many business people are seeking ghost-writers in English? (What’s the demand?)
  2. How many business ghost-writers are there in English? (What’s the supply, or what’s my competition?)
  3. What is the likelihood of beginning a conversation with individuals seeking a ghostwriter?
  4. How likely, once a conversation is begun, am I to emerge as the chosen writer for the project?

Interestingly, answering the first question will probably improve my chances considerably. This is an advantage of using the forecasting methods prescribed by Tetlock and Gardner to one’s own projects — you can affect the probabilities by your own actions.

Base rate and adjustments to forecasts

But there is much more. Many factors will help me assign my starting probability (or base rate), such as other in-flight projects that draw my attention away from any scheduled business development or samples I’m preparing for potential clients.

Once established, this opening probability would change over time as I adjusted my various routines. (Again, as an actor in the world I get to influence my own probabilities.) For example, I could improve the overall probability by allocating time to activities that I learn will markedly improve the probability of the outcome I want. Humorously, my fluctuating desire to achieve the goal may influence the probability from day to day.

Tetlock writes about the importance of using the outside view to establish the base rate before adjusting it with the inside view. Otherwise, in my example, on a day where the sun was shining as I looked out from my renovated flour mill apartment over the Mississippi River, I might inflate my expectations.

Can I apply this methodology to more projects?

As I set down the book, I realized I could assess many or all of my current work and writing projects with this methodology, creating something of a personal work wager matrix.

I probably won’t do this (at least not immediately), preferring to allow curiosity to guide, for example, my week-to-week reading, as I did for this essay. (I planned to read a different book today, but instead opened and closed Super Forecasting in just three sittings in under 24 hours.)

But maybe I’ll improve in my analysis and forecasting and be able to make these sorts of calculations faster and carry a working spreadsheet of sorts to manage my projects in a slightly more organized fashion than my status quo.

What will you forecast?

Tetlock and Gardner’s storytelling style, clear explanations, and memorable examples make this read an easy 5 stars. I’m already recommending to friends and family and colleagues who have begun but not finished Thinking, Fast and Slow.

The main question I suspect many readers will leave with is, What should I spend time forecasting?

If this happens to you as it did for me, then Tetlock and Gardner will have succeeded.

If you want to skip the book and try your chops at forecasting right now, you learn more here or make your first forecasts here.

Further reading:

Isaiah Berlin’s ‘The Hedgehog and the Fox’
The Success Equation, Mauboussin
Expert Political Judgment, Philip E. Tetlock

I need your help:

Can you help me find Doug Lorch’s methodology for serving himself a news article or book from a wide range of political leanings and sources? Apparently he has created a program that does this.

(Edit: Based on some searching, the code Doug used is primarily for self-use — we will have to find our own ways of getting diverse news sources! Cultivating this range of reading seems hard but necessary for each and every one of us.)