Survivorship Bias
What damaged planes and horse racing can reveal about our mind’s hidden blind spots
Back during World War II, the Allies were engaged in a brutal bombing campaign against German cities.
As one might expect, Germans didn’t enjoy being bombed very much. German guns were doing their best to shoot down Allied aircrafts, and preventing losses fast became a major issue.
It was apparent that planes needed more armor to make them sturdier against German fire. But the more armor you added, the heavier the plane got, which meant higher fuel costs and diminished range.
So it became very important to know exactly where on a plane to add armor in order to have the highest return on investment without weighing down the machines.
A Counter-Intuitive Solution
As You Are Not So Smart’s David McRaney tells it:
The military looked at the bombers that had returned from enemy territory. They recorded where those planes had taken the most damage. Over and over again, they saw the bullet holes tended to accumulate along the wings, around the tail gunner, and down the center of the body. Wings. Body. Tail gunner. Considering this information, where would you put the extra armor?
The obvious solution would be to put the armor in the spots featuring the most bullet holes. But army statistician Abraham Wald realized that the military was looking at a biased sample: one that only included plane that made it back from their mission.
In other words, none of the bullet impacts they could now observe had been critical enough to shoot down the plane. Consequently, these impacts did not correspond to the spots in most dire need of being reinforced, and it followed that these spots were actually the ones without any bullet impacts.
This is a classic example of survivorship bias: in situations where a cut-off process kills off part of a group’s members and we are only presented with the survivors, we often fail to consider that the group may originally have had other members, and how this may impact the situation at hand.
Reverse Survivorship Bias
It’s interesting to notice that survivorship bias can also work in reverse: failing to consider that the original group didn’t include more members.
A familiar illustration comes from amateur sports competitions: little Timmy’s third place trophy looks pretty impressive, until you learn that the whole category only had three participants to begin with.
In both this example and the bombers story, we’re confronted with an end result, but fail to realize that knowing the sequence of events that led to this result is vital if we want to put it back in its proper context.
World Cup Predictions
Survivorship bias also abounds online. Back in the summer of 2014, the World Cup had just ended. Fresh from their amazing upset against Brazil, Germany had just beaten Argentina in the finals.
Nobody could’ve predicted something like this. Nobody that is, except for one awesomely prescient Twitter user.
Not only did @FifNdhs predict the final score, but they also identified which player would score the winning goal one week in advance.
What had happened? Were the matches fixed? Had Twitter been somehow hacked to change the tweet’s date? Or were this user’s precognition powers the real deal?
The truth was much more pedestrian: @FifNdhs had simply tweeted out every possible scenario (Germany winning, Argentina winning, Player X scoring the goal, Player Y scoring the goal, etc.) one week in advance, and then deleted all their erroneous guesses once the actual results were known.
Like with all good tricks, it’s so simple that you wonder how you ever fell for it. But because of survivorship bias, we never stop to consider that this final remaining tweet might not have been so unique after all.
Surviving Search
The digital nature of the online world makes similar schemes a lot easier to automate.
Have you ever typed a few keywords into Google, and been surprised to find a whole page apparently dedicated to the very topic you searched for?
For example, typing “atlanta to phoenix flights” in Google brings up a paid link for a page entitled precisely “Atlanta To Phoenix Flight”. What are the odds!
A naive user might click that link thinking they’ve been really lucky to find a site that specializes in this particular connection. After all, we’ve been trained to believe that on the Internet, each link corresponds to a specific piece of content.
But of course, the truth is that this site has generated thousands of landing pages for every possible flight search, and one of them is simply getting matched up with your query through Google’s ad serving algorithms.
This is another subtle application of survivorship bias: in this case, the search itself acts as a cut-off process, conferring artificial value to the “surviving” link by hiding its countless duplicates.
Survivorship Scams
So far we’ve seen survivorship bias used online for harmlesss pranks and increased clickthrough rates. But the damange could be much, much worse.
British illusionist and hypnotist Derren Brown’s The System documentary shows how to use survivorship bias to scam someone out of their hard-earned cash.
The method is simple. First, obtain a large enough list of contacts. Email each one saying you have the power to predict the results of horse races, and that you’ll prove it to them.
Assuming each race involves five horses, divide your contact list in five groups, each of which you’ll “predict” a different winner for.
You’ll fail miserably with 80% of your list, but 20% will believe that you were able to predict the correct outcome at least once.
Repeat the process enough times with a large enough list, and after 4 or 5 rounds of this you’ll end up with a few die-hard believers convinced you can accurately predict the outcomes of horse races, and more than ready to trust you with their savings.
Now Derren’s version involved personally emailing and contacting hundreds of people. But do this online, and you can easily automate the whole thing. It doesn’t take much of a leap to imagine similar scams around online sports betting or stock trading.
Conclusion
Most psychological biases gently nudge us to act in one way or another. But survivorship bias is unique in that it creates entire blind spots in our reasoning, with potentially dramatic consequences.
So whenever you’re trying to extrapolate rules or guidelines from a group, ask yourself first how that group was selected. You may find that a hidden cut-off process is biasing your sample, and warping your results.
And above all: whatever you do, don’t trust people’s Twitter predictions!
If you’d like to know more about how psychological biases influence us online, sign up for my newsletter and I’ll let you know whenever I publish a new article.
