When data gives the wrong solution
During World War II, researchers at the Center for Naval Analysis faced a critical problem. Many bombers were getting shot down on bombing runs over Germany.
The naval researchers knew they needed hard data to solve this problem and went to work. After each mission, the bullet holes and damage from each bomber was painstakingly reviewed and recorded. The researchers poured over the data looking for vulnerabilities.¹
The data began to show a clear pattern (see picture above). Most of the damage was to the wings and body of the plane.
The solution to their problem was clear. Increase the armor on the plane’s wings and body.
But there was a problem. The analysis was completely wrong.
Before the planes were modified, a Hungarian-Jewish statistician named Abraham Wald reviewed the data. Wald had fled Nazi-occupied Austria and worked in New York with other academics to help the war effort.
Wald’s review pointed out a critical flaw in the analysis. The researchers had only looked at bombers who’d returned to base.
What was missing from the data? Every plane that had been shot down.
But the research wasn’t a wasted effort. These surviving bombers rarely had damage in the cockpit, engine, and parts of the tail. This wasn’t because of superior protection to those areas. In fact, these were the most vulnerable areas on the entire plane.
The researchers’ bullet hole data had created a map of the exact places that the bomber could be shot and still survive.
With the new analysis in hand, crews reinforced the bombers’ cockpit, engines, and tail armor. The result was fewer fatalities and greater success of bombing missions. This analysis proved to be so useful that it continued to influence military plane design up through the Vietnam war.
This story is a vivid example of survivor bias. Survivor bias is when we only look at the data of those who succeed and exclude those who fail.
Survivor bias is all around us, especially in the media. You read articles about entrepreneurs who risked everything financially and are now a success. But no one profiles the hundred other entrepreneurs who followed the same strategy and went bankrupt.
Or consider the business classic, Good to Great, which profiled successful companies and the characteristics that made them “great.” But what about all the companies that failed but also had “Level 5 Leaders” and “the right people on the bus”? The analysis excludes these companies “missing from the data”.
How to Use It:
When solving a problem, ask yourself if you’re only looking at the ‘survivors.’
Your solution might not be in what is there, but what is missing.
Thanks for reading! If you enjoyed it, hit that heart button below : )
Footnotes
- As told Matthew Syed in his excellent book, Black Box Thinking pages 33–37.
Originally published at trevorbragdon.com on September 7, 2016.
