Survivorship Bias Makes You a Poor Decision Maker

During World War II, researchers at the Center for Naval Analysis faced a critical problem. Many bombers were getting shot down on runs over Germany. The naval researchers knew they needed hard data to solve this problem and went to work. After each mission, the bullet holes and damage from each bomber was painstakingly reviewed and recorded. The researchers poured over the data looking for vulnerabilities.

The data began to show a clear pattern (see image below). Most damage was to the wings and body of the plane.

Abraham Wald Suggested an Alternative Reason

Abraham Wald was a Hungarian mathematician and a member of the Statistical Research Group (SRG), where he applied his statistical skills to various wartime problems.

The holes in the returning aircraft were areas that need no extra armor — since the bombers could take damage and still return safely. On the other hand, the areas where the returning aircraft were unscathed are those areas that, if hit, would cause the plane to crash and be lost.

The solution to their problem was clear. Increase the armor on the plane’s wings and body.

But there was a problem. The analysis was completely wrong.

Wald’s review pointed out a critical flaw in the analysis. The researchers had only looked at bombers who’d returned to base.

Missing from the data? Every plane that had been shot down.

But the research wasn’t a wasted effort. These surviving bombers rarely had damage in the cockpit, engine, and parts of the tail. This wasn’t because of superior protection to those areas. In fact, these were the most vulnerable areas on the entire plane.

The researchers’ bullet hole data had created a map of the exact places that the bomber could be shot and still survive.

With the new analysis in hand, crews reinforced the bombers’ cockpit, engines, and tail armor. The result was fewer fatalities and greater success of bombing missions. This analysis proved to be so useful that it continued to influence military plane design up through the Vietnam war.

This story is a vivid example of survivorship bias. Survivor bias is when we only look at the data of those who succeed and exclude those who fail.

Survivorship bias is all around us, especially in the media. You read articles about entrepreneurs who risked everything financially and are now a success. But no one profiles the hundred other entrepreneurs who followed the same strategy and went bankrupt.

Or consider the business classic, Good to Great, which profiled successful companies and the characteristics that made them “great.” But what about all the companies that failed but also had “Level 5 Leaders” and “the right people on the bus”? The analysis excludes these companies “missing from the data”.

Wald then proposed that the Navy reinforce areas by adding more armor to them — which was a perfect demonstration of how to not fall prey into the survivorship bias.

What is survivorship bias?

Survivorship Bias is a logical error that leads to false conclusions by concentrating on the people or things that made it past a particular selection process. And when we do this, we tend to overlook those that got ignored, typically because of their lack of visibility.

It happens a lot in our day-to-day lives and negatively impacts our decision-making. A great example is copying what successful people have done and receiving advice from the so-called gurus and experts:

• Bill Gates, Steve Jobs, and Mark Zuckerberg all dropped out of college and became wildly successful. But for most college dropouts, it means unemployment, limited future career opportunities, and potentially more immediate student debt.
• Messi, Cristiano Ronaldo, and Neymar are getting paid highly as soccer players. But the truth is, most players never make it into a game in their lifetime. In the United States, only 1.4% of soccer players who play at the college level go on to be professional soccer players.
• Motivational gurus talk about following your passion and trusting your gut feelings — but there is no shortage of people who followed their passion and ended up seriously wrong.

When we’re listening to the success stories in any field, we get inspired by the companies, portfolios, and people who made it to the top. What we don’t hear and see are those who tried and failed because generally, people don’t talk about them.

How to avoid falling prey into survivorship bias

Understand that survivorship bias itself helps to prevent it from happening in the first place. When you understand what it is and means, it becomes easier for you to spot it in future situations.

The next step is to seek the other part of the story that is missing.

When solving a problem, ask yourself if you’re only looking at the ‘survivors.’

Your solution might not be in what is there, but what is missing.

Sources

Yeong, Dean, Survivorship Bias: What World War II Taught Us About Our Mental Flaws, https://www.deanyeong.com/article/survivorship-bias.

–, Abraham Wald, Wikipedia, https://en.wikipedia.org/wiki/Abraham_Wald.

Bragdon, Trevor, When data gives the wrong solution, September 7, 2017, https://www.trevorbragdon.com/when-data-gives-the-wrong-solution/.

–, What Percentage of Soccer Players Go Pro?, Authority Soccer, https://authoritysoccer.com/what-percentage-of-soccer-players-go-pro/.