Why we solve the problem of survivor bias in our market research and marketing

Survivor bias, also known as survivorship bias, is a phenomenon that causes your marketing or market research to fail. It’s the reason why certain kinds of marketing advice never seem to work.

You must be wondering why I’m saying this, being a digital marketer, myself. While I’m definitely not saying that all marketing advice is destined to fail and that you should ignore it when it comes your way, survivor bias is something you need to be aware of in this market.

Above all, follow the advice that forces you to dig deeper and answer the tough questions — not advice that simply tells you to march in a certain direction.

Survivor bias is the human tendency of focusing on successful outcomes — people, businesses, and strategies — and ignoring the accompanying failures that lead to these successes, simply due to the lack of visibility of these failures.

This behaviour can lead to false conclusions or overly optimistic beliefs, simply because failures are ignored. It can also lead to the false belief that our successes are the result of special tactics rather than mere coincidence or luck.

It might be difficult for you to grasp why survivor bias is bad, because why shouldn’t we focus on how success came about? The problem, here, is that survivor bias creates this false idea that these successes will work for anyone at any time, which is simply not the case.

We are taught from an early age to learn from our mistakes. Survivor bias is the exact opposite of this — it focuses on what worked over what did not. It, therefore, creates a template for others to follow and gives you the false assurance that this template is sure to work every time.

What’s scary about this is how we tend to adopt these opinions to structure businesses and make decisions, without examining all the data, and then stare in disbelief when it leads to failure. Learning how to avoid survivor bias is, therefore, a crucial step in making sure that your efforts are not wasted.

The history of survivor bias

In World War II, the Allied forces wanted to add protective armour to their warplanes. Since resources were limited, experts had to decide where exactly they should include it for the planes to benefit from additional protection.

To do so, the experts decided to study the planes that had been shot but successfully made it back home. They found out that these planes had incurred bullets around the tail, body, and wings, except the cockpit and engine.

So, this seemingly obvious train of thought led them to place armour around the places that had incurred bullets and avoid the places that had no bullet holes. Sound logic, right?

If you really think about it, though, the planes they studied are the ones that made it back to the base, meaning that the bullets weren’t fatal to the plane — the tail, body, and wings could survive a rain of bullets.

Mathematician and statistician, Abraham Wald, was the one who pointed this out and recommended that the military attach the armour to the areas that the surviving aircrafts had no bullet holes, instead. In all probability, he saved many lives and also gave birth to the concept of survivor bias.

How does survivor bias apply to market research?

Let’s imagine, hypothetically, that I have a client who sells wooden furniture and they want to venture into producing metal furniture to increase their profit. Our team then analyses types of furniture that were profitable over the past year and find out only 10% of them are made out of metal.

Based on this market research, we will convey to the client that they should not venture into this idea and that it’s profitable to stick to wooden furniture because research shows that that’s what made the most profit.

However, what if in the unprofitable segment of furniture models, only 1% are metal furniture? What if out of the rest of the 9%, most are wooden furniture?

The lesson we need to learn from this data set is that it would make more sense for the client to expand their collection of furniture to include metal furniture if they want to increase profit. Just by looking at the success rates, we become victims of survivor bias and are prone to make the wrong decisions.

How does survivor bias apply to marketing?

Just imagine, if all that’s on the internet are stories about winners — then you only read tactics as part of success stories.

Say I write about an email campaign that worked well for our company. In it, I point out how I used a blue background for the attached brochure instead of a white background. After reading this post, someone decides to use blue backgrounds in their email campaigns.

But what if 99 other people who used blue backgrounds for the emails in their email campaign failed? No one reads about these experiences! The only story out there that’s read is my success story, the survivor story. So, the audience believes that emails with blue backgrounds are the winning tactic for email campaigns.

It isn’t necessarily bad to stick to a strategy with a 1% success rate. It’s better than a 0% success rate, but only if you know what made it work in the first place.

The risk of survivor bias is that it often makes the wrong tactics look good. When readers see this on the internet, they stop looking, don’t make further inquiries, and stop digging. Since these tactics aren’t surefire ways to succeed, this leads to disappointing results more often than not.

How do you avoid survivor bias in the marketing field?

You are now aware of what survivor bias is and how it can present itself in market research and marketing. Meaning, you are less likely to be fooled by it. So, even if I did put myself in a position for you to question what I have documented in our blogs, this article has helped you understand that what worked for me, may not necessarily work for you.

Recognise that what works for one person may not work for another. Understand that with one success there could be a thousand attempts that took the same path and failed and that luck and chance play a big role in success.

Don’t stop questioning

Always keep digging! Don’t stop at the success stories. Question why something worked for it to be successful. Dig deeper into what did not work and what went wrong on their path to success. Do some research to see if that tactic worked for other people/businesses too.

Be very sceptical of individual gurus, books, podcasts, and other resources that appear to give you the ‘answer’ or ‘secret formula’ to achieving wealth, success, and happiness. This all may sound hypocritical because I write a lot of tactical blog posts — in fact I read and use this material because they’re fantastic resources.

Take a balanced approach

What’s important, though, is that you ask yourself ‘why did this work?’ before you blindly use it.

Don’t dismiss the most successful, influential and popular success stories out there. Clearly, there’s something that’s done right. All I’m saying is consider the potentially devastating downsides and consequences that lead to these successes too.

Instead of trying to mirror these wins, draw some key takeaways and come up with your own way to incorporate them into your business.

Do what you need to do but always account for survivor bias

Remember — don’t take any advice as the law. Take it as one of several considerations. Only by being aware, questioning and taking a balanced approach will you be able to find your own path to success.

Originally published at https://www.hypeinsight.com on January 20, 2020.

The Startup

Get smarter at building your thing. Join The Startup’s +788K followers.

Sign up for Top 10 Stories

By The Startup

Get smarter at building your thing. Subscribe to receive The Startup's top 10 most read stories — delivered straight into your inbox, once a week. Take a look.

By signing up, you will create a Medium account if you don’t already have one. Review our Privacy Policy for more information about our privacy practices.

Check your inbox
Medium sent you an email at to complete your subscription.

Ari Vivekanandarajah

Written by

Ari has over 15 years of experience in Content creation and digital marketing. He has published a book on marketing and has contributed to numerous companies

The Startup

Get smarter at building your thing. Follow to join The Startup’s +8 million monthly readers & +788K followers.

Ari Vivekanandarajah

Written by

Ari has over 15 years of experience in Content creation and digital marketing. He has published a book on marketing and has contributed to numerous companies

The Startup

Get smarter at building your thing. Follow to join The Startup’s +8 million monthly readers & +788K followers.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store