What can weaken your Growth hacking strategy ?

Usually we see startups and companies basing their growth hacking strategies of what another company has done or of one A/B experiment that worked for them one time .We ended up by reading headlines like “How these folks improves their retention by 800% with this strategy .”

What you don’t see, is what number of companies attempted that same strategy and didn’t perceive any results. whether, they lost money or they invested such a great energy in it without any outcomes. Similarly, if you’re making a decision based on one experiment that worked one time for you but failed ten others, you’re making a decision based only on what survived and are ignoring data from failed attempts and this called the survivorship bias.

In case you’re not aware what is survivorship bias, it is best clarified with the act of drawing conclusions from an incomplete set of data because that data has ‘survived’ some selection criteria.

Survivorship bias is one of the susceptible mistakes to make in when you are basing your growth strategy. In growth , there are a various ways that can prove itself. Let’s take a better look of how I’ve seen survivorship bias misconceive people’s cognition of growth and growth strategies.

Why Conversion Rates start to decompose !

One of the ways survivorship bias can manifest is when conversion rates decay over time. Let’s suppose you launch a new signup box on the landing page or another activation email that is essentially helps your conversion rate. Quickly after you deploy the experiment , everything looks awesome. However, when you return a 6 months later, you see that the conversion rate has relentlessly dropped back to your initial baseline from before the experiment deployment. What is going on here?

Well, when you initially deployed the new signup box, it was something new that nobody has seen before. Some percentage of users seeing the box for the first time decided to sign up . Those people are currently no longer part of the segment seeing the box since they have accounts. From that point, a part of people seeing the box for the second time chooses to sign up. Again, they are now expelled from the segment.

For who are growing at scale, in the end you’re left with a segment of who may have just seen that specific box 5 or 6 times and weren’t persuaded within those initial 5 times and will probably keep not be convinced from seeing the same box. What this means for a Growth team is they need to budget time to revisit important conversion prompts with the expectation that as users get acclimated to the prompt, its effectiveness will start to decline. On a broader scale, it might also mean you need to change up your entire conversion strategy as you move further along the S-curve of adoption.

Push Notifications Unsubscribes :

in this case I will discuss where survivorship bias can misdirect you is when its comes to push notifications engagement. A major area in email/push notifications is deciding the ideal frequency to notify users. Startups and retention folks frequently take a look at the notifications and email engagement of their current users to define what the ideal frequency to send. In any case, this can be a biased approach, in light that you’re existing users who are still getting notifications have decided that they are Ok with this frequency .Most who detested the frequency are already uninstalled or stop using your product .

To really find the ideal frequency to notify users , you have to test on a clean unbiased segment : new users that just joined. Just by experimenting on new users, who have never received messages/notifications from you recently, you will be able to figure out what the ideal retention frequency for your product .

The existing user base are biased :

Another way that survivorship bias can show itself is in your active user base. Each active user you have today has made sense of how to use your product and is getting enough value to keep on using it. Every other person that didn’t get it has likely already churned.

Experiments and A/B testing on new users will show you that your existing users had effectively made sense and figured out how to use the product and built up the propensity. Everybody who hadn’t made sense of that had already stop using your product. this made a lot of sense that the Experiments and A/B testing should be done on new users .

About The Author :

Badr El Fahim is a Growth Hacking Manager at Avito.ma , the leading classified website in Morocco and part of Schibsted Media group, an international group with employees in over 30 countries.