Stop! Drop everything you’re doing.

The importance of finding your ‘Drop Dead Metric’ whilst experimenting on an established product.

We product people love to run experiments, we can’t get enough of them. A good Product Manager will spend a lot of time upfront understanding the key metrics that will help them understand the real impact of changes that aim to deliver great business results.

We want to run experiments to create positive outcomes, we don’t do things because we believe they will have a negative impact. In an established product or business, there is often a certain level of ‘status quo’ to maintain to help continue to pay the bills. It then becomes vital that we also consider ‘what does bad look like?’.

Here enters the Drop Dead Metric.

The Drop Dead Metric is the point at which you stop any experimentation and resume normal activities. This is the point that your experiment is causing such a negative impact that it might create significant long-term damage to your product.

For example, when the FT launched a new homepage in December our main objective was to increase editorial flexibility and improve CTR but we included a Drop Dead Metric to sustain X amount of Ad Impressions Per Page View to ensure we can continue to meet our advertising objectives.

With recent Google Search changes meaning publishers no longer need to use Google’s Accelerated Mobile Pages (AMP), we at the FT have been considering how we might experiment to understand the impact on FT search traffic if we decide to turn it off.

Whilst there are many reasons the FT may want to stop using AMP, as a mature business we rely on a certain level of search traffic to help us acquire new customers, help existing subscribers to re-visit the FT, and to increase our advertising inventory.

Due to technical constraints, we are unable to conduct a standard AB experiment leaving us with the option of just turning it off and measuring the impact. Adding to the complexity is that search traffic is highly volatilie due to ever changing news agenda.

Case in point, when the FT started to cover the Evergrande story, when it was announced they may collapse due to excessive levels of debt, we saw an uptick in search traffic, almost 50% higher than in the previous days.

So, with the FT being an established business, dependent on Search traffic, we needed to come up with our Drop Dead Metric so we can begin experimenting with AMP. At what point would we need to turn AMP back on because the impact on the business is too painful?

We landed on using the proportion of mobile search traffic to FT.com: whilst still dependent on news agenda, it’s going to be less volatile than actual page view volumes. We calculated the average standard deviation over the last 90 days (to account for changes in Google Search algorithms) and decided that anything outside two standard deviations away from the mean would give us enough confidence to say that switching off AMP has had a significant negative impact that falls outside normal ranges. You can read more about standard deviation and how to use it here.

Whilst we have not begun experimenting, having the data point upfront allowed us to get stakeholders on board. As we did our research and we understood their challenges and potential concerns, they have put their trust in us not to break everything and given us the green-light.

When we eventually begin experimenting with AMP, we can use this Drop Dead Metric as a daily sense check just to make sure we are heading in the right direction. Coincidentally, this is also our success metric for this project but in many other projects, this won’t be the case.

Here’s how you can approach and implement a drop-dead metric for your product:

1. Chat with your different stakeholders and understand their key business objectives.
2. Identify a metric (ideally one) that has to be sustained at all costs. If this metric was to be negatively impacted, people would start to panic.
3. Work with analytics and your stakeholders to define a ‘danger zone’ for your metric.
4. Agree and communicate with the rest of your team as needed
5. Once the test is up and running, review the metric at regular intervals and communicate with stakeholders for reassurance.

Final thoughts… getting the right metrics to measure the success of your feature is vital, but if your business relies on a minimum level of something (conversions, visits, ads etc), it is essential that you understand it and incorporate it into your test. This will give you and your stakeholders the confidence that you appreciate their challenges and are ready to mitigate risk/avoid damage if needed.