Conquering A/B Testing (2): How should I set my metrics for an A/B Test?

This article is the second post of Hackle’s ‘Conquering A/B Test’ series. This series deals with the common questions many people have with regard to the entire process of A/B test design, preparation, result interpretation, and final decision-making. To check out the first post in the series: “When should I conduct A/B testing?”, click here.

Hackle’s Conquering A/B Testing covers the following topics:

1. When should I conduct A/B testing?

2. How should I set my metrics for an A/B test?

3. How long should an A/B test run for? What should my sample size be?

4. How should I set the user identifiers?

5. Can we deduce a causational relationship from an A/B test?

6. When is the right time to stop an A/B test?

7. How can I reach my conclusions with ambiguous A/B testing results?

8. What should I do if I want to restart an A/B test?

9. How should I conduct review processes within my organization?

‍

***

In the process of designing your A/B test, you will inevitably face problems related to setting the right metrics. Setting the correct metrics is important in order to confirm and validate the changes in user behavior caused by the A/B test.

You may assume that in an A/B test, you only need to check the metric that you want to improve from your website or app, such as the click-through rate or purchase conversion rate from your new version of the page compared to your existing version.

However, is this “success” metric or the metric that defines the key success of your A/B test really the one and only metric you should keep tabs on?

Why Do We Need Different Metrics (other than Success Metrics)?

Let’s take a situation where you release a feature to reduce the overall subscription cancellation rate. You want to conduct an A/B test experimenting with a new feature, a cancellation prevention message that pops up whenever the “cancel subscription” button is clicked.

• As your main indicator of success for the A/B test, you set the subscription cancellation rate or the number of subscription cancellations as your success metric.

However, even if it was found that the number of subscription cancellations decreased during this A/B test, data for the latency metric showed that the latency of the service increased due to the pop-up page, resulting in customers who were unable to reach the final subscription cancellation page. This scenario could consequently lead to an increase in customer support inquiries related to subscription cancellations.

In this case, can we conclude that the A/B test was successful just because the success metrics moved in the desired direction?

The above scenario shows that if important company-wide guardrail metrics are not monitored together, there is a risk of making a wrong decision.

Although it may vary depending on the scope of influence of the experiment, it is desirable to set at least 4 to 5 metrics in a single A/B test, and dozens of metrics may be set in an A/B test with a large company-wide impact.

Success Metrics, Supporting Metrics & Guardrail Metrics

We learned from the above scenario that you should not start your A/B test with only a single success metric set. This is because even if that specific metric moves in the desired direction during the A/B test, other important metrics may be indirectly affected and move towards an undesirable direction. So what exactly are the other metrics that need to be set alongside the success metric?

At the most basic level, it is recommended to set a success metric (the main primary metric that can test the hypothesis of your A/B test), guardrail metrics (the metrics that are important to the company or can be negatively affected by the A/B test), and various supporting metrics (the metrics to keep tabs on along with the main success metric).

• As guardrail metrics, you can set sales, purchase conversion rate, latency, and the number of customer support requests which are all important company-wide metrics to keep tabs on. You can also include the number of customer support inquiries related to subscription cancellation, which is also an important metric managed by the service sector.
• As supporting metrics, you can set the total number of membership inquiries, the number of users who enter the membership page, or any other metrics that could be affected by the A/B test with the new pop-up feature and wouldn’t hurt to check.

Setting Metrics for A/B Tests (with Hackle’s A/B Test Dashboard)

A/B testing platform, Hackle, provides the following metric setting options.

Click on the Set Metrics button‍. (Source: Hackle Dashboard)

Set A/B test metrics based on events or user actions, ex. sign up, purchase, item viewed. (Source: Hackle Dashboard)

‍

If the desired metrics do not appear in the recommendation list, you can create your own metric by setting the numerator and denominator of the metric. (Source: Hackle Dashboard)

You can set your own desired metric, such as ‘average purchase amount per user who purchased an item or triggered a purchase event’. (Source: Hackle Dashboard)

‍When using various different A/B testing platforms, many users find it hard to set the exact metric they want, and in most cases, the metrics are not calculated in the way they desire. As seen from the above screenshots, Hackle’s dashboard allows you to manually set the event, calculation type, and filter for both the numerator and the denominator of a metric, allowing you to have a high degree of control and flexibility over the type of metrics you can set for your A/B test.

Check out Hackle at www.hackle.io in order to start creating your own A/B tests ‍to release the right features to maximize your customers’ experience.

‍