Is It Time To Start A/B Testing?
Five Signs That Suggest Your Organization Is Ready
A/B testing is a strategy to evaluate changes made to a product on a subset of users using metrics to quantify the impact.
Many companies already use this methodology to understand how product decisions affect their key business, product, and user metrics. However, many more companies are not practicing A/B testing that have good reasons to try it. If you have yet to incorporate A/B testing as a prerequisite to deploying product and engineering changes to users, consider the following indicators. If one or more of these things are true, your organization is ready to invest in building an A/B testing platform:
1. Product changes are not well-received. Your leadership team realizes recent changes to the product have fallen flat — users have expressed disinterest in a feature, or you’ve seen degradation in critical metrics and are unsure what feature resulted in such an outcome.
2. Goal to be data-driven. Your engineering and product organizations want to be more data-driven.
3. Already have a feature flag system. You’re already releasing changes behind a feature flag or a system that can direct users to specific features and, therefore, can extend the same engineering system to support segmentation logic for an initial A/B testing platform.
4. Desire for bigger changes, but risk averse. Your product team wants to make more significant, grander changes to the product, such as drastic changes to the UX but is a bit risk-averse as they’re unsure of the impact it would have on key business metrics.
5. Evaluation of machine learning models. Your product and engineering teams are investing in applications for machine learning and need a way to evaluate changes beyond offline metrics.
If this list is relatable, check out my book Practical A/B Testing for a practical way to get started with this experimentation strategy.
