Learning A/B Testing: The Easy Way (Part 1)
I work as a Software Engineer and I have used A/B Testing to launch more than 100 features at various companies. A part of “The Easy Way” series: this blog post aims to demystify the fundamentals of A/B Testing.
First of all — what’s A/B testing?
A/B Testing is the process of conducting experiments to optimize your product’s customer experience.
In short, if you have to make some changes in your product — let’s say color of a button from blue to green. You have 2 variants of your product now -
- one having a blue button and
- the other having a green button.
Now, you need to divide your traffic into 2 parts —
Half of your users will get to see the blue button while the other half will get a green button. You would want to run the experiment for a certain time period (generally 1–2 weeks) and observe the behavior of the users in both the variants. The variant which shows better user engagement wins the experiment. Once you have a winning variant, you can now show that variant to 100% of your users.
Note that, although we have used a fairly simple example of changing the color of a button, this process applies to bigger changes like new user feature, improvements to UI, backend improvements, etc.
Why should you do A/B Testing?
Your users decide what stays
Let’s say you have a “great” idea for a new feature in your product. You somehow need to decide if that idea is actually good for your business/product or not. Once you implement a feature, you need to take a step back to see whether your users like that new feature or not. You need to measure how your users’ behavior changes with that new feature.
Using A/B testing to validate your new feature makes sure that users have the power to decide whether that feature stays or not.
Easy to measure the business impact
Ideally, you want to make sure that your new feature isn’t impacting your business/product in a negative way. You may be interested in measuring the impact of your new feature on the conversion rate or the profit margin of your product. Or, in case of a social network, you may want to measure the impact of your new feature on the engagement level of your users.
A/B testing helps you in quantifying the impact of the new feature on your business/product in terms of various metrics, for example — conversion rate, signup ratio, etc.
Not dependent on individual opinions
When you perform A/B testing for your new feature, you eliminate the effect of your or your boss’ opinions or assumptions. People have different opinions and they can be wrong a lot of times.
A/B testing allows you to rely on real user signal and eliminate the effect of individual opinions or assumptions.
Code deployments are easier
When you roll out a new feature, it may introduce some hidden errors/warnings. However, if you wrap your changes in an experiment showing your new changes to only 50% of your customers, you get two advantages —
- Your bugs/errors get less exposure (only 50%).
- You can easily rollback those changes by switching the experiment off.
Hence, A/B testing enables you to deploy your code more reliably and provides the ability to rollback fast in case of errors.
Learn from your customers
A/B testing is a really powerful way to learn about your product and your customers.
“The key to learning is feedback. It is nearly impossible to learn anything without it.”— Stephen D. Levitt
So far, we have covered what A/B testing is and why it’s important for you to use it to make better decisions about your feature launches. In Part 2 of this blog post, we will cover how to run A/B testing and make sense of the results. Don’t forget to subscribe to get updates once Part 2 is published.
