What is A/B Testing? Its Benefits and Steps to do Test

What is A/B Testing?

A/B Tests are controlled experiments which are used in industry to make data driven decisions when any new feature is rolled out. A/B test in its simplest form can be explained as an experiment between two features, products or any other thing and two versions are Control A and Treatment B. Control group is the group with existing features and Treatment group is the upgraded one with new features. In this test we don’t have any focus group and we observe customers’ visits. It results unbiased data as we neither ask nor tell the customers that their visits are being tested.

Why A/B Testing?

When any company wants to upgrade their product but want to confirm if customers will like it or not they go for A/B Testing and below are the benefits of the same.

1. Increase revenue and conversions
2. Rapid iterations. It increases ability to improve your business
3. You can learn and work as you are just working on data and learn from the findings and make quick decisions.
4. Unbiased data as we are not picking any random data but actual customer visits.
5. This test is data driven and helps in decision making.

What to A/B Test?

1. Site pages, flow, elements (pricing, videos)
2. Business models
3. Backend Functionality and Algorithm
4. New products or services
5. Machine Learning Models

Steps to perform A/B Testing:

1. You require a testing tool. There are many free and paid tools available to perform test.
2. Define primary success metrics which you want to influence. It can be Increase revenue and conversions or any other thing.
3. Define idea/question to be tested.
4. Define variations and learning. You should know in advance about the learning related to every win or loose.
5. Create the variations
6. Measure and activate. Track with measurement and it will be helpful for us to make comparison.
7. Analyze the results. Check which one has the biggest impact.
8. Document learnings and Evangelize. After performing analysis always document your learning.
9. Ideation. Once you perform test and learn new things, update your idea with the learnings.
10. Repeat the process.

How to design A/B Test?

How long should we run the A/B Test? This is common question everyone carries in their head when they want to do A/B test. Sample size decides duration. To calculate sample size, we will need 3 components listed below.

1. Type II error or Power ( 1- Type II error)
2. Significant level
3. Minimum detectable effect

Sample size = 16σ2/δ2

Note : 2 is square here

Here δ is the difference between treatment and control. More the samples will be needed if σ is larger and less samples if δ is larger.

How to estimate Parameters?

σ is obtained from data. To estimate δ, we can use minimum detectable effect which is smallest difference matters in practice and usually decided by stockholders. It will help you to calculate sample size and once you have sample size you can decide duration accordingly.