3 Things to Report in an A/B Test Analysis
--
Learn the important things to include in your A/B test analysis
AB tests are quite common at Bukalapak, as we always utilize them to validate new ideas, with the ultimate goal of providing the greatest possible experience for our users. As a result, doing AB test analysis is one of the routines of a Bukalapak data scientist.
In this post, I’ll discuss three critical components of every AB test analysis. By providing these three inside the analysis report, we may better understand what our experiment is truly telling us, as well as the context of the experiment. Enjoy!
Working Example
Throughout the blog, we will use the sample case study as provided in my previous article below.
For convenience, let me (re-)expose the experiment details as follows.
Suppose a digital company wants to improve the redemption rate of its promo vouchers by revamping their current MyVoucher page design. So, we have the following two competing designs:
- Control: the existing design
- Variant: the revamped design
They roll the experiment by serving each user with one of the two designs randomly and record their action accordingly — whether or not they redeem the voucher.
Suppose we have the following result.
As a data scientist in the respective squad, you are tasked to analyze the experiment. But then, of course, a natural question is, what to include in the analysis report?
Result’s Significance Status
Perhaps the most trivia one to be included is the significance status of the experiment, i.e., does the variant perform significantly better/worst than the control? It is the must-have of an AB test analysis.
