A/B Testing
What is A/B Testing:
A/B testing (bucket tests or split-run testing) is a randomized experiment with two variants, A and B. It includes application of statistical hypothesis testing or “two-sample hypothesis testing” as used in the field of statistics. A/B testing is a way to compare two version’s of a single variable, typically by testing a subject’s response to variant A against variant B, and determining which of the two variants is more effective.
As the name implies, two versions (A and B) are compared, which are identical except for one variation that might affect a user’s behavior. Version A might be the currently used version (control), while version B is modified in some respect (treatment).
Application for A/B Testing:
IE Recommendation Engines
What is Recommendation Engine
It’s a recommendation engine or system which works as an agent to connect the products and services of your website with the nature of the product a user is already interested in, based on what another user with similar behavior did, or on a popularity basis of the products of same genre, etc.
E-commerce shops use recommendation engines as a marketing strategy. Set it up properly, recommendation engines will help boost conversions, CTR, and revenues. However, e-commerce recommendation engines work best for websites which provide a large number of related products. Recommendations based on customer taste encourage customers to add related products to their order increasing the order’s value.
How to evaluate and measure the Recommendation Engine with A/B testing
The best for online user evaluation is to do A/B testing, which will help users to know whether recommendation engine will give them more clicks, sales, and increased usage or not. The data comes from A/B testing allows users to judge the result between ‘original’ and ‘variation’. The data shows clearly what effect a change can or cannot give to the website if shown live. A good quality A/B test and best Recommendation Engine can evaluate the business to a whole new level.
Use Case of A/B Testing
Case Study: Click Through Rate increases by Adding Pictures under the Bakery Food.
Test carried out by an online bakery, selling birthday cakes, cookies and bread.
Data:
Null Hypothesis: Adding pictures under the bakery food will not have an influence on the click-through rate.
Alternative Hypothesis: Adding pictures under the bakery food will lead to an increase in the click-through rate.
Analysis: (A/B Testing & Bayesian A/B Testing)
To reject the Null Hypothesis we need a p-value that is lower than the significance level, i.e. P < 0.05.
- A/B Testing:
The p-value is 0.016 which is less than 0.05, so we can reject the null hypothesis that the click-through rates are equal (Adding pictures under the bakery food will not have an influence on the click-through rate).
However, it cannot directly conclude that adding pictures under the bakery food will lead to an increase in the click-through rate. We can use Bayesian A/B Testing to further analyze the data.
Bayesian statistics in A/B testing is mainly based on past or prior knowledge of similar experiment and the present data. By calculating the posterior probability distribution for each variant, we can express the uncertainty about our beliefs through probability statements. The main advantage of the Bayesian approach is the ability to include historical data and to select a prior distribution.
R results show that P(A>B) is by 0.00279%. So, B (the click-through rate of adding pictures under the bakery) is much better than A (the click-through rate of without adding pictures under the bakery). And the posterior expected loss for choosing B over A is low (P = 0.1909818).
Reference:
