Why It Matters Where You Randomize Users in A/B Experiments

A/B Testing at Colorify

An user flow chart. At the beginning is is a user signup box, and then two paths branch from it. 1st path: user browses for pencils then user buys. 2nd path: user browses for crayons, then user buys. An arrow pointing to the user browses for crayons box, calling out that we’re A/B testing a buy-now button for crayon users only. Finally, there’s thinking-aloud clouds that asks the reader, “do i randomize where user signs up?” or “randomize where user browses for crayons?”
User flow diagram.
A funnel chart. The top step is user signs up. 20% of new users sign up for crayons, which leads to the next step, user browses for crayons. 10% of crayon users buy, which leads to the last step, user buys. We call out that we’re a/b testing a buy-now button at the “user browses for crayons” step. There are thinking-aloud clouds asking us, “do I randomize where user signs up?” or “do i randomize where user browses for crayons?”
Funnel chart.

🖍️ 🖍️ Scenario 1: All Our Conversions Come From Crayons

Table, first number refers to crayon users and second number refers to all users. Baseline conversion rate: 10%, 2%. New conversion rate: 11%, 2.2%. Relative change (MDE): +10%, +10%. Total sample size needed at 80% power: 29502, 161363. Days to reach sample size: 41, 45.

🖍 ✏ ️️Scenario 2: Half Of Our Conversions Come From Crayons And Half From Pencils

Table, first number refers to crayon users and second number refers to all users. Baseline conversion rate: 10%, 4%. New conversion rate: 11%, 4.2%. Relative change (MDE): +10%, +5%. Total sample size needed at 80% power: 29502, 308607. Days to reach sample size: 41, 86.

✏️ ✏️ Scenario 3: Most Of Our Conversions Come From Pencils

Table, first number refers to crayon users and second number refers to all users. Baseline conversion rate: 10%, 12%. New conversion rate: 11%, 12.2%. Relative change (MDE): +10%, +1.7%. Total sample size needed at 80% power: 29502, 834797. Days to reach sample size: 41, 234.

Conclusion

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store