Member-only story
Pitfalls in Product Experimentation
Common to-not-do-lists often overlooked in product experimentation causing poor and unreliable results
We all know product experimentation is important, and its benefits have largely been proven by organizations, enabling data-driven decisions on products, features, and processes. Google was testing 40 shades of blue on a link in the search results, and the right blue shade led to 200M in revenue. Booking.com has acknowledged the scaling and transformation of the organization were made possible by numerous testing and experiments conducted there.
However, product experiments, like any other statistical testing or experimentation, are prone to pitfalls. These are design and/or execution flaws, which might be hidden or unsuspected throughout the process. It is the duty of the data team — Product Data Analysts/Data Scientists —to guardrail experimentations execution and analysis to get reliable results. And hence it is important to understand the common pitfalls and how to treat them, as they might mislead the analysis results and conclusion.
If the experiment is not configured and analysed properly, it might lead to poor and unreliable results, defeating the initial purpose of the experiment — which is for testing out the treatments and gauging the impact.
