A Product Manager’s Guide to Experimentation

A/B Testing + Product Management

Product Managers develop hypotheses on an almost daily basis. It’s a critical part of the job.

They develop hypotheses about new features, changes to UX, even the specific copy that best conveys to a customer what the customer should do next.

Many times, however, especially at smaller companies or startups, these changes are simply “rolled out” or deployed to 100% of visitors/users/customers without a second thought.

However, this can result in several significant problems:

• You can never definitively measure the impact of the change.

Because many changes often happen at the same time, including changes outside of your product features and functionality (seasonality, user mix, etc.), you cannot determine the impact of one specific change without these other variables confounding your analysis.

• You may actually cause issues or changes in parts of the product that you’re not paying attention to or didn’t think would be impacted.
• You may make the experience worse for users.
• You may negatively impact your KPIs and your business.

Why Do We Test?