A Product Manager’s Guide to Experimentation

Image for post
Image for post

A/B Testing + Product Management

Product Managers develop hypotheses on an almost daily basis. It’s a critical part of the job.

They develop hypotheses about new features, changes to UX, even the specific copy that best conveys to a customer what the customer should do next.

Many times, however, especially at smaller companies or startups, these changes are simply “rolled out” or deployed to 100% of visitors/users/customers without a second thought.

However, this can result in several significant problems:

  • You can never definitively measure the impact of the change.

Because many changes often happen at the same time, including changes outside of your product features and functionality (seasonality, user mix, etc.), you cannot determine the impact of one specific change without these other variables confounding your analysis.

  • You may actually cause issues or changes in parts of the product that you’re not paying attention to or didn’t think would be impacted.
  • You may make the experience worse for users.
  • You may negatively impact your KPIs and your business.

Why Do We Test?

We don’t have (all) the answers.

Image for post
Image for post

I can’t tell you how many times the majority of my company (myself often included) thought a particular test would have one result and then we were completely surprised that it turned out the complete opposite! Usually, for an very intuitive reason that we missed in our initial discussions.

Even if you think you understand your user personas very well, it’s incredibly hard to predict exactly how they’ll react to a new design, a new pricing scheme, or a new onboarding UX.

We A/B test because it allows us to “control” for all of the other changes that are happening and measure the impact of only 1 specific change (or group of changes) for our testing group.

You also can learn much more from a test than just “win or lose.”

For example, with a simple pricing grid test, you can understand price sensitivity, how merchandising different features impacts plan selection, which design treatments that work better on mobile vs. desktop, and much much more.

Plus you can learn what makes you more money!

What Makes a Good Test (Or a Bad one?)

Good

* Small, focused
* Simple to explain
* Paired with KPIs that you expect the test will move
* Stastically significant results in weeks
* BIG win or BIG lose

Bad

* Large, bloated
* Complex, lots of moving parts
* Not tracked or doesn’t affect any important KPIs
* Significant results in months or never
* No impact, more likely to have a neutral result than a win or a lose

Note: This is not an exhaustive list of what makes a good or a bad test, but includes many of the more important items to keep in mind as you build your A/B tests.

Thanks for reading so far! If you’re enjoying this post, I know you will enjoy the great content at The Modern Product Manager.

Image for post
Image for post
www.themodernproductmanager.com

If you prefer books, I think you’d also enjoy Disrupting Yourself (How to Succeed in the New Economy) or Building Digital Products (Handbook for Product Managers).

How Can You Create a Testing Culture?

Image for post
Image for post

Are you starting from scratch with testing at your company or startup? Here are a few tips to help you successfully launch your testing culture and get the organization excited about experimentation!

  1. Start by identifying a few very high impact tests

Very clearly define the hypotheses of these tests, how they will be tested, what metrics you expect to impact, and how you’ll determine which testing cohort won.

2. Determine what testing software to use → Find a company with a free trial

I’ve used both Optimizely and Split and would recommend either for helping you manage the cohort assignment, measurement, and other aspects of A/B test management.

3. Present your plan for “testing” testing to an executive sponsor and get buy-in

Do any of your proposed high value tests require resources outside of you?

Can those resources be reduced or eliminated by designing the test differently without compromising the testing learnings?

Even if you are the only resource needed for these initial tests, make sure you have an executive sponsor who sees the opportunity as clearly as you do!

A/B testing has the tendency to shake up things a bit and having an executive partner to calm down other parts of the business when your tests impact their world is very valuable.

4. Create a Slack/Teams channel and invite the initial testing team

Overcommunicate!

Leverage this channel to share and discuss the planned tests, how they will be measured, when and where they will run, and invite feedback.

Beyond your executive sponsor, try to engage at least 1 person from each major functional area (Marketing, Sales, Engineering, etc.) to join this channel and represent their group.

5. Make results as public as possible (win or lose!)

Results should be highly visible and paired with an analysis (by you!) of the “why” behind the “what.” Results are also a great chance to share additional follow-up hypotheses and gather additional experiment ideas from others in the business.

6. Host a testing hackathon!

Once you have a foundation, testing software, and an experimentation team, hackathons can be a great way to “launch” A/B testing to the rest of the organization and get people excited!

Need a few tips? Check out my post below.

Should We Test Everything?

Image for post
Image for post

Some large tech companies, Facebook for example, test almost everything via smaller exposures before deploying more widely.

However at your company, especially if you are in the early days of testing, testing everything is not a great idea.

Channel your testing energy into high impact tests that match the characteristics of a “good” test that I shared before.

In addition, you almost certainly don’t have the volume of traffic, users, etc. to test everything. Facebook is clearly in a unique place in this regard with over 2B users to experiment with.

What Have You Learned About Experimentation as a PM? How Have You Built a Testing Culture?

Let me know in the comments or on Twitter at @amitch5903!

More from Alex Mitchell on Medium

Disclosure: Some of the links in this post are affiliate links, meaning that at no additional cost to you, I earn a commission if you click them and purchase a product.

Originally published at https://themodernproductmanager.com on October 13, 2020.

Product @Kinsured | 5x Product Leader/Founder | Blog: medium.com/@Amitch5903 | Author: @producthandbook @disruptbook | Newsletter: bit.ly/MPMnewsletter

Sign up for The Modern Product Manager

By The Modern Product Manager

Each week, The Modern Product Manager newsletter focuses on a major happening in the world of Tech. Take a look.

By signing up, you will create a Medium account if you don’t already have one. Review our Privacy Policy for more information about our privacy practices.

Check your inbox
Medium sent you an email at to complete your subscription.

Learn from Alex Mitchell’s inside startup perspective and get smarter each and every week. Find more at https://themodernproductmanager.com/

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store