A/B Testing + Product Management
Product Managers develop hypotheses on an almost daily basis. It’s a critical part of the job.
They develop hypotheses about new features, changes to UX, even the specific copy that best conveys to a customer what the customer should do next.
Many times, however, especially at smaller companies or startups, these changes are simply “rolled out” or deployed to 100% of visitors/users/customers without a second thought.
However, this can result in several significant problems:
- You can never definitively measure the impact of the change.
Because many changes often happen at the same time, including changes outside of your product features and functionality (seasonality, user mix, etc.), you cannot determine the impact of one specific change without these other variables confounding your analysis.
- You may actually cause issues or changes in parts of the product that you’re not paying attention to or didn’t think would be impacted.
- You may make the experience worse for users.
- You may negatively impact your KPIs and your business.
Why Do We Test?
We don’t have (all) the answers.
I can’t tell you how many times the majority of my company (myself often included) thought a particular test would have one result and then we were completely surprised that it turned out the complete opposite! Usually, for an very intuitive reason that we missed in our initial discussions.
Even if you think you understand your user personas very well, it’s incredibly hard to predict exactly how they’ll react to a new design, a new pricing scheme, or a new onboarding UX.
We A/B test because it allows us to “control” for all of the other changes that are happening and measure the impact of only 1 specific change (or group of changes) for our testing group.
You also can learn much more from a test than just “win or lose.”
For example, with a simple pricing grid test, you can understand price sensitivity, how merchandising different features impacts plan selection, which design treatments that work better on mobile vs. desktop, and much much more.
Plus you can learn what makes you more money!
What Makes a Good Test (Or a Bad one?)
* Small, focused
* Simple to explain
* Paired with KPIs that you expect the test will move
* Stastically significant results in weeks
* BIG win or BIG lose
* Large, bloated
* Complex, lots of moving parts
* Not tracked or doesn’t affect any important KPIs
* Significant results in months or never
* No impact, more likely to have a neutral result than a win or a lose
Note: This is not an exhaustive list of what makes a good or a bad test, but includes many of the more important items to keep in mind as you build your A/B tests.
How Can You Create a Testing Culture?
Are you starting from scratch with testing at your company or startup? Here are a few tips to help you successfully launch your testing culture and get the organization excited about experimentation!
- Start by identifying a few very high impact tests
Very clearly define the hypotheses of these tests, how they will be tested, what metrics you expect to impact, and how you’ll determine which testing cohort won.
2. Determine what testing software to use → Find a company with a free trial
3. Present your plan for “testing” testing to an executive sponsor and get buy-in
Do any of your proposed high value tests require resources outside of you?
Can those resources be reduced or eliminated by designing the test differently without compromising the testing learnings?
Even if you are the only resource needed for these initial tests, make sure you have an executive sponsor who sees the opportunity as clearly as you do!
A/B testing has the tendency to shake up things a bit and having an executive partner to calm down other parts of the business when your tests impact their world is very valuable.
4. Create a Slack/Teams channel and invite the initial testing team
Leverage this channel to share and discuss the planned tests, how they will be measured, when and where they will run, and invite feedback.
Beyond your executive sponsor, try to engage at least 1 person from each major functional area (Marketing, Sales, Engineering, etc.) to join this channel and represent their group.
5. Make results as public as possible (win or lose!)
Results should be highly visible and paired with an analysis (by you!) of the “why” behind the “what.” Results are also a great chance to share additional follow-up hypotheses and gather additional experiment ideas from others in the business.
6. Host a testing hackathon!
Once you have a foundation, testing software, and an experimentation team, hackathons can be a great way to “launch” A/B testing to the rest of the organization and get people excited!
Need a few tips? Check out my post below.
10 Tips for Hosting a Successful A/B Testing Hackathon
There is no better way to find new testing ideas, generate excitement about testing, and push the boundaries of…
Should We Test Everything?
Some large tech companies, Facebook for example, test almost everything via smaller exposures before deploying more widely.
However at your company, especially if you are in the early days of testing, testing everything is not a great idea.
Channel your testing energy into high impact tests that match the characteristics of a “good” test that I shared before.
In addition, you almost certainly don’t have the volume of traffic, users, etc. to test everything. Facebook is clearly in a unique place in this regard with over 2B users to experiment with.
What Have You Learned About Experimentation as a PM? How Have You Built a Testing Culture?
Let me know in the comments or on Twitter at @amitch5903!
More from Alex Mitchell on Medium
How to Get Product Experience Without Being a Product Manager
It’s becoming harder and harder for people without Product experience to get junior Product roles. Here’s how you can…
The 10 YC Companies (S20) I’d Invest In
India, Pakistan, and SE Asia are startup economies to watch closely over the next few years. Author: Alex Mitchell
The 9 Different Types of Product Management Interview Questions and How to Answer Them
Preparing for a Product Management Interview? These are the insights you need to be successful. Author: Alex Mitchell
Good Product Manager / Great Product Manager
It’s been decades since Horowitz penned the instant classic: Good PM / Bad PM. It’s time for an update to great…
The Top 10 YouTube Channels for Product Managers
Here are the YouTube channels you need to subscribe to as an Aspiring or New Product Manager. Author: Alex Mitchell
How to Be a Great *Remote* Product Manager
It’s easier than ever today to go remote in tech. But it takes serious effort to be a Great Remote Product Manager…
Top 10 Podcasts for Product Managers in 2020
Subscribe to these podcasts and you’ll stay entertained all year long (and learn a thing or two!) Author: Alex Mitchell
Top 10 Audiobooks for Product Managers of All Levels in 2020
Audiobooks had a great 2019 and an amazing decade. Here are the top audiobooks for Product Managers in 2020. Author…
Disclosure: Some of the links in this post are affiliate links, meaning that at no additional cost to you, I earn a commission if you click them and purchase a product.
Originally published at https://themodernproductmanager.com on October 13, 2020.