Experiment Best Practices

Now that you’re on your road to becoming a data-driven product manager, it’s time for a refresher course on A/B testing & Experiment Best Practices.

Define metrics of success before you start

One of the most common mistakes PMs make A/B testing is not defining what metrics they are trying to move before they start an experiment. You won’t know how to react when an experiment finishes if you don’t know what success looks like. Pick the metric or metrics that you are trying to impact with a particular experiment and think carefully about other metrics that you want to check and make sure you don’t impact. If you are willing to make a trade-off — for example you’re OK losing 1% of users if you get the other 99% to use your app 10% more — decide what that looks like ahead of time.

Think about who you’re targeting

Most experiments come back with lacking statistical significance. This could mean a feature isn’t impactful, but it can also be the result of too small a sample. I know it can be scary testing with a large group of users, but it’s better to test once on a bigger group of people, than rerunning and wasting cycles.

When you’re figuring out that sample size, think carefully about the users you want in the experiment. When I’m running an experiment on a brand new feature, I want to make sure I’m only looking at users in the latest app version where the feature will actually show up. Or if I’m testing a change on our Favorites page, I need to think first about how what percent of users actually look at that page. Think through all these variables before choosing a sample to make sure you get back meaningful results the first time.

Don’t run experiments you aren’t willing to launch

I mentioned earlier it can be scary to run experiments on your user, but if you’re only running experiments you’re willing to launch it’s a lot less scary. It may still be an MVP or proof of concept and might not be ready to launch as is, but it should be something you believe in enough to get out if you’re going to expose your users to a feature. Similarly, make sure you have organizational buy-in. Will your company commit to seeing a feature through if you test it and it performs well? As long as you’re testing with your users best interests in mind, this shouldn’t be too challenging. We use a/b testing to make our users’ experience with the app better, not worse. Test responsibly.

If at first you don’t succeed… iterate

That said, don’t just test once and abandon. This can be tempting especially when your organization is new to testing and you have so many things you want to test. But features often fail in their first iteration, only to perform with some clever tweaking. Take your failed feature back to your users and watch how they use it in person or via online user testing to see where your test feature fell flat. Then iterate and retest!

Don’t be afraid to abandon features

It’s 10X easier to ship a feature than remove one. One of the great things about experiments is that you can make sure a feature is meaningful before releasing it to all users. Don’t be afraid to abandon a feature you worked hard on if you can’t get it to perform in testing. It can feel easier to launch than to tell the team something they worked hard on isn’t going into production, but that is something everyone needs to get used to once you’re at a data-driven company. Besides, no one wants to be supporting a feature that only power users use 5 years later.

Keeping these best practices in mind will help you make the most of each experiment. For more on what to do when you get test results back, read Boone’s piece on Evaluating Experiments. Now go forth and start testing!