Product management skills: A/B Testing
A product manager is a multifaceted professional who must be a natural problem, solver. To face day to day challenges, there is a wide variety of tools to which s/he can turn to. In this post, we’ll discuss one of the most versatile tools in a product manager’s utility belt: A/B testing.
What is A/B testing?
In simple terms, it is an experimental method that compares two variants — A and B — with the objective of determining which variants produces the best results. Also known as split testing, A/B tests consists of dividing an audience in two or more groups (depending on how many variants are being compared) and presenting each group with versions of the same impulse or content. The difference between the versions usually lies in one variable, which allows the tester to isolate the cause of the difference in results.
The image above shows an example of an A/B test ran by Amazon on the day that this post was written. The sole difference between the two versions of the homepage presented is the second image of the carousel.
What can be A/B tested?
Almost anything can be subjected to A/B testing. Websites, emails, ads, and even app interfaces are frequently tested. Within those, there are tons of different variables that can be manipulated to generate variants for the test. Some of the most common variables are:
- Headline
- Call to action text
- Call to action color
- Call to action location
- Images
- Subject lines (for emails)
- Ad copy
Before you decide what to test, it is important that you formulate a hypothesis of how these variables can affect your results. This will allow you to make a decision of which variable should be tested first.
How long does an A/B test last?
The duration of the test can vary depending on how many versions are being tested at the same time and the amount of traffic you are receiving. You want to allow the test to run for enough time so that each version is viewed by enough people, which will enable you to extract unskewed conclusions. However, you should also avoid running the test for too long, since those results could be influenced by too many outside impulses over which you have no control.
A/B testing can help a product manager to improve results as top of the funnel as click through rate or bounce rate, to lower funnel outcomes such as revenues. At Product School, we teach aspiring product managers how to access success metrics through A/B testing. I invite you to go to our website and check out the complete syllabus, to get a full view of what we cover as a part of our 8-week program.
Originally published on The Product Management Blog
