Why A/B tests are outdated
TL;DR
You can replace your A/B tests with A/B recommendations using classification machine learning algorithms. They will automatically segment your visitors and show the most relevant variation to every newcomer.
Today, A/B testing has become a silver bullet for data-driven decisions. Companies from your local grocery store to giants like Amazon and Google are building their business using iterative experiments and tests. If such big companies are heavily relying on A/B testing, then could these tools be wrong? The answer is — yes, but it depends.
A/B testing is great for making iterative changes. If a “sign up now” button works better than “sign up for free,” then you can choose the first version, without a doubt. The weak side of A/B testing is that it is totally impersonal. A/B testing does not care who your users are.
The weak side of A/B testing is that it is totally impersonal. A/B testing does not care who your users are.
Let’s take a look at an example. Here we’re testing old school screenshots versus brand new product video.
After collecting enough data, we can see that conversion with screenshots is significantly higher than with the video. Our assumption is that there is still a group of visitors who prefer the video. How could we identify them? The answer is in data. Let’s take a look at the chart of conversion rate versus visitor time.
As you can see, video works better in the evening than the rest of the time. Why is this happening? Maybe it’s a lot easier to watch videos at home rather than at work, and that’s probably because of the sound.
To get maximum from both variations we could show a landing page with video in the evening and screenshots at other times. Here what the result may look like:
Clearly, this simple improvement not only satisfies both groups of users but gains you measurable conversion improvement. And this is one special case which uses just one visitor characteristic. In fact, you can analyze a lot of data: devices, browsers, OS, countries, cities, languages, search keywords, sources, marketing campaigns, or even data provided by your users like age, sex, or interests. The more data you have, the more patterns you could find.
The more data you have, the more patterns you could find.
The only outstanding question is, How we will find dependencies between what users like and who they are? What about complex patterns based on a bunch of visitor characteristics? What if women from the UK on MacBooks, who are visiting in the evening, in most cases prefer cats over dogs? How the heck we can figure this out?
That’s where machine learning comes on stage.
That’s where machine learning comes on stage. Using machine learning algorithms, you could automatically find dependencies between user traits and variations — even if these dependencies are hidden behind a set of different parameters. The great thing is that with every registered or bounced visitor, the quality of your model will improve and recommendation accuracy will grow. The only barrier here is implementation. You will need to collect data, analyze it, and provide low latency to your servers worldwide. Anyway, this is still pretty solvable. We’re not going to give you a course of machine learning in this post, but if you want to learn more take a look on classification algorithms or Landy, the tool which we’re working on.
Conclusion
- A/B testing works well when you’re making small, iterative changes.
- A/B testing fails if variation A works perfectly for one group of visitors and variation B for another.
- To get the best possible result from your tests, you need to show the most relevant variations using your data.
We believe that with today’s amount of data, you don’t need to choose just one winner any more. Instead, you can find the most suitable variation for every new visitor and increase your click-through-rate automatically.
Welcome to the brave, new, data-driven world!
Hope you enjoyed the read. If you have any thoughts on this topic, feel free to share them in the comments.
Landy on web
