MARKETING AND OPTIMIZATION

Personalization and Testing. How They Are Different and When to Use Them

And how I got blinded by these cool terms

Manuel Panizo Vanbossel

Published in

The Startup

8 min readApr 17, 2020

We have been stuck with the same conversation for two hours. We are building an online process to apply to a financial product and we want to empower our customers to tailor terms for themselves. Our goal is to give them flexibility, freedom and relevance through personalization. But the problem is we cannot find the right way to do it.

We go one way and the number of options is so extensive that we risk overwhelming our customers and cornering them into indecision. We try another way and we make the process of customization too complex. We try to make it easy and straightforward, but then we sacrifice what we can offer.

We keep going in circles until someone says. “Let’s test it. That way we will know for sure”.

The mere mention of testing is the deus ex machina that outweighs any other argument. But were we actually talking about personalization? And could we expect testing to solve our problem?

The difference between testing and personalization

Testing and personalization are often used interchangeably, but they are very different concepts. While both techniques attempt to maximize conversion on online channels by offering relevant content, they represent opposite approaches to the same challenge.

Testing is about observing a few users and using their behavior to predict how most users will behave. In that sense, testing is an exercise of generalization. Personalization, on the other hand, is about individualization. The aim is to create unique experiences for every user.

Testing

In testing, we attempt to find the single best performing version of a content piece for our general audience. To get there, we publish different versions of the same content. Any user is randomly shown only one version and as more users arrive and interact with our content, we monitor the performance of each version until we have collected enough data to decide what version to keep. All other versions are discarded.

To understand the process, let’s start with its simplest form: A/B testing.

A/B testing or split testing

Imagine that you have just published a book. You also run your own blog, where you have amassed a certain following. You may want to use your blog to advertise your new book, so you are working on a banner that would always be visible at the top of your website.

You have worked on two versions of the banner, but you are not sure which one would get more readers to buy the book. So you decide to test version A against version B.

Infographic by Manuel Panizo Vanbossel using icons by Kirill Kazachek

For a short time, to each user that comes to your website, you randomly show only one version of your banner. Over the first days, you may notice that one version gets you more buyers than the other. So you take down the low-performing version and keep the one that is generating more book sales.

This type of test is known as an A/B test or a split test and some tools can help articulate this. They will help split traffic between versions, keep track of the results produced by each version and determine when you have enough data to make a decision. The most popular tools are, from more accessible to more costly, Google Optimize, Optimizely and Adobe Target.

When is A/B testing useful?

There are two types of cases where A/B testing is effective:

You are comparing two similar versions that only differ in one element. For example, you wonder whether a CTA button would be more effective on a different color, or you want to try a different text, but A/B testing is not the best option to test color and text at the same time.
You are comparing two entirely different concepts and you are more interested in validating ideas than specific elements. For example, one version of your ad uses pictures of real people, while the other uses illustrations.

A/B/n testing

The concept of A/B/n testing is self-explanatory. It is the same as A/B testing, except you are comparing more than two versions.

When is A/B/n testing useful?

A/B/n testing is useful when you are comparing several versions of the same element. Rather than comparing two colors in a button, you may compare several. Or instead of comparing the effectiveness of two calls to action, you can compare multiple text versions under one same test.

Similarly to A/B testing, A/B/n testing is not recommended to test more than one element at the same time.

Multivariate testing

The idea behind multivariate testing is similar to A/B testing, but multivariate testing is much more powerful because it allows you to find the best possible combination given a set of elements (heading, picture, copywriting, call to action…), each of them with several versions. In other words, it allows you to test multiple elements under the same test.

An additional limitation of A/B testing is that you can only compare the whole package. If you test two changes on a landing page, for example, the title and the color of your call-to-action button, and you see an improvement in your conversion rates, you will not be able to tell if the improvement is due to one of the changes or partially attributable to both. What is more important, you will not be able to tell if changing only one of the two elements would have yielded better results.

Of course, testing elements one by one requires a lot of time. That would be the only problem of A/B testing if the whole equaled to the sum of the parts, but this is not true. The effectiveness of an individual element on a webpage depends on how it relates to other elements. In other words, a big orange call-to-action button may work with a given set of elements and perform poorly with others, even when the target audience is the same. The infographic below shows how multivariate testing works.

When is multivariate testing useful?

Multivariate testing is useful in two cases:

You want to test multiple elements at once
You want to test, not just for individual elements, but to find the best performing combination of elements.

Multivariate testing has more to do with permutation than with personalization

The fact that multivariate tests deal with so many components to create a large number of versions reminds many of personalization, but the concept of multivariate testing has more to do with permutation than with personalization.

Personalization

Personalization consists of creating a customized experience for each person by leveraging multiples sources of data that allows you to predict what each person will respond to.

Multivariate testing is sometimes confused with personalization because both attempt to find the right combination of elements to maximize performance. The difference is that personalization leverages all kinds of information to give each user a tailor-made experience. So while multivariate testing aims at finding the best possible solution for the masses, personalization aims at building the best possible solution to each user.

Types of data used in personalization:

The types of data used in personalization can be viewed in three dimensions.

Information explicitly provided by the users versus information derived from their behavior on digital platforms
Information obtained directly from the use versus information acquired from third-parties
Actual information we know of a user versus information we infer from similar users (lookalikes)

When is personalization useful?

Personalization can be useful in almost every environment. Each person is more likely to respond to certain triggers and the key to personalization is activating the right triggers.

This can be very simple. For example, most people pay attention when they hear or read their name, so you can personalize your email communications by using the addressee’s name. Dear John… Simple.

Or it can get incredibly complicated. An example of a very successful personalized feature is Spotify’s Discover Weekly playlists. Netflix is often used as an example of personalization, but my favorite, though, is Roon’s Radio. Roon is a music player and their Radio feature keeps playing music after queued songs have finished playing. To make sure they play songs that you will enjoy, they take into account both your listening history and also the history of other Roon subscribers with similar tastes in music. Additionally, you can use thumbs up and thumbs down to train their engine, so it can improve over time.

Conclusion

Looking back at the discussion I was having with my colleagues, I realize we may have gotten all too excited about the terms “personalization” and “A/B testing”.

In our efforts to personalize our product packages, we wanted to leverage information we knew of the customer individually, but it was really an issue of user experience that we were struggling with. And our resorting to A/B testing was, at the very least, misguided. Our product had not even launched yet, we did not have anything that we could optimize based on actual user behavior. We could do user research and gather feedback on different versions of our application, but we certainly could no do an A/B test.

To wrap up, let’s highlight what testing and personalization have in common, as well as why they are different.

In common:

Testing and personalization are both strategies for optimization
Both are based on actual, un-biased, user behavior, as opposed to using surveys or other kinds of market research

Different:

While testing uses generalization to choose one version that will yield the best overall results, personalization is about individualization, it attempts to create a unique version for each user.