A/B Testing vs. Multivariate: What’s the Difference?
What is A/B Testing?
Also known as split testing, is the method of testing two different concepts for improving a user interface. In a split test, concepts are very different from each other by changing multiple elements of an interface; either color, layout, or both. You typically start a split test with two concepts; the control (the original design) and the test sample (the redesign).
Although you are testing both concepts on the same audience, you are not providing both concepts to the same person. In a split test, you need to decide before-hand your objective and how many people you want to test. Once you decide on objective and audience size, divide your audience in half. Present one concept to one half of the audience and the other concept to the other half of the audience. The concept that reached or was closer to reaching your objective is the winning concept.
What is Multivariate Testing?
Similar to a split test but instead of only testing two concepts, you are testing multiple (3+) concepts at a time; the control (original concept) and multiple test samples (test concept 1, test concept 2, etc.). Like a split test, you are equally dividing your audience into groups and showing one concept to one group. The concept that reached or was closer to reaching your objective is the winning concept.
Unlike a split test, your concepts should not be drastically different. The main purpose of a multivariate is to test smaller changes to a designs; a button color or positioning on a page, different headlines of an article, etc.
When to use either?
Use a split test when you are testing major changes to an interface. This could be a total redesign or a landing page, or any major changes to the user behavior. You want to test your concepts with an audience close to your user sample size for accurate results. However, you may start to see a clear winner early into the test depending on how many changes were made to the interface. There is one main goal of a split test — which concept did better?
A multivariate test should be used when refining, with very few changes, to an interface. As mentioned above, minimal changes could include the color of a button, or different headlines. Because there are so few changes to the interface, you want a large audience size (as many as you can) to gather accurate insights. There are two main goals to multivariate testing - which concept did better and what specifically made that concept better?
When not to use either.
Both tests are quantitative research methods, not qualitative. You are only testing the performance of user behavior on different concepts. You don’t want to use either test if you want to test user’s attitudes or sentiment value of your concepts.
Because both test are quantitative research methods, you do not want to use either when you only have access to a few users. You are splitting your audience into groups and the less people you have to test concepts, the less accurate your insights will be.
Specifically with multivariate testing, you do not want to test multiple changes to a design at one time. Again, the two main goals are, which concept did better and what specifically made that concept better? If you test multiple changes, you cannot provide a definite answer to what specific change caused the improvement.
Key TakeAways
- Both tests are quantitative research methods and should be use to test the user behavior on large-sized audiences.
- Be precise with your audience. You should not be testing different concepts with multiple persona types. For accurate insights, make sure your testing with only 1 type of user.
- Have an objective before you begin testing to know which concept reached, or will reach, your objective.
- User split testing when you are making several changes to a UI at once. Use multivariate testing when you are testing 1 or 2 small changes to a UI.
