Sitecore A/B testing (pt 1): Using statistics to maximize tests and see results
By: Sitecore MVP Randy Woods
Content optimization using Sitecore A/B testing is a crucial tool in a marketer’s toolbox, but only if you know how to use it. This series will provide you with the knowledge to maximize tests and see results.
Many marketers know that the Sitecore Customer Experience Platform empowers them to make content testing a consistent step in the content creation process. This is good news as we here at nonlinear believe that A/B testing is the most underused online marketing tactics. The corollary is that for most marketers, content testing holds the greatest potential for improving marketing results, but there are limits. When selecting Sitecore marketing tactics to deploy you must understand where testing can be helpful — and where it can be dangerous.
You need reliable numbers
You cannot rely on content tests that are not statistically valid. This is the single most frequent mistake that we see when marketers are experimenting with A/B testing. Results obtained from a small sample size aren’t just uncertain; they are very often misleading. Consider the following figure that illustrates the percentage of times a flipped coin comes up heads over the course of 500 coin flips.
(Penn State University — online course in probability) <add lines showing the 40 and 100 flip marks>
- After about 40 flips, it appears that the coin is weighted towards tails — heads only appears to win 40% of the time
- After 100 flips, it is clear that the coin is weighted towards heads — it is “winning” about 57% of the time
- Of course, it eventually becomes clear that the coin is balanced and neither option wins, but if you drew conclusions too early, you would have over invested in one outcome.
So how long must you wait?
This, of course, raises the question of how many tests you need to run for the results of a test to be reliable. The answer depends upon:
- Your current conversion rate (the “control”)
- The amount of lift you achieve from the content being tested
- The level of certainty you need (we would love a 95% confidence level, but 85% is a pragmatic number for most marketing purposes)
Consider the following case: A marketer wants to test a new promotion on one of their website’s product pages. The page has modest traffic — about 500 page views per day — as might be typical of a mid-sized B2B manufacturing site. The current conversion rate is 3% and they are hoping that the new promotion will improve that by 10% (to 3.3%). For the test to reach statistical validity, they need about 14,500 page views. It would take 29 days to complete the test, which is just on the edge of a practical length of time.
In our experience, any test that can reach statistical validity in less than 30 days is worth attempting. A test that lasts more than 30 days, on the other hand, is not (too many external factors change that make the test less reliable — and marketers can’t afford that level of patience).
Implications for Sitecore users
The Sitecore Customer Experience Platform provides marketers with a large palette of marketing tactics. A/B testing is often the most promising, but it will not be useful on every page on your site. Many, perhaps most, pages on your site will not have enough traffic for A/B testing to be useful.
In my next posts, I describe how you can proactively identify pages where A/B testing promises results — and what you can do with lower trafficked pages most effectively.
