Sitecore A/B testing (pt 3): Understanding the limits
By: Sitecore MVP Randy Woods
The final post in a three part series on Sitecore A/B testing provides insight into the limitations of running such tests and gives a methodology for overcoming them. (See the 1st and 2nd post in this series).
The Sitecore Customer Experience Platform grants marketers a whole range of new capabilities for deepening their relationship with their customers. Content optimization is one critical capability enabled by Sitecore, but it is not universally applicable:
- In the first post in this series, I argued that Sitecore-powered content optimization is critical but is not effective on pages with inadequate traffic volumes.
- The second post gives you the tools to identify those pages on your site where you should focus your Sitecore CEP content testing activities.
This, the last post in the series, answers the question — “What do I do with those pages that do not have enough traffic for content optimization to be valuable?”
The waterfall approach
Commonly, within any section of a site, or selection of database-driven content, there will be a small number of pages that attract a high percentage of all page views. It is these pages where A/B testingis likely to be most valuable. The waterfall approach sees you flowing the lessons you learn by testing on the most popular pages down to the many similar, but lower trafficked pages.
An example: A for-profit institute of higher education offers more than 390 different courses. The website presents each of these courses in an identical template with the only element varied being text content.
Pareto’s law applies: the top 10 courses offered by the college receive 50% of all page views, and it is on these 10 course pages where content testing is practical — they have sufficient volumes of traffic that A/B tests reach statistical validity within 30 days.
The waterfall approach makes the assumption that what works on these highly-trafficked pages will probably work on the lower trafficked pages as well. This may not be true, but it is a reasonable hypothesis. Once you’ve determined that to be true, your next actions should be as follows:
- Run A/B tests on the top 10 most popular courses
- Find out what works (perhaps an optimal call to action)
- Apply this optimal element to the other 380 course pages
- Watch the collective conversion rate for all pages in that group
This last bullet point is important: while there isn’t adequate traffic to test on these pages individually, you can reasonably evaluate them as a group. If you make a change to all 390 course pages and subsequently conversion rates improve, you can reasonably conclude that the change you made works even on lower-trafficked pages.
Look to personalization
In another common scenario, important content targets an important audience or persona, but this group visits infrequently. nonlinear was recently engaged in a project where this situation was faced, while working the Sick Kids Foundation in Toronto. Their most highly trafficked pages included pages on which individual donations might be made. These tended to be more relatively modest amounts; at least modest in contrast to those made by major donors and corporations.
Content targeted at these later groups did not receive sufficient traffic for A/B testing to be practical, but the visits that did take place were critical — these groups had the capacity to make single donations that dwarfed the sum total of individual donations.
Sitecore personalization and A/B testing
Sitecore supports implicit personalization — it allows marketers to target content to visitors based on their site behaviour. This tactic is a perfect fit for these infrequent, but critically important visitor groups. It allows the marketer to build rules that shortens the path for these visitors to find the information that may influence them. For example, a visitor who spends time in the corporate donation section of the site may find that the home page hero image, navigation and testimonials change to reflect the considerations that a corporate donor would have.
In the end
These are certainly not your only options. Sitecore provides marketers with a broad spectrum of tactics for more deeply engaging visitors online (and increasingly across online and offline channels). We would be very interested in hearing about your experiences in optimizing lower content pages.
