Sitecore A/B testing (pt 2): How to select your strongest content
By: Sitecore MVP Randy Woods
Determining the strength your Sitecore website content will help you to identify the best candidates for A/B testing. We provide you with a tested methodology and A/B testing duration spreadsheet. (See first post in this series).
The Sitecore Customer Experience Platform opens up a new world for online marketers — putting enormous power in their hands and making practical tactics that previously required significant IT resources to execute. One of the most promising of these tactics is A/B testing, but as I suggested in my previous posts, A/B testing is only useful on pages that have sufficient traffic for the tests to be statistically valid in a reasonable length of time.
Which pages should be targets for Sitecore content optimization?
When creating your Sitecore customer experience road map, your first step should be understanding where content optimization might be useful. Fortunately, visitor data help us to identify the pages where A/B testing is a realistic option and those where other tactics should be applied. You can gain this insight in four steps:
- Determine the average number of page views per day for each page on your site
- Enter that date in this spreadsheet
- Adjust the spreadsheet based on your current conversion rate and anticipated increase
- Review the results
Step 1: Find average page views per day
If you’re quite familiar with Sitecore, you’ll be able to extract this data directly from Sitecore, though we suspect that most making use of this spreadsheet will not have sufficient data within Sitecore quite yet. If that’s the case, and you’re running Google Analytics, navigate to Sitecore Content > All Pages and select show 1000 rows. Select a one month date range and then export this content in CSV format and open in Excel.
Step 2: Enter the data in this spreadsheet
You will need to download nonlinear’s Sitecore A/B Duration Spreadsheet by clicking on the image below.
Open this sheet and click on the Excel tab Insert Data from your Site. Replace the sample data with actual data on page views that you exported from Google Analytics or extracted from Sitecore.
Step 3: Enter data on your conversion rates
On the Excel tab Answer Questions on Conversion you need to answer two questions:
1. What is the current conversion rate on your site? We use a default figure of 2.5 percent — you can modify this if you believe your average site conversion is higher or lower.
2. How much do you believe a test might improve this conversion rate? We use a default figure of 20% and find it to be a useful starting point. As a general rule, the higher your current conversion rate, the less a test is likely to improve upon it. If you are uncertain, we recommend using the 20 percent figure. One note — this figure it relative. It means that you expect a 20% improvement in your current conversion rate (from 2.5% to 3%) not a leap from 2.5% to 20%.
Step 4: Review the results
Select the Excel tab Pages where AB Testing Works. The first column will contain the pages you pasted into the spreadsheet; the second column provides an indication of how long it would take for an A/B test on that page to reach statistical validity.
The sheet automatically colour codes GREEN any test that will take less than 30 days to complete and tests taking longer than 30 days RED. We have found 30 day duration to be the outside limit for a practical A/B testing program, but feel free to launch tests that last longer if you feel they are particularly promising. At least you will have a sense how long you will need to wait for results.
What comes next?
The data in the spreadsheet is necessarily a rough estimate, but we expect that for most organizations it will reveal that content testing activities should be focused on a relatively small proportion of the pages on their site. Given that reality, what tactics can a marketer employ the remaining, lower trafficked pages? I’ll address that question in the last post in this series.
