An introduction to A/B testing with Sitecore
By: Sitecore MVP Randy Woods
A collection of our Valtech thinking on the ins and outs of A/B testing and its relation to other elements of digital marketing including ecommerce and persona analysis.
The evolution of digital marketing has been marked by a few major milestones: the introduction of web based agencies, the creation of clickable ads (most notably by Google), the domination of various social networks and, with the rise in smartphone and tablets, a push towards a mobile presence for brands. We’d venture to add the reliance on analytics to this list, as organizations begin to demand actionable insights into their marketing efforts that must be measure and optimized.
What began as a battle for simple page views and clicks has matured into the creation of comprehensive strategic plans with the aim of increasing conversions, enhancing usability for visitors and optimizing elements to create a better overall experience. Finding your analytical sweet spot is no easy task- it requires serious skills and a strong background in the field, but doing so can improve your website conversions dramatically. In terms of business value for money, conversion and optimization testing should be a no brainer every time.
We’ve told you how important it is to test test and retest, and now you want to know how. There are a number of optimization techniques that can be used to improve your site, chief among this analytical renaissance has been one method for creating, testing, and reevaluating presented content: A/B testing.
A/B testing: An introduction
With its extravagant reputation across the board, from marketing to IT, the C-Suite to the front page, Big Data has gained an almost superstar like status. We’re here to tell you that, despite the hype, any organization can start to gain insight through data they are likely already collecting, without any special software or an additional hire in the form of a data scientist. Valtech SVP Allison Simpkins reveals her own hesitancy in embracing the idea and provides three simple ways to start small with Big Data:
Now that you have a good understanding of how you could potentially start using the data you’re collecting today to help guide your decisions tomorrow, let’s take a look at how A/B testing works and how it can help you optimize your web presence to encourage more conversions by testing content in real time. As noted here, understanding how an A/B test works in a bubble is one thing, but having the skills to choose what to A/B test and when is a little more of a challenge:
Constant vigilance! Potential A/B testing pitfalls
A/B testing can improve your web content by leaps and bounds, but with the good comes the bad and with A/B testing, the bad can be incredibly detrimental. As with any form of analytics, interpretation of your test results is crucial to gain any real value from the exercise. Known for being notoriously misleading, marketers must be cognizant of the structure of their test and their desired outcome before embarking on an A/B testing journey. Combine those two considerations with a strong understanding of your customer segments and you should be prepared to begin testing:
Educating yourself on the limitations of A/B testing is just as important to the process as the tests themselves. As we mentioned, they can be an incredibly effective addition to your optimization, but only once you have determined that you’ve checked off all the boxes on the “misleading A/B testing” checklist. Here are three more to be aware of:
Deep diving into A/B testing
If you search for best practices when it comes to A/B testing, you will likely find a overwhelming majority of marketers are encouraging others to employ the tactic on a landing page, an online presence specifically optimized for a particular campaign or offering. While this approach certainly has its merits, not least of which is that they are often better funded and are given more attention than other sections of your website, we hypothesize that by switching your A/B efforts to your corporate website, you’ll see greater benefits.
Harkening back to our word of caution about how A/B tests have the capacity to be misleading, we look at just how easily you can be led down the wrong path with a concrete example that results in inaccurate conclusions. Using the same example, we provide a methodology that can help you avoid such a fate by incorporating personas: classifications for your customers based on their motivations, interest, emotional attachments to the brand etc., and some simple statistics.
We’ve established that to avoid the common pitfalls of A/B testing you must incorporate visitor personas, but just how exactly do you create those personas? The process for taking what your know about your visitors and categorizing them based on their options is outlined in this post, again using a concrete example to better illustrate how it’s done. Now you have your personas, your A/B tests and your data, but what do you do with it? We discuss your options for further segmenting this data to provide some insight into how your personas behave using a product you are likely familiar with: Google Analytics.
As with any digital marketing technique, evaluation of your testing tool is imperative before making any conclusions based on the data; A/B tests are no exception. Determining the statistical significance of your tests is just as important as running the tests themselves and can be accomplished through a number of different methods. As avid Sitecore users here at Valtech, we tend to deploy and evaluate client A/B tests using the platform’s own functionality coupled with traditional one and two tailed tests to determine the t-value of each condition.
A/B testing in the real world
At this point you may be so deep down the A/B testing rabbit hole that you’re having trouble seeing the forest through the trees. Don’t worry, we are going to show you some how A/B testing can be used to improve a pretty common online presence: the ecommerce site. Whether you are looking to determine which products to post to your front page, solidify your product navigation, streamline your checkout process or increase conversions, we’ve got you covered:
- How to take your ecommerce site to the next level with A/B testing
- How to take your ecommerce site to the next level with A/B testing: The redux
The bottom line:
As you can tell if you have read to the end of this post, we are pretty passionate about the benefits of A/B testing. And in that vein, let us end with one of our most read posts from the blog in recent months:
Have we gotten you as excited about A/B testing as we are? Let us take a look at your site and we’ll give a optimization report that outlines what’s working, what’s not, and what we can do to fix it.
