Why Is A/B Split Testing Crucial To Success? — Part 2

Published in

Inside Revenue

9 min readNov 30, 2018

The Intermediate Guide

In my previous guide, I took you through the basics of A/B Split Testing, as well as the various benefits that the process provides your brand. Whether you are looking at using A/B Split Testing to improve your website or landing page performance, optimize your sales outreach process, increase the effectiveness of your advertising efforts or more, you should know what the best brands have used this process for and how to help you get on the right path. If you’d like a refresher on the benefits that A/B Split Testing can afford you, read “Why Is A/B Split Testing Crucial To Success? — Part 1 (The Beginner’s Guide)” now.

In this article, I will take you through the basic process for implementing the testing process as well as some of the various use-cases for A/B Split Testing.

“Success is making those that believe in you look brilliant.” — Dharmesh Shah ( Founder/CTO @HubSpot) @dharmesh

How Can A/B Split Testing Help You Hit Your Company’s Revenue Targets?

In this article, I will cover…

#1 How To Implement A/B Split Testing

#2 A/B Testing & SEO

#3 Examples Of Using A/B Split Testing

Let’s dive in…

___________________________________________________________________

#1 How To Implement A/B Split Testing

For this example, I will use testing on a web page or landing page to illustrate my point. However, this process contains the basic principles o f the testing process that can be applied to nearly every effort your are making in your sales and marketing campaigns.

In an A/B test, you take a webpage or app screen and modify it to create a second version of the same page. This change can be as simple as a single headline or button, or be a complete redesign of the page. Then, half of your traffic is shown the original version of the page (known as the control) and half are shown the modified version of the page (the variation).

As visitors are served either the control or variation, their engagement with each experience is measured and collected in an analytics dashboard and analyzed through a statistical engine. You can then determine whether changing the experience had a positive, negative, or no effect on visitor behavior.

The Following Is An A/B Testing Framework You Can Use To Start Running Tests

1. Collect Data

Your analytics will often provide insight into where you can begin optimizing. It helps to begin with high traffic areas of your site or app, as that will allow you to gather data faster. Look for pages with low conversion rates or high drop-off rates that can be improved.

2. Identify Goals

Your conversion goals are the metrics that you are using to determine whether or not the variation is more successful than the original version. Goals can be anything from clicking a button or link to product purchases and e-mail signups.

3. Generate Hypothesis

Once you’ve identified a goal you can begin generating A/B testing ideas and hypotheses for why you think they will be better than the current version. Once you have a list of ideas, prioritize them in terms of expected impact and difficulty of implementation.

4. Create Variations

Using your A/B testing software (like Optimizely), make the desired changes to an element of your website or mobile app experience. This might be changing the color of a button, swapping the order of elements on the page, hiding navigation elements, or something entirely custom. Many leading A/B testing tools have a visual editor that will make these changes easy. Make sure to QA your experiment to make sure it works as expected.

5. Run The Experiment

Kick off your experiment and wait for visitors to participate! At this point, visitors to your site or app will be randomly assigned to either the control or variation of your experience. Their interaction with each experience is measured, counted, and compared to determine how each performs.

6. Analyze Your Results

Once your experiment is complete, it’s time to analyze the results. Your A/B testing software will present the data from the experiment and show you the difference between how the two versions of your page performed, and whether there is a statistically significant difference.

If your variation is a winner, congratulations! See if you can apply learnings from the experiment on other pages of your site and continue iterating on the experiment to improve your results. If your experiment generates a negative result or no result, don’t fret. Use the experiment as a learning experience and generate new hypothesis that you can test.

Whatever your experiment’s outcome, use your experience to inform future tests and continually iterate on optimizing your app or site’s experience.

See Hubspot’s blog of 3 Real-Life examples of A/B split testing to get you excited.

#2 A/B Testing & SEO

Google permits and encourages A/B testing and has stated that performing an A/B or multivariate test poses no inherent risk to your website’s search rank. However, it is possible to jeopardize your search rank by abusing an A/B testing tool for purposes such as cloaking.

Google Has Articulated Some Best Practices To Ensure That This Doesn’t Happen

No Cloaking

Cloaking is the practice of showing search engines different content than a typical visitor would see. Cloaking can result in your site being demoted or even removed from the search results. To prevent cloaking, do not abuse visitor segmentation to display different content to Googlebot based on user-agent or IP address.

Use Rel = “canonical”

If you run a split test with multiple URLs, you should use the rel=”canonical” attribute to point the variations back to the original version of the page. Doing so will help prevent Googlebot from getting confused by multiple versions of the same page.

Use 302 Redirects Instead Of 301s

If you run a test that redirect the original URL to a variation URL, use a 302 (temporary) redirect vs a 301 (permanent) redirect. This tells search engines such as Google that the redirect is temporary, and that they should keep the original URL indexed rather than the test URL.

Run Experiments Only As Long As Necessary

Running tests for longer than necessary, especially if you are serving one variation of your page to a large percentage of users, can be seen as an attempt to deceive search engines. Google recommends updating your site and removing all test variations your site as soon as a test concludes and avoid running tests unnecessarily long.

#3 Examples Of Using A/B Split Testing

As I’ve already mentioned, A/B Split Testing can be used for more than just improving your website results. Below, I will give you a brief run through of just some of the different types of use-cases for A/B Split Testing.

Web Pages

If your company is an SaaS provider for example, your metric may well be the number of sign-ups you receive. Considering different versions of your web sign-up page will help optimise the page and increase sign-ups. For example, you may have an idea that changing the colour of your call-to-action button from blue to red would make it stand out better and increase sign-ups. In this case, you would use the existing blue design as your control, version A, and the new design with the red button as version B, and equally divide your website visitors between the two designs for a given period of time. At the end of the test you see which one work the best and you use that one. You may then choose to test the red design against another colour to further test or check your results.

Remember when running such test you’ll need to make sure the sample size you are dealing with is statistically relevant. For example, if you normally get just two or three sign-ups per day, then ten click-throughs won’t produce a relevant result. The larger the sample size the greater the reliability of your test results. However, the result will also depend upon difference in performance. If you normally expect a 5% sign-up from your blue button, you will need to determine what change in volume will make the variation relevant. If you are able to test in thousands, a 5.6% may mean a significant increase in business but if you are only testing in tens then the result will not be reliable. Whilst testing low traffic will never achieve significant scientific results it will still provide a level of insight but you will need to repeat the test frequently to ensure you are getting the best conversion rate.

Newsletters And Emails

E-newsletters and mass email marketing face a great deal of competition in a crowded inbox so making your message stands out could be the deciding factor as to whether your email is opened or not. Testing which title has the greatest click-through rate before mailing to the rest of your database could mean a significant difference in your campaign success.

Google Ads Campaigns

Google Ads is the ultimate tool for A/B testing — it was made for it! You can create any number of advert variations and measure their success with Google Analytics. For example, if your business is CRM systems you might test the following advert titles:

CRM made simple
CRM for small business
CRM Free Trial

You might also test different landing pages on your website with the same advert to see how that impacts upon your results. You can then set up campaigns between Google Ads and Google Analytics to accurately record your click-throughs, sign-ups and sales by determining the page a customer needs to get to for the transaction to be qualified.

The campaign should run over a set period of time, say 7 days, with these advert variations delivered in equal numbers at the same time of day. You might also use a test to see which day of the week, or time of day works best for your target audience. With accurate results Google Analytics allows the savvy marketer to schedule an advert with the most powerful title, at exactly the right time of day on the right day of the week. The result is that the advertising spend can be targeted where it is most effective, and an improved ROI can be achieved.

Market Insights

A/B testing does not only provide you with quantitative data that there is just no argument against, but it also provides insight into customer behaviour that can be used across other areas of your marketing. If you know that a red call-to-action button is more effective than blue, you might use this on other web pages. And if you know that you get a better response to one title than another on a newsletter use that insight to change other text in your promotional materials.

Segmentation

It is likely that that different versions of your web pages or campaigns will appeal to different customer segmentations, for example, according to gender, age, geographic location or industry. If so, use this intelligence to further target your marketing matching customers to particular products or services.

A/B testing should not be considered a one off activity. People and trends change so run regular checks to test your results and for each new campaign or product. Keep testing on a regular basis and remember that not every test will work. Be prepared to start again if you don’t get a decisive response.

Make sure you plan a time period for your test and a minimum number of responses needed to make it meaningful — ending it too soon could mean inconclusive results and dragging it on longer could lead you to select a poor version.

And finally, don’t be tempted to let your instinct overrule a test result — sometimes the outcome can be surprising!

Examples Of Communications Where A/B Testing Would Prove Useful

E-Newsletters
Email marketing campaigns
Web sites
Apps
Internet advertising (banner/PPC/Google Ads)
Examples for the variables tested
Subject titles and subtitles
Product descriptions
Text (length, style)
Offers
Price
Images
Call to action button (text, colour, position)
Colour schemes
Forms (length, question)
Page layouts

___________________________________________________________________

Even if you adopt an A/B Split Testing process in your campaigns, there are always things that can go wrong if you aren’t careful. Many brands before you have tried and failed in one aspect of A/B Split Testing or another, so for this reason, I’ve scoured the internet to find you the most common mistakes that brands make in this field to help you ensure that you leverage these processes to achieve the best results possible, with fewer missteps.

Take a look at my article “Why Is A/B Split Testing Crucial To Success? — Part 3 (The Advanced Guide)”, where I will take you through the following: