3 steps to A/B testing mastery

There are many mistakes in business that can cost you dearly. The most dangerous and quite often overlooked of them is the mistake of adjusting your website design and UX/UI without a thorough A/B testing. This is a crucial part of marketing for startups, yet still poses a significant potential damage for any company, especially in the areas of software testing, web development and marketing. Today we will explain what A/B testing is, why is it so important and will list 3 steps to a true A/B testing mastery.

Step 1: Learning what A/B testing is and how it works

A/B testing (also known as split-testing or bucket tests) is a form of an experiment (in terms of web development industry — held on a website) and aimed at comparing the efficiency of two variants of a variable (like the design of the contact form or a banner) in order to determine which of the two variants is more efficient. If the web development field, due to the fact that most of marketing software nowadays comes with A/B testing tools out-of-the-box, this type of testing can be effectively used non-stop on almost anything, providing the means for continuous improvement of website design and user experience.

The experiment itself has the following mechanism:

• All of the website visitors are split in two groups — the control and the alternative (these settings are stored in their browsers, so if the next time they visit the site from the same browser, they will again be the members of the same group).
• The control group is presented with the current design of the website, while the alternative group sees the currently tested, updated variant.
• The outcomes of the visit are evaluated (did the CTR of the banner increase? Did more visitors subscribe through an updated form, etc.). The variant with the best performance is then propagated to become the main and be shown to all visitors.

There are several important moments to keep in mind while planning the A/B testing:

• Statistical significance is the percentage depicting the chance that the testing results were accidental and forming a probability that the same result can be obtained again. Therefore, if we say variation B from the picture above is better with statistical significance of 95%, meaning we expect the same result to occur in 95% of all cases. Naturally, reaching a 99% statistical significance is the best variant.
• To ensure the correct results are achieved, all the external factors must be kept out of the equation, or should be exactly the same. Thus said, both control and alternative groups should receive the same treatment in terms of customer support, be allowed to spend the same amount of time on the website, have equal access to all website pages and functions, etc. This ensures the difference in the results can be considered solely a result of the evaluated factor in action.
• Credible A/B testing results cannot be obtained overnight, nor can they be collected in a couple of days. To severely reduce the influence of unaccounted and uncontrollable factors, the company must collect the A/B testing statistics for quite a prolonged period of time — several weeks or even a couple of months. While this might seem an exuberantly long period of time (while several types of testing take from several minutes to an hour to complete), the more time you invest into gathering A/B testing results, the more precise they would be.

There are, indeed, variants like multinomial testing, where several factors are tested at once, yet in order to provide trustworthy statistics, these tests should be performed by a skilled professional that will be able to filter out the unreliable data and concentrate on valid results instead. There are also certain kinds of data where simple A/B testing is useless, no matter the longevity of testing. These kinds include non-experimental situations, namely quasi-experiments and observations, which demand a much more complex testing logic to deliver credible results.

Step 2: Planning to do A/B testing for your startup

The practical implementation is actually quite simple and was described in general above.

However, the testing itself should be preceded by the research and planning. Think of the things like:

• What outcomes do you strive to achieve? (increase the sales funnel efficiency, grow the average check, increase the clickthrough rate, increase the numbers of subscribers, etc.)
• What current factors impair your website performance and what are the most pressing issues that stop you from reaching the desired outcomes? Obviously, testing the efficiency of the ads banner will lead to nothing, if the landing page is sub-par and the visitors leave the page at once. The most damaging issues should be addressed first, with design polishing left for later.
• What Key Performance Indicators (KPIs) will you monitor and what KPI levels can be considered a success, a sign the issues can be overcome? For example, growing the number of subscription to your news feed might seem great, yet if these subscribers do not turn into leads or unsubscribe after the very first email marketing campaign — all of the hard work of turning these visitors into subscribers was in vain in the first place.
• What actions will you take to overcome the issues once the testing process provides you with credible results? Knowledge itself is no power; it becomes power only once applied to get the most of the current situation. If the new landing page CTA button design results in slightly better conversion — might it indicate the need to revamp the page as a whole? Is the increase of your average order check feasible, considering the decreased number of sales overall? Answering questions like these in advance can help either steel your determination or show another round of planning is needed.

Just keep in mind adjusting the design is simple, yet changing the app workflow might be a complicated task demanding hiring software developers.

Step 3: Performing a continuous A/B testing of your product, website or service

Consider leveraging specialized marketing software, either a proprietary solution of your choice or a custom-built and tailored for your needs. Using said solution, split the incoming traffic into two groups — a control one and an alternative one. The control group will receive the current website design and app user experience, while the alternative group will be exposed to the updated design, workflow or features.

After sufficient quantity of historical data is gathered and the testing reaches the statistical significance, you can apply the updates for all of your customers and move on to the next round of testing, as A/B testing must be a continuous process of adjusting the available resources to achieve the maximum performance.

To sum it all up, split-testing is one of the most important tasks for any business, both a startup and an established enterprise, as only by adjusting to meet and exceed the customer’s expectations can a business ensure the success of their endeavor. We hope this guide will help you achieve the A/B testing mastery, and don’t hesitate to ask us if you still have any questions!