CXL Institute CRO Minidegree Review Part 9

7 min readMay 24, 2020

A/B Testing Mastery, Advanced Experimentation Analysis, and Optimizing for B2B

This is part 9/12 in my series reviewing the CXL Institute CRO Minidegree. I will be posting a new part every week!

CXL Institute offers some of the best online courses and industry-recognized certifications for those seeking to learn new technical marketing skills and tools highly useful to growth professionals, product managers, UX/UI experts, and any other marketing profile looking to become more customer-centric.

I was given an amazing opportunity to access and review one of their online course tracks, the Conversion Rate Optimization (CRO) Minidegree. For the next few weeks, I’ll be discussing the content of the course as well as what I think of it as I go through it. Here is part nine!

A/B Testing Mastery

Ton Wesseling of Online Dialogue introduces this next course. A/B testing is perhaps one of the most powerful tools you can use for decision-making. However, as seen in previous parts (most notably the content on A/B testing statistics), there is a lot of room for mistakes.

Around 2010, A/B testing was made available to a larger audience of professionals due to the simplicity introduced by new tools such as VWO and Optimizely. You didn’t need to be overly technical to use these tools anymore, but that also meant that a far greater number of people would make mistakes due to underestimating the statistics behind experiments.

You can use testing to deploy something new, to find out if some elements on your website are irrelevant, or in order to research new potential changes that can raise your KPIs. It is a way to add effectiveness to your decision-making, so it should not be taken lightly. Similar to how new medicine or medical innovation is approved in the health industry through randomized controlled trials, it is at the top of how truth can be discerned and how you can manage risk in your company.

Starting off, do you have enough data to run A/B tests? Wesseling suggests that 1000 conversions (or whichever goal you are trying to optimize) in a month is the absolute minimum in order to start A/B testing, otherwise it will be difficult to find true winners. If you don’t have the minimum, this isn’t necessarily a stopping point, it will just involve more risk. However, you should focus on testing campaigns, emails or other traffic sources in order to drive up the conversions you need first.

The more conversions you have, the more tests you can run. For instance, at 10,000 conversions, you can start running as many as 4 tests per week. At the minimum of 1000 conversions, you will have to beat the control by 15%. This rate becomes lower the more conversions you have. At 10,000, the minimum impact needed is 5%. Wesseling recommends using this tool to calculate minimums for your case.

Statistical power and significance are also important concepts that you need to take into account. Significance can be adjusted as your testing program matures, but 90–95% is recommended when starting out. Similarly, power should be kept at 80% or higher, otherwise, you run the risk of running into false negatives.

Next, which KPIs should you choose? This will depend on your case, but these are the most typical options:

Clicks: Should be used as a very basic measure of interaction and behavior.
Behavior: Can be used as a measure when your conversion numbers are too low.
Transactions: This is the foundation of business growth, and a key metric you should always keep track of.
Revenue per user: On top of simply tracking transactions (which doesn’t say much about how much money you’re making), you should also measure average revenue per user.
Potential revenue per user: As the final layer, this is referred to as the “golden goal” metric. It is quite difficult to calculate, but it is a great measure for conversion optimization.

Alongside these common choices, you should always have a “North Star” metric (also known as the One Metric that Matters), which is a common metric for your entire department/company.

Next, Wesseling discusses the 6V model, a broad research method that can provide insights for A/B testing. 6V stands for:

View: What problems do we see in web analytics and behavioral data?
Voice: What insights can we find from voice-of-customer data?
Versus: What does competitor analysis tell us, and what market best practices can be found? What tests are they running?
Validated: What insights can we draw from previous experiments?
Verified: Which scientific research, insights, and models are available for this case?
Value: Which company values are important/relevant and which focus drives the most impact?

Wesseling also suggests that prioritizing hypotheses based on this research should come down to the typical ICE/PIE models, with an added metric: ease of finding a significant outcome or a minimum detectable effect. The potential of each hypothesis can also be based on how much data you have for each case from the 6V model. This can be used to rank hypotheses.

Here are a few more tips:

If you’re starting out, only run A/B experiments, not A/B/C/n. Use the entire population of your website early on.
Calculate the minimum detectable effect (MDE) for each specific situation/page.
In Google Optimize, you should ideally start by creating 3 variations: original, default, and challenger. By doing this and setting a custom percentage split to 50/50 for the last two, you can avoid issues such as bot traffic being allocated to tests.

Advanced Experimentation Analysis

Chad Sanderson, a program manager from Microsoft’s Experimentation Platform discusses this course on how to analyze your experimentation data without simply relying on your A/B testing platform. Instead, Sanderson argues that coding, specifically R Studio, can be a powerful tool to gain more insights than the average optimization specialist has access to.

Unfortunately, since this course is mostly taught in a code editor, it is difficult to cover it in this review. I can certainly recommend looking into coding and R for analysis and data visualization, and this course in particular does a good job of taking you from the basics all the way to more complex (but easy to understand) functions that you can use to analyze your data quickly.

Optimizing for B2B

Bill Leake, a former McKinsey consultant and veteran of Dell Computers, gives us advice about how B2B is different in this next short course. Here are his most important points:

The conversion experience for a B2B scenario should be seen as a process. The conversion cycle is usually more convoluted.
Your attribution model should take into account the longer conversion path.
Search ads, display ads, retargeting, and other forms of early digital channels are typically hugely undervalued in attribution because the sale happens much later.
Establish a balance between quantity, quality, and cost.
You can add Myers-Briggs type attributes, and professional role assessments to personas in order to better understand who you’re communicating to.
Even in B2B, visitors have low attention spans, so focus on “nugget-sized” content that can deliver information easily, such as videos.

Key takeaways:

As established in previous parts, A/B testing has its pitfalls. Make sure you calculate and meet your minimums. The 6V model can be used to find conversion optimization opportunities, and to rank hypotheses.
R is a great coding language for data analysis and visualization. Mastering this tool will allow you to analyze experiments far better than most CRO specialists.
B2B is a different context that involves a much longer and much more convoluted sales cycle. Adjust your optimization process, personas, and attribution models accordingly.

Thoughts:

The content of the A/B Mastery course contained a lot of the theory from previous statistics courses but in a much more understandable manner. It also provides a lot of insight into which minimums you need to get started with A/B testing in your company. The 6V model is also interesting since it covers slightly different areas when compared to the ResearchXL model introduced in earlier courses.

Coding will always be a powerful tool for any kind of growth expert. From HTML and CSS to R and Python, there are many useful things you can do in marketing if you learn to code. The R code introduced in the Advanced Experimentation Analysis is also not as complicated as I thought when I started the course, and I will definitely look into this area more in the future.

In the next part, I’ll be covering “Customer Value Optimization”, “Creating a Segmentation Strategy”, and “Digital Psychology and Behavioral Design”!