The Art of Conversion Research: My Honest Review of CXL Institute’s Growth Marketing Mini-degree — Week 2
The 2d week of my scholarship in CXL Growth Marketing Mini-degree has passed by, and I’m ready to share my learnings. This week has been all about research and experimentation, and I’ve learned a ton.
Without further ado, below, I’ll be sharing my top seven learnings:
- Statistical significance is absolutely not the most meaningful metric when deciding when to stop an A/B test.
One of the most common definitions of statistical significance is:
“In the context of A/B testing experiments, statistical significance is how likely it is that the difference between your experiment’s control version and test version isn’t due to error or random chance. For example, if you run a test with a 95% significance level, you can be 95% confident that the differences are real.”
Source: SurveyMonkey
If one is not well-versed in statistics, it’s easy to start treating any statistically significant results as true. After all, the very definition of statistical significance clearly states that ‘you can be confident that the difference is real’.
The acceptable level of statistical confidence is between 95% to 99%. So, it’s tempting for our brains to interpret it literally and think that we can be 99% sure that the result is real. After all, who would not like to be 99% confident that their decision is the right one, right?
The main problem with this is that you’re supposed to calculate statistical significance only when the required sample size for your A/B test has already been reached. Originally, statistical significance has been designed to be used in Agriculture where it’s not common to calculate the results prematurely. A typical example of an Agricultural A/B test is planting two groups of plants with different genetic treats and observing which of them will have a better yield. The results are only meant to be measured once — at the end of the harvesting season.
It just doesn’t make sense looking at the field of plants that have been recently planted and trying to evaluate what the final yield will look like. Sure, it seems like the field A shows slightly more sprouts than field B, but can you be confident that this means a better yield for the field A at the end of the season?
So, the main criteria that should decide when you stop an A/B test are as follows (in order of priority):
- Your test has been running for at least 3 weeks (4 weeks is even better).
- The sample size that you’ve pre-calculated has been reached. You have to calculate in advance what is the required sample size for your test and stick to it.
As the rule of thumb, each variation of the test should have approximately 250–400 conversion per variation in order to be valid.
Peep Laja shares several examples of how the results of the test can change 180 degrees after several weeks of running.
A lot of A/B testing tools will prompt you to end the test when the statistical significance has been reached. So, it’s important to remember to commit to the full length of your experiment and always keep the required sample size in mind.
At the same time, Laja also warns against testing longer than 4 weeks. By this time, many of the visitors' cookies will be erased and some visitors will see both variations of your test. So the results will be invalid as more than 20% of results might be miss-attributed to the wrong variation.
2. It’s possible to do user testing with a few testers.
I’ve always assumed that similar to A/B testing, a sample size for user testing should be big in order to be able to derive representative conclusions. And, I couldn’t be more wrong.
Both Peep Laja and Paul Boag say that usability testing can be done with very few testers involved (Boag suggests 6 testers, and Laja suggests 10–15). These numbers have firstly seemed surprisingly low to me, but they actually make sense.
Since the goal of usability testing is to identify main issues that visitors might have when using your website, it’s not required to test with many users. According to Laja, as much as 90% of all issues can be discovered by 10 people, and 15 users will discover 95% of issues.
Both experts also say that you don’t necessarily have to run user testing with representatives of your target audience if your audience is rather broad (e.g. think mass-market products). That said, ideally, it’s most effective to test with your actual target users.
What’s more, both Boag and Laja say that sometimes testing with one user (!) is better than no testing. This basically means that there are very little excuses to be made for not running user testing in a company of any size. Even a small organization can easily arrange to interview 10 people, and the gains from this are too valuable to ignore.
- Stopping testing after the first solution has failed is one of the most common mistakes.
Each test that you design is a solution that has been created to address a specific conversion problem on your website. As with any problem, there are multiple ways to address it. So just because the first ‘treatment’ has failed, it doesn’t mean that the problem you’ve identified is not valid. You can still succeed in testing other solutions. In fact, iterative testing is how you’ll gain some of the most useful learnings about your customers.
For instance, in one of the case studies shared by Laja, it took 6 rounds of tests until reaching the winning version that had a 79.3% higher conversion rate than the original. The first tests didn’t show positive results. Yet, with a series of follow up tests that were continuously refined based on the results of the previous tests, they were able to boost the results, step by step. If they’ve stopped after the first test, they wouldn’t have been able to reach this goal.
So, if you’ve identified a conversion problem (backed by research and data) that is really a blocker for your users, don’t stop after the first solution has failed. Keep testing, and you’ll be rewarded with higher conversions.
4. Technical analysis is one of the most critical steps in conversion research.
The first step in the Research CXL conversion framework designed by Laja is Technical Analysis. The goal is to identify technical issues on your website that might be influencing your key metrics, e.g. revenue per visitor, conversion rate, etc. Go to your analytics tool and compare performance across different browsers and device combinations.
In Google Analytics, you can start with Audience -> Technology -> Browser & OS report. It’s important to apply segments (Desktop and Tablet traffic, Mobile traffic) on top of these reports as the numbers averaged for each device category might not show any significant differences. Look for significant differences in device performance, e.g. 4x less conversion rate on a certain device.
I’ve done this exercise and I’ve found a 3.46% difference between the performance of Chrome vs Safari on Desktop in my company’s Analytics account. Given that Chrome visitors are almost 70% of the website visitors this is quite significant. It could mean that fixing potential issues with Chrome could bring 22% more conversions. Of course this requires investigating more, but it’s definitely not a bad clue for an issue that took less than 15 minutes to identify.
That’s a great exercise to identify low hanging fruit and get some results for your company or the client early. There are always gaps in desktop vs mobile experiences.
5. Measuring all interactions (events) with Google Tag Manager is crucial
In my experience, setting up additional tracking in Google Tag Manager for all website interactions is often time-consuming. It’s typical to track key events and goals, but it’s not often that all the events are being tracked. Laja makes a strong case for measuring each of these interactions as this allows us to understand which website behaviors correlate with higher conversions.
While a certain section can be not popular with all users, segmenting these interaction data can show that certain user segments find it more relevant than others. It even can be that viewing this section correlates with higher conversions, so it’s certainly worthwhile to have this data to be able to spot patterns.
6. Always import results of A/B testing in your Analytics tool
Stemming from the previous point, it’s also crucial to import the results of your test into your analytics tools as you can segment the results and see how each user group performed in the test. This can potentially bring much more insights into the behavior of each segment than looking at the aggregated results.
What’s more, this will allow you to double-check the validity of results produced by A/B testing tools, which are not always correct.
7. Even the most experienced optimizers can be wrong.
The first step in the Research XL framework mentioned earlier is Heuristic analysis.
According to Laja, this is ‘as close as we can get to using opinions in conversion optimization’. In a nutshell, heuristic analysis means reviewing your user’s journey on the website, page by page, and identifying potential areas for improvement (remember to check separately for each device as the website content typically varies).
There are 6 key parameters to evaluate on each page:
- Clarity — How understandable is what’s been offered and what’s the value?
- Relevancy — Does the web page relate to what the visitor expects to see? Do pre-click and post-click messages and visuals align?
- Incentives to take action — Is the offer clear? Is there some sort of urgency that is credible? What kind of motivators are used? Is product information sufficient? Is sales copy persuasive?
- Friction — What kind of issues could make users to not complete their desired action on the page? (e.g. difficult process, not enough information, poor readability and UX, bad error validation, fears about privacy & security, any uncertainties and doubts, unanswered questions)
- Distracting Elements — what are the possible distractions on the page? (e.g. flashing banners, animations, too much irrelevant information, etc.)
- Buying journey — Is the content on the page relevant to their stage in the buyer’s journey? Are they being asked to buy when they’re not yet familiar with the product? Or are they not being prompted to buy when they’re ready to?
As a result of heuristic analysis, you can identify so-called ‘areas of interest’. These are potential problem areas that should be validated by the data from customer surveys and analytics. Experienced optimizers can get very good at identifying potential issues as they’ve worked with lots of websites and can have developed a sense of what works (Laja compares this to art buyers who are able to develop a gut feel to spot the star artists when no one else sees it yet).
Yet, this doesn’t mean that they will be able to identify your website issues just by looking at it. Most importantly, it doesn’t mean that these issues will be critical for your users, and what works for one audience in a certain environment doesn’t work for another.
In one of the videos, Laja says that he’s able to identify issues just based on his opinion about 60% of the time. That’s a great reminder to steer clear of opinion-based optimization. The data is king, and we should not forget it.
Useful tips for staying on track with the program
The best productivity tip that helped me to keep up with the program so far is dedicating at least 30 minutes daily to learning. Some of the lessons are very information-packed, and watching all of them in one sitting is not productive. You’re likely to get tired, pay less attention, and miss out on some valuable information.
Another thing that I’m practicing is writing down my notes after watching each video based solely on my memory (this technique is from Learning How to Learn book by Barbara Oakley). According to Oakley, this is more effective than writing notes as you’re listening to the video as this way you’re actively training your memory.
And, plan the activities to complete each week and stick to it. Here is my plan for week 3:
- A/B Testing Mastery by Ton Wesseling
- Statistical Fundamentals for Testing by Georgi Georgiev
- Google Analytics for Beginners by Chris Mercer
- Intermediate for Google Analytics by Chris Mercer
