Learn More from Your A/B Tests with Google Analytics
Looking to get more out of your experiments? Analyze your results with custom segments in Google Analytics
As a Product Manager focused on E-commerce, I’m constantly testing to refine our digital experiences. Segmenting test results is a technique I’ve grown to love but I haven’t seen it covered by any testing guides out there.
It’s a useful trick to expose hidden trends that sharpen your hypotheses and lead to better theories, even in experiments that you thought were inconclusive.
What We’ll Cover
At its core, A/B testing is a way to compare multiple versions of something to figure out which performs better. We’ll skip the basics, which you can read about here.
In this article, I’ll focus on what happens after your data rolls in.
We’ll go through:
- Why post-experiment analysis is overlooked
- How it works
- How it sharpens your theories
- How you can apply it to website
Note: In this post, I’ll reference Google Optimize and Google Analytics, as they’re free and available to organizations large & small. But the methods discussed here are broadly applicable to any major software in this space.
First, let’s discuss why this important phase of analysis gets ignored in the first place.
Why Post-Experiment Analysis is Overlooked
As experimenters, we’re encouraged to plan every detail beforehand. Prior to running a test, we need to:
- Develop our hypothesis
- Define our target audience
- Determine the data we’re capturing
- Identify the test duration
This emphasis on preparation is understandable — it helps us avoid common pitfalls in human bias when we conduct our scientific experiments.
But the upfront structure has a downside: it promotes a mindset that once our experiment is complete, our jobs are done. We interpret our results according to our predefined parameters and then we call it day.
This causes us to miss a big part of the story.
Your test may be finished, but your results are rich with hidden gems.
All it takes is 5 minutes of applying your segments to prove it.
So how does it work?
Let’s dig in using a real-world example. The following experiment was conducted by a direct-to-consumer brand offering wall art.
We’ll go through:
- Their initial hypothesis and experiment
- How we segmented the results in Google Analytics
- How these learnings improved their theories
The Hypothesis
The brand’s original hypothesis was inspired by the product’s uniquely visual nature.
The bolded sentence is the hypothesis that drove the test, while the final sentence defines the test itself.
Visitors who browse our products on a Collection page see a large number of options at once.
We hypothesize that users prefer to look for a visually appealing piece of art before they consider price.
By excluding pricing from the Collection page, we will minimize visual & cognitive distraction to visitors, resulting in a greater likelihood of conversion.
The Experiment
To test this hypothesis we used Google Optimize, an experimentation tool that you can use on your site for free. Within Optimize, we created an alternate experience to hide pricing across Collection pages:
We ran a split test with a primary goal of Revenue per Session — a useful KPI to account for changes in both Conversion Rate and Average Order Value.
After a couple weeks, here were the results… *drumroll*
No change in behavior! *deflated applause*
At face value, there was no meaningful takeaway here.
The problem with this type of analysis is that it’s flat. It treats each one of your visitors the same, melting them into one homogenous pool.
There are better ways to represent our customers in this dataset.
Segmenting Results in Google Analytics
We pulled our dataset into Google Analytics to compare behavior across custom segments that better fit our users’ varying experiences.
If you have both Google Optimize and Google Analytics and you’ve linked them, all it takes is the click of a button:
We then applied our custom segments to the experiment results. One of my favorite tricks is to segment the results according to where a user first entered the site — their Landing Page (I’ll explain why later).
Now things get interesting. Here’s how they compared:
That’s up to a 67% increase in revenue! Let’s break it down.
Visitors who began their session from the Homepage or a Collection page spent 64% more per session when viewing the alternate Collection page with prices hidden.
Conversely, visitors who began on a specific Product’s page and then navigated to a Collection spent 27% less per session after viewing the price-free Collection page.
Let’s step back to understand what happened:
The positive reaction of the visitors who began on the Homepage or a Collection Page was neutralized by the negative reaction of the segment who began on a Product Landing Page.
We can see that this significant but opposite behavior netted a result that at first seemed to have no difference. That’s the power of segmentation.
Note: This example is to demonstrate the benefit of using segments in analyzing experiments, not to convince you that this specific UX will apply broadly. Whether you think hiding prices on collections could work for your site or not, be sure to test it yourself. Every brand is different and your results will differ based on your product, audience and experience.
How it Helps: Sharpening Your Theory
A hypothesis starts as a block of marble; it’s a thick generalization that must be whittled away by more precise analysis. Running experiments and segmenting your results help to chip away at that block.
What’s left is a clearer picture of something that seems obvious in hindsight, a perfectly rational behavior fully illuminated.
It’s time to revisit our original theory and sharpen its definition. We can now express the hypothesis from our example more precisely:
We hypothesize that visitors who land on the Homepage or a Collection page are looking for a visually appealing piece of artwork before considering price.
We hypothesize that visitors who land on a Product page have likely already identified a visually appealing piece of artwork, and are now considering price.
By forming a clearer hypothesis, you’re more likely to draw a conclusive result on the next iteration on your test.
Using Your Own Custom Segments
We’ll help to identify segments to use. To actually create them in Google Analytics, check this guide.
Your library of custom segments will evolve as you get more reps in this type of analysis. Many are situational based on your hypothesis and test.
With that said, there are a few foundational segments that come in handy time & time again:
Segment by Landing Page
The Oxford Dictionary notes that a landing page is typically the website’s homepage. This is no longer true.
It would be nice to envision our website in a single dimension where every visitor moves from the top down, accomplishing the same goals in the same order.
In reality, the visitors of today are nuanced in their experience. Many move through the marketing funnel before ever stepping foot on your website. Customers view and shop your products using Facebook carousel Ads, Google rich snippets and more.
For the brand that ran the collection experiment, we can see the homepage is far from the most popular landing page:
With this in mind, we can update to our incredibly (un)scientific mental model:
A visitor’s Landing Page can give you an indication of their mindset.
Someone who lands directly on your homepage may be focused on finding a product that suits them, while someone who lands directly on a product page has signaled that they’ve already identified at least one product they’re interested in.
Segmenting by Landing Page cuts through the noise to help you understand where in the shopping process your visitors are.
Create segments for the following Landing Pages:
- The Homepage
- Collection Pages
- Product Pages
- Blog Pages (if your organization has one)
- Any other landing pages with significant traffic — Work with your marketing team or check your Landing Page report in Google Analytics to identify these.
Of course there are potentially many Product pages. You’ll want to group these pages together into one segment by using a URL-based rule.
The specific rule will depend on your site’s URL structure, but here’s an example:
The same applies to Collection & Blog pages as well.
So far, I’ve spent the most time discussing Landing Page segments because it’s been the most useful for me personally. With that said, there are a couple other dimensions that are worth mentioning.
Segment by Acquisition Channel
What are your primary channels of acquisition?
Work with your marketing team & do some digging in Google Analytics to identify your top sources within both Paid and Organic. These channels bring visitors with priorities and mindsets that can differ greatly.
Someone who clicked on an Instagram prospecting Ad, for example, may behave differently than someone who found a product from Google Shopping.
There will be some overlap here with your Landing Page segments, and you may find those are enough to draw out the behavioral differences you seek. You’ll have to judge this based on your results.
Segment by Device
The modern focus of digital design is to build for Mobile first, but Desktop still has a place at the table.
While it’s true that Mobile devices now lead in traffic, Desktop still sees higher conversion rates and a relatively higher percentage of revenue compared to traffic.
Historically, Device differences have been viewed through the lens of technology. We ask questions like
“How will this screen size negatively affect the experience?”
What we know today is that shoppers switch frequently between devices as they move through the marketing funnel. Segmenting by device isn’t just a tool to check for mobile screen limitations: it’s a way to gauge visitor frame of mind.
These are default segments in Google Analytics, but one quirk is that the segments are “Desktop & Tablet” and “Mobile & Tablet.” I’d consider creating a “Desktop Only” segment for better separation of behavior when comparing against Mobile/Tablet.
Segment Anything else!
There’s no clear answer on which segments will work best for your business. Your results will depend on your product, your marketing channels, your audience and your experience.
As you analyze your A/B tests, question your assumptions and tinker with new custom segments — you never know what will reveal a divergence in behavior.
Once you identify a useful segment, “star” it in Google Analytics to keep it handy for your next analysis.
Ready, Set, Go
Now that you’re properly set up in Google Analytics and your tools are in place, it’s off to the races.
Let’s review what you need to do:
- Revisit your organization’s past experiments, especially the ones that seemed inconclusive
- Open the experiment results in Google Analytics
- Apply your key segments and watch for any notable discrepancies in behavior
- Once you find something that didn’t stand out before, revisit the experiment’s original hypothesis: Can you sharpen anything based on what you’ve found?
This exercise isn’t meant to draw conclusions — it’s meant to uncover new leads worth pursuing.
In the end, it’ll inspire you and your team to have richer conversations about how visitors experience your site.
Find any segments or results worth sharing? Let me know!
