Running an A/B Test with Google Analytics

Google’s free Content Experiments tool can help your startup run an A/B test with very little setup

A/B tests are an important tool in making difficult product decisions, and measuring the impact of changes you make to your website or application. However building an A/B testing tool from scratch can be a huge engineering investment- you have to make sure you are evenly exposing test groups, shielding test groups from one another, and it has to be easy to read results.

At kit.com we are always looking for ways to accomplish tasks with a small engineering and product team. Rather than build our own custom segmentation and result dashboard, we looked for existing products that could handle our use case out of the box.

Using A/B testing to make product decisions at Kit

Anyone can create a kit, which is a set of products that go together for an activity or experience. In order to create a kit, one must first sign up for an account. We had a hypothesis that more visitors would sign up if the prompt to create an account happened after they created their first kit. This isn’t the type of question that can be easily validated through user interviews, so we decided that an A/B test would be a good way to get more data.

We decided to run an A/B test to 50% of our logged out audience; users in this test group were not required to create an account before making their first kit. Instead, we prompted them to create an account after they added products to their kit. The team was excited about the test, and all of us were convinced ahead of time that the new experience was going to lead to more new accounts. However, after running the test for 3 weeks, we were surprised to learn we were wrong. As it turned out, the test group performed worse than our control group — both in terms of accounts created and also kits created.

Without A/B testing this, we could have accidentally shipped a feature we felt strongly about, but would have actually hurt our user experience and our growth numbers in a way that would have been difficult to correct.

Why use Google Content Experiments?

Google Analytics is one of the most commonly used frameworks on the internet and actually has a built-in A/B testing tool called “Google Content Experiments”. While the tool is actually marketed as a way to route traffic between pages with static content, you can easily access the segments in JavaScript to run your own tests.

It is really easy to set up and get started with, and it will be easy for both technical and non-technical members of your team to gain insights from your tests in the results dashboard.

How to set up an A/B Test

1. Create your A/B test in Google Analytics “Experiments” tab. You will find it under “Behavior”.
2. Pick a primary objective for your experiment. This can be one of Google Analytics’ standard metrics like Pageviews or Session Duration, or it can be a custom event of your own choosing. This is important both for measuring conversion on your test as well as for aligning as a team on what your expectations and objectives for the test will be.
3. Pick the percentage of traffic that you want to include in the test. Running the test on a smaller percentage of traffic will mitigate the risk for some bigger tests, but will also take longer to reach significance in your results. You can also open the advanced options to turn off Multi-armed bandit traffic routing which is on by default.
4. Configure your control and test variants- Google Content Experiments supports up to 10 variants per test.

Running your test

After all the steps above you will get an alphanumeric code for your experiment.

Use that code to load the following script in addition to your normal Google Analytics snippet

<script src=”//www.google-analytics.com/cx/api.js?experiment=YOUR_EXPERIMENT_ID"></script>

This will insert an object called “cxApi” on your javascript window object. Wait for it to load and in your application script call cxApi.chooseVariation(). Based on the returned value (0 for control, 1 for variant 1, etc), you can change your application behavior.

Reading Results

A sample experiment result dashboard in Google Analytics

Results should start showing up in your experiment dashboard about 6 hours after you begin your test.

One of the most significant benefits of using the Google Analytics framework is that they provide easy tools to visualize the results and to compare behavioral metrics of your test segments. It should be easy to gain insights by comparing the observed behaviors of your control and test groups.

You can even use the segments tool of Google Analytics to specifically see the effects of the test on isolated groups of your users, such as mobile users, users from specific continents or countries, returning users, or really any other dimension that Google Analytics has visibility into.

For dimensions of our data that are not visible in Google Analytics we also fire all of our custom events into a database that we can join with our transactional database using Metabase to gain greater insights into our data.

Gotchas

• Make sure to handle adblockers gracefully. If a user has an adblocker, the cxApi object will never load so make sure that you catch this error and serve the user an appropriate experience. If GA doesn’t load, we give our users the control experience and also make sure that their data is excluded from A/B test results
• If you are using Google Content Experiments for segmentation you can only run one frontend test per page. In our case where we are running a single page app this really means we can only run one test at a time, so it is very limiting and requires some planning.
• Google Content Experiments can only run for 3 months max, so this is not suitable for monitoring the long term effects of a test.
• I recommend providing an override mechanism for testing purposes so you can force yourself into one of the test or control experiences by providing a specific flag on the url. When this flag is provided, make sure you do not include the data in the A/B test results.
• I recommend only calling the cxApi.chooseVariation method right when the user is exposed to the behavior related to the test. For example, if your test only affects logged out users, make sure you’re not calling this method when a logged in viewer visits the site. This will make it much easier to see signal when reading results and reduce the noise.

Thanks!

We found this tool really easy to use and wanted to share our experience with others. If you also enjoy building and optimizing experiences that customers love, please check out kit.com/jobs to see if you might be a fit for our team!