Segment analytics with Firebase AB testing

Sometimes we want to know not just if but what our segments are up to

Firebase AB testing is an excellent tool to easily get started with AB testing.

It allows you to combine the Firebase remote configuration with Google Analytics to track the performance of your test.

I’ve written about how to do this in practice before so here I will focus on a recent issue I had to solve. How to segment your analytics in more detail using BigQuery.

What the Firebase dashboard is missing

I love how the Firebase AB test dashboard gives you a powerful test comparison without becoming very cluttered.

It breaks down the variants into clear groups and automatically measures the events you have selected.

Example of comparison of view_item events as success criteria

As seen below here, you can even easily measure changes in purchases or ad revenue using the native Google Analytics events.

Example of ad revenue as success criteria of AB test

You can even track counts for different events as additional metrics when setting up your A/B tests, with the caveat that these must be conversion events.

Measuring success by the number of events (conversion events only)

What the A/B testing console doesn’t give you (yet), is the ability to compare and count any events in general beyond conversion events, or perform different path or funnel analyses for different segments of users in your A/B tests.

When do you want to track the number of events

The use case; I wanted to measure the engagement of our new data selection algorithm.

Using the built-in dashboard, I could compare how many users triggered an event compared with the older version but I was not able to compare if the new version generated more events overall than before.

I wanted to see if I got more item views overall than before, not only if more users generated any events at all.

How to segment users

I have yet to find any documentation on how to do this officially. Instead, digging into the events I found that Firebase adds a user property named with the AB test ID appended and a value for which variant the user belongs to.

Using this I was able to segment the users in BigQuery by selecting the events together with the variant the users belonged to.

This is how the query looks like:

Note that although I will not cover it in this article, this same query to segment users by the experiment variant they were exposed to can be adapted in this post on building your own funnels in BigQuery as well.

How to get the Firebase AB test group

Let me explain the query above step by step.

If you inspect the rows in BigQuery you will find that each event has an array of user properties logged with it.

Example output of Google Analytics events in BigQuery showing user properties

The user properties for the AB tests are pretty easily identified because of the naming. If you look at the URL of the AB test in Firebase you will also find that it contains the ID of the test logged as the user property.

Firebase URL shows the ID of a Firebase AB test

To get the user property, since it is an array of properties, we need to use the UNNEST in BigQuery which joins the row with each entry in the array.

FROM `<project_name>.analytics_<id>.events_*`,
UNNEST(user_properties) as user

I then filter out the specific AB test I’m looking for. In this example, I only use one specific test but if you are doing tests on two platforms (iOS & Android) you can use filter more than one test property.

and user.key = 'firebase_exp_0'

The different variants will then be logged as 0, 1 etc. To make this more readable in my final query, I map these using a case to show which group the results belong to.

case user.value.string_value     
when '0' then 'Baseline'
when '1' then 'Other version'
else 'Error'
end as experiment_value,

The Error here will actually never appear but I added it to make sure I didn’t hide any mistaken mapping. I have not tried this with more than two variants but I would expect the values to continue with 2, 3, 4 etc for each subsequent variant that is added to the test.

I group by the value to be able to count totals per variant

group by experiment_value

And for the output, I calculate both the unique number of users activated, the total event count and the ratio between these:

count(distinct(user_pseudo_id)) as users,    
count(*) as events,
(count(*) / count(distinct(user_pseudo_id))) as ratio

Other tips

For more analytics queries, check out my BigQuery cookbook. For visualization, I really recommend Metabase which I use with BigQuery to show data.

Let me know if the above helps or if by chance you have another way of doing it.

Thank you for reading,
David

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
David Dikman

David Dikman

Active reader, developer, manager, entrepreneur & sporadic writer here and at https://greycastle.se. Currently working at https://styler.link.