Google analytics sucks. A lot.
One week we hated Google Analytics. Here’s why.
Like the majority of startups, most product decisions at carwow are based on data. Also like many startups we use google analytics. A month ago, however, we decided to try using Mixpanel. Why?
Let’s take a look.
Building a car is a complex process. You need to select a trim, an engine, a colour, the options, and so on. carwow manages to get this process done in just four pages.
We know that every page in the process adds friction and increases dropouts. Consequently in November we designed an experiment to reduce it to three pages using backbone.js.
The result is, in our humble opinion, an amazing page. We were actually so proud of our work we strongly believed the variation would destroy the other one in term of conversion to sign up.
Here are some snapshot of the different versions:
Variation using backbone.js (the part in blue only appears if you click “view engines”)
Pretty sweet, don’t you think?
So we tested it, sending 50% of our traffic on each variation and guess what… after two good weeks of data, we saw a 15.5% decrease in conversion rate to signup, which made it one of the worst features we’ve ever released.
If the decrease was around one or 2%, it would have been almost acceptable but this was clearly too high. Obviously we reverted the change. And decided to do a full analysis of the data to understand what was happening. This is when GA enters in the story.
Before the test, we added events on almost all the clickable element of the page. Consequently, we had lot of data. Yet, we were not able to get anything from it. The conversion rate was down whatever the device used, browser, source, etc.
Something was wrong in the page.
Trying to get page insights using Google Analytics
1 — User flow: Although you can see the picture in the big line, it is actually hard to get trustworthy data for the dropouts:
2 — In-page analytics: OK, so GA says our users click at an equal rate on each car maker and we have 396% of people clicking on those links. What? We could not trust this one either!
3 — Sequences in advance segments: this option sounded like a good solution. However we quickly realised that more than two conditions gave wrong numbers. Also it turned out that using them was slowing down GA to a point where it was almost unusable.
If you add to those problems the fact that data in GA is calculated using sampling which adds a layer of noise to a report you already only half trust, and it really renders the data useless.
Great — so we have a test that failed and no idea why! The bonus point is that as a product and insight manager, it was actually my job to find why, so the time invested in this project wasn’t wasted.
At this point it was clear we needed to move away from GA if we wanted to learn something. We needed a software package that is fast, not using sampling, which shows dropouts very easily and not super expensive. Several option are available. We selected Mixpanel simply because carwow’s tech team has used it before.
Mixpanel is event based. You pay 0.0000x for each event. This means that if Mixpanel wants to make money, they HAVE to record everything.
And they did, in real time.
This is what we discovered:
In the control version, we did not “force” our users to pick a trim whilst, in the new version — well, without realising — we did. And this was bad.
A car can be completely different from one trim to another. Making our user select at this stage which version of the car they want, added an unnecessary friction as (1) a good part of them don’t know differences between trims and (2) they can change it whenever they want until the last minute before they buy the car.
Sounds simple right? Well, GA — despite the amazing engineering technology it uses — wasn’t able to tell us this.
Behind the scenes
The picture is not all bright however. Moving from a universal solution with an amazing community like GA to a smaller one that works differently, takes time.
For example, we had to change the analytical structure on the back-end and completely review the tag management solution.
Also connecting events from different sessions was a nightmare, and integrating our A/B testing tool was very hard. We now have to do the user-level event segmentation on our own, because Mixpanel’s feature for this is super pricey.
If you really want to understand where are the friction points in your user journey, we advise not to use Google Analytics.
For the “big pictures” metrics it is an amazing tool, and this is why we now use GA in combination with Mixpanel.