Device Experience Mining for Ecommerce Conversion

I was showing a student a runthrough of an ecommerce site model the other day and they liked it so much, I decided to document this for rapidly helping you ‘find stuff that costs money’.

There’s a very quick and easy way in GA to find device and browser opportunities. What are these anyway? Bugs? UX issues? Rendering problems?

Broadly speaking, it’s anything that’s holding back your product from performing ‘how it could be’. Just being aware of where your product is deficient and how much it’s worth to fix it and how long it might take is vital. If you’re a pothole company, you need to know how many, where, how much to fix.

Just like potholes, you can’t fix everything. Knowing what’s wrong though is the first step in finding out what’s easiest and most impactful to fix first. If you’re the sort of fool who says “We test for this stuff” or “We test all the popular browsers and devices” then I will say one phrase — Prove It. If you don’t know where the potholes are and how many there are or how big they are — don’t tell me about your bullshit strategy for potholes.

For example, let’s assume that unbeknownst to you, the developers who recently tweaked your funnel introduced a small bug. It only affects certain android and iphone models and makes it really hard to find the address. In the big sea of data or dashboards that people look at, this change simply isn’t surfaced or is lost in a larger ‘average’ of measurement. It still sucks 100 grand out of your potential sales each month and you don’t even notice.

It’s a kind of neglectful discrimination you’re practicing — inflicting all these potholes on us. When we enter a shop in our high street, we don’t expect the experience to be fundamentally ‘broken’ just because of what we’re wearing or the kind of shoes we have on. That’s against the law.

Imagine if you couldn’t go into shops because a sign said “Only iPhone owners supported here” or “Droid wearers will be discriminated against because our developers hate chrome” or “No Blacks, No Irish, No Dogs, No Samsungs”

There would be an outcry! Lawyers would earn hourly rates. This stuff would get taken down.

But IRL 2017, it’s like the dress rules on nightclub doors, except someone wrote them in invisible ink and you have to trust it will work. Real life would be a bitch if we followed the same quality standards as web code.

So I often look for symptoms of these issues. What alerts me?

Variances. Outliers. Oddities. Patterns. When I segment the data the right way, they stand out. So here’s a quick teardown of assembling a grid for some of the easy places to look in Google Analytics. If you fill out this sheet using the instructions, you’ll have your first clues.

My example is for an e-commerce site but you can adapt this for any session or user based goal outcome. I’ve done an excel because this works for everyone and I can share the custom reports where you grab the data values to plug into the excel. I’ll also show you an mocked up example — to clue you up on what to look for:

Custom Report Grab

Firstly, you’ll need this report:

Then, grab this blank spreadsheet.

Fill in the blanks:

I had to create completely faked up data, in order to create you a nice walkthrough document — this was a pain in the a** but it makes explaining this so much easier. I wrote an original version of this walkthrough and it sucked without the nice easy pictures. By giving you the reports and showing where to grab the data points, it will stick with you.

Load up the custom report above and you’ll be presented with 3 rows of data — mobile, desktop and tablet traffic. This is the ‘high level’ view by device category in Google Analytics.

You’ll see something like this:

Custom Report — Top Level

Device Category

Using the custom report values, go to your Excel sheet and fill out the columns for Sessions, Users, Sales (number of transactions) and Revenue. It should look like this now, using the example above:

Device Category Figures

The column titled “% of this section” you can grab from the ‘users’ column — which gives you the percentage for each category shown above.

Well done — that gives us our high level — now let’s do the mobile section:

Mobile Section

This is your next section on the spreadsheet. First, edit the heading to say Mobile (70%) — this acts as a reminder of what share of the user pie this chunk is.

To fill out this new mobile specific section, we need some new data. Go back to Google Analytics and Click on the ‘Mobile’ device category drilldown link as shown here:

You will now just be looking at mobile devices, split by their operating system, like this:

Just like my example shows you, iOS and Droid will be dominant in nearly every setup. It’s rare that I find a significant (i.e. 5%+) base of Windows for mobile devices, even in some odd markets.

That means you should have 2 or 3 rows here, MAX. Be aware of sample sizes if you include anything but droid or iOS here! Don’t sweat the smaller segments here or anywhere else in this model.

So using the example above, you fill the spreadsheet out like this:

Mobile Section — Filled Out

Tablet Section

Let’s grab the tablet data.

Go back to the top level report and click on the tablet drilldown:

You’ll see the following data shown:

Again, fill out the spreadsheet data with the values above.It’s rare for there to be anything but mainly an iOS/Android/Chrome/Windows show here. The other segments are too small in my example to worry about — the two above are 99% of tablets.

Your filled out spreadsheet now has a tablet section looking something like this:

Filled out Tablet Section

Desktop Section

Go back to the top level report and click on the Desktop Drilldown.

You’ll see something like this:

Desktop Device Category Drilldown (by OS)

These are the 4 main groups — Windows, Mac, Chrome and Linux. There’s a smattering of smaller stuff on every setup but these are always the main players. I’m going to focus on Mac and Windows — because they are more decent samples and comprise over 97% of stuff on desktop.

Now at this point you are wondering why Mac Desktop posts so high — that’s a mystery for another day. Let’s say it got you curious eh? Me too — and yes, there’s a reason for this but it’s not always evident.

Remember here that there are two reasons for devices to convert differently — either this ‘population’ contains a group of people that are significantly different (behind the phone) or it contains a group of device experiences that are significantly different (on the phone). Sometimes the reasons Mac and iPhones convert higher is because they’re linked to a different socio-economic group. They. Have. More. Money. Sometimes the reasons other devices convert badly is because the experience is shit.

This is one of the reasons you can’t jump to conclusions about why these differences exist. It’s true to say Macs convert differently here but not true to assert that this is somehow down to a much better browsing experience. You just don’t know why yet.

Our work isn’t done yet here either (on the desktop front). What we really need to know is what kind of browsers make up those averages? I also split this data by resolution but that’s for another post some day!

Drill down from the top level ‘Desktop’ report. Then click on the report tab called Browser Split.

Let’s just split by browser first — see what comes up:

The first problem here is that browser isn’t enough data — all these browsers behave differently depending on what version people are using and what Operating System they are on. We need a better split than this one — we need to break it down by Desktop -> Operating System -> Browser class

So, fill in the values into your desktop section but bear in mind these are flawed. I’m now going to drill down into browser versions by OS.

So, head back to the main report tab, go to the top level (mobile/desktop/tablet) and select ‘Desktop’ as your drilldown. Now choose Windows as a second drilldown. You will see something like this:

The Internet Explorer browser class sucks here — Edge and Firefox could convert better too, but they are smaller traffic and not as badly off the average.

If you drill down into the “Internet Explorer” option above, you’ll get a series of rows for each version of the browser.

In this case, I’ve shown IE 11.

For Chrome, Firefox and Edge — you have a huge amount of browsers that are automatically updated. You won’t find any easy splits of browser versions but it’s useful to know what the latest versions are to test against, when it comes to QA.

In my case, I found two key differences — Windows IE11 vs. Chrome on Windows and Mac Safari vs. Chrome on Mac.

I grabbed these two pairs of data by drilling down using the Mac and Windows drilldowns on the custom report. You’ll end up with something like this:

Filled out Desktop — OS — Browser split

So what do you look out for then Craig? Well — I know the following:

Edge is an auto updated browser — Microsoft’s latest — so I should always be testing or checking conversion for this browser type as it’s a growing segment.

Internet Explorer 8,9,10,11 — *may* be significant for you. However, you need to check what these are worth in terms of cost to fix vs. opportunity. Making code work across browsers *including* older versions of Internet Explorer is devilishly difficult and costly in testing iterations.

Chrome again is an auto updated browser.

Safari is a bit more complicated (depends on platform and OS what the update strategy or version is)

For the auto updating browsers out there you’ll normally find there’s a small spread amongst the latest versions and a smattering of older stuff. Although it’s tempting to analyse all these sub versions, it isn’t practical — because we don’t know how these browsers differ. Testing them is another problem entirely. Let’s just think about it logically though — if 12% of users are on version 192.10 and 40% of users are on 192.09 — should you test 192.09?

Nope. Because most of those will be using 192.10 by next week. I normally recommend testing the latest version that has some significant traffic — since most users will end in this bucket by the next time you look!

Internet Explorer shows significant differences (and I include Edge as a sort of Internet Explorer 12+) on almost all platforms. Don’t get carried away though — unless you have significant traffic that’s having a crappy experience.

You did pay marketing dollars for them to have a painful time? There is traffic for these browsers? Shame on you if they’re not converting! You ask them to come and then discriminate against them?

One last split to show you here — for iPhone models. I’m not going to break down the Droid stuff in this article because it’s more complicated but you can use this specific report to pull the Brand, Model and Browser splits for just Android devices:

iPhone Section

You’ll need this custom report to fill out this part:

The iPhone model detection is complicated in GA — because there are different security models for content viewed ‘through’ an app and content visited by a browser on the phone. GA isn’t much help here, so we have to rely upon the resolution that the model reports to GA.

Load up the report and you’ll see something like this:

Operating System = iOS, Mobile Device info contains “iPhone”, Device Category = mobile (split by resolution)

So what models are these then? Handily there’s a guide here — the resolution that iOS reports to Google Analytics is not the real resolution, hence the need to check this table:

I know from hands on testing that these work very well — so, let’s decode them for you — it’s not perfect but it will do:

320 x 480— iPhone 4 or lower

320 x 568 — iPhone 5 or SE

375 x 667 — iPhone 6, 6S, 6+, 6S+, iPhone 8

414 x 736 — iPhone 7+ or iPhone 8+

375 x 812 — iPhone X

THE ANALYSIS

If you’ve been following, you’ve probably spotted a few things. Let me concentrate your mind — we need to work out what the ‘value’ of fixing these segments are.

One way of looking at this is to pick an anchor or ‘hero’ converting group within the segments I have shown you. For example, is there any reason why Droid in this example converts at nearly half of iPhone visitors? Why?

One good way to figure this out is to know what the good and bad experiences are by testing them. But for a quick and dirty model, what if all browsers, tablets, mobile devices — converted as well as the ‘reference’ standard here? What if Internet Explorer 11, for example, converted as well as Chrome?

This at least gives us some opportunity value to testing this segment — we have to temper this with how hard it will *actually* be to fix any bugs uncovered but we know the ‘raw’ size of the opportunity in a ‘theoretical’ scenario. I do find differences that I can’t explain but most of the time, it’s a bug or flaw in the experience that causes these huge differences.

Back to our spreadsheet. You’ll see some additional columns I’ve added — called:

User Ecom CR — This is the sales/users ratio (better than session metrics here)

What if? — This is a calculation of what would happen if we pulled this segment up to reference standard.

The calculation is:

(users*reference ecom CR)- current sales)*AOV

This basically asks — what if this segment had the conversion rate (of users) of its higher converting sibling?

Once I pile all these into the model we made, I see this:

BY CATEGORY

I’m not expecting to plug 100% of these gaps as I don’t know why they are different— it’s just interesting to see the variances between desktop, tablet and mobile.

On many sites, unless there’s a darned good reason, I’m expecting tablet CR to be close to (over or under) the desktop CR. Let’s look a bit deeper

MOBILE ONLY

This is quite low in my experience — but it varies. The only way I can be sure is to test this. It’s worth a chunk of money though!

Let’s keep going….

TABLET ONLY

We already know that tablets are lower performing than expected. We can see a clear difference between droid and iOS tablets — which is one opportunity.

The bigger one is to increase the raw conversion rate of the tablets. We would need to test a late model iPad to figure out if there was an opportunity here. I’m making a big leap/assumption here — but I modelled what it would be worth if iPads converted as per the average desktop (which is a low target).

We sometimes have this problem — comparing a reference browser — because what happens if that reference standard is crap, but just not as bad as what you’re comparing it to? These are clues I’m finding — not solutions, so I have to be wary and yes, sometimes I find nothing. C’est la vie!

Let’s look at desktop now and compare them to my reference of Chrome:

DESKTOP ONLY

As you can see, the real poor performer here is Internet Explorer — Edge (as a more modern browser) shouldn’t be this low compared to Chrome, so that tells me something is wrong with Edge and IE.

I’ve excluded Safari here because the way these visitors are targeted means we end up with high motivation groups here using Macs. That’s why this comparison is flawed — because it spans across different Mac and Windows platforms. I can see hints but not a final pattern. We need to compare like with like.

DESKTOP WINDOWS ONLY

Here you can see how much fixing IE11 is worth. What I haven’t mentioned here is that I’m going to test Edge anyway (as well as IE11) because the proportion of people using it will increase week on week. Future proofing.

DESKTOP MAC ONLY

I’m interested why there’s such a big difference here — why is Chrome so much better performing? Is one version of Safari pulling this average down? If I can find a segment driving this, we can do testing.

iPHONE MODELS

Now this is really interesting and I see this pattern crop up a bit — it doesn’t mean I can assume anything until I test though.

What this looks like is that the smaller screen iPhones are converting atrociously compared to our larger screens. Even if we could approach the conversion rate of the iPhone 6, it would be a start. iPhone 4 & 5 models are still a big chunk of traffic (>50% of iPhone models) so it needs to work on all of them. Testing a model 5 seems like a dead cert here.

So what did I end up with then from this analysis:

(1) Test a late model iPad and droid tablet (find out why tablets and in particular droid tablets are converting so poorly)

(2) Test top Android models, probably an S7 or 8 (find out why droid mobiles are converting so poorly)

(3) Test Internet Explorer 11 and Edge (find out why these two are much lower than chrome on windows)

(4) Test iPhone 5 (find out why smaller iPhones CR decays so badly)

(5) Test Mac Safari vs. Chrome (find out why Chrome much higher)

REALITY BITES

I probably won’t get enough time to test everything — so what would I choose if I had limited time? Would I skimp on the number of devices or the time spent with each device checking the journey?

I’d always go for depth rather than skim here — it’s a tradeoff though and there are many ways of working it out. You might not have the devices. It might be expensive. There may not be the time budget.

So — you should start with the biggest pile of money left on the table. For this site, we can see that:

Mobile droid on par with iOS = £85K

iPhone 5 upped on par with 7 = £46K

iPad on par with average desktop CR = £46K

iPhone 6 on par with 7= £37K

Internet Explorer 11 on par with Win Chrome = £17K

Internet Explorer Edge on par with Win Chrome = <10K but worth doing

WHAT NOW?

At this point, you’ve got a good idea what devices and/or browsers are good or not so good in terms of CR. You still don’t know why.

You can at least now sit down and get an estimate of the key journey to exercise, by testing the most promising device or browser. For the above, I’d start with the iPhone or Droid mobile experience.

Once you’ve done one pass through the key journey (that’s another article I need to write) then you have a rough estimate of the repeat time (problems begin to repeat in later tests)

One problem is that I have to continually trade off what I would ‘like to test’ with ‘what is within budget and possible’ — so please do your best to make this as good as it can be and most representative of your users, not what you have on your fucking machine. You will be surprised how much money this is worth.

I hope this is a good example of how a small amount of time helps you pinpoint potential pain areas. I would normally now create a funnel or step drop diagram for these segments — which tells me where to focus the testing time I have for each device. Data helps take what seems like an impossible task and cuts it down to size!

In this particular example, there are masses of bugs that really did appear on the tablet experience (for both droids and iPads).

On mobile devices, the droid usability was afflicted by multiple problems including bugs in navigation that made it incredibly hard to use.

On desktop, The bugs in IE11 and Edge showed us that there was a good reason for these performing so poorly. We even found a bug on Mac Safari too.

Total time to build the model, pinpoint the tests and run the tests = 3 days

Total value of fixing the worst stuff? Nearly a million quid every 3 months

— and this money quite literally was leaking for months out of this site like a bust oil pipeline, due to the range and depth of experience issues on every device.

Nobody checked or fixed anything. Customers didn’t complain or if they did, they were ignored. Very common for people NOT to report these — they just assume your site is a piece of dog poo.

I suspected there might be a problem with smaller screen iPhones and I was right (luckily). There are navigation and page overlays that totally shaft conversion — because they were designed for a bigger iPhone, not the one that customers actually use.

If you’d like me to teach you, show you or actually find missing latent conversion in your site, that’s easy to fix, shows immediate results and makes customers happy — let me know!

Hope you enjoyed this walkthrough — it took nearly a day to put this together as a demo, so please give me clap or whatever shit Medium has these days. Good luck and please ask questions.

Cheers!

C.

DOWNLOADS

Device Opportunity Report:

Android Mobile Devices Split:

iPhone Models Split:

Excel Spreadsheet (blank):

C.