How to identify & deal with Dark Traffic

Let there be light in your analytics reporting

by Jonathan Swain

Looking at darkness and light on a Star Wars-esque symbolic level, “dark traffic” doesn’t sound great. And it’s certainly problematic if you rely on analytics data to gain insight into your website’s performance.

What is Dark Traffic?

Dark traffic is any traffic that comes from sources stripped of referrer information, meaning the source is not attributed correctly within your Analytics platform. This causes dark traffic to get bundled in with your genuine direct traffic.

Dark traffic has three main areas: dark search, dark mobile and dark social. Sources of dark traffic can include:

· Secure search

· Image search

· Links from non-web documents (e.g. Word and PDF docs)

· Links shared via messenger platforms

· Traffic from apps

Google had already made life increasingly difficult for us Jedi Knights of SEO, both inadvertently and intentionally, with a few recent changes:

  1. The introduction of ‘not provided’ keyword data back in early 2012
  2. Search Console data linked to Google Analytics accounts now only having a time span of 90 days
  3. The use of the Keyword Planner tool being restricted to those with active AdWords campaigns

Add all these factors to the dark traffic problem, and you’re at risk of flying into a metaphorical asteroid belt when it comes to accurately tracking and reporting your data.

If you’re responsible for organic performance, then you need to make sure you have a handle on this and know how to identify and correctly report this ‘missing’ organic traffic.

And it shows no signs of improving. Indeed, it’s likely to get worse. But providing you know what it is, where it comes from and how to identify it, you can bring your data back from the Dark Side.

How does it affect analytics data?

It’s hopefully obvious that pesky dark traffic is going to cause mild havoc with your Analytics data. Maybe your hard-earned social traffic from that great campaign you worked on is getting chucked in as direct traffic. Or a heap of organic traffic to your well-optimised and informative web pages is also being claimed as direct. In short, this is not the traffic you are looking for.

If you’re responsible for organic performance, then you need to make sure you have a handle on this and know how to identify and correctly report this ‘missing’ organic traffic.

Dark search

‘Dark search’ occurs whenever a referrer string is not passed during a visit. When this happens Analytics sees the visit as direct traffic. There are a number of reasons this can happen, such as browser privacy stopping the referrer data being passed if you are browsing using Incognito mode. The same applies for some browser add-ons which block referrer data. HTTPS to HTTP traffic is also a culprit here, as referrer data is not passed during redirects from https to http page versions.

It also became apparent in May last year that there was another issue affecting mobile organic traffic. On the 27th April 2016 there was a spike in traffic from the referral channel in Google Analytics. As it turns out this spike in traffic was coming from the Google App for Android, due to the latest update of the app. This situation is very similar to the iOS6 issue from October 2012, although those searches were showing up as ‘direct’ traffic rather than ‘referral’.

The graph below shows an example of the increase in misreported traffic, from 27th April onwards, coming from Google Android app search reported as referral rather than organic.

Dark social

The term ‘dark social’ was originally coined by Alexis C Madrigal in 2012. It’s defined as traffic to your site which originated from the share of a URL which was subsequently clicked on, but is attributed as direct in Google Analytics. This can happen when links are shared via encrypted conversations which stop referrer data being passed — email, Facebook Messenger, Twitter Direct Message, WhatsApp etc.

Dark mobile

Unless you have been living under a rock, you will be aware that there has been a huge increase in mobile search in recent years. Fun fact: mobile now drives over 55% of traffic to top sites according to SimilarWeb. This growth looks set to continue, and as such, the rise in ‘dark mobile’ is also set to continue.

Messaging apps continue to increase in popularity and number, and with this comes the inevitable increase of people sharing links with their social circles via this method.

How to identify & report dark traffic

As you can see there are many issues that can affect the reliability of analytics data. Now that you have an understanding of what dark traffic is, and where it’s hiding, you can follow the methods below to help with identifying and reporting this data:

1. Create a custom report in Google Analytics

Set up a custom report with Google Analytics to show your direct traffic with an advanced filter applied to exclude your site homepage and other main pages that people may potentially access via a ‘true direct’ visit, i.e. someone physically typing the full URL into the address bar.

Your main navigation pages would generally be a sensible guide to use here, based on the assumption that it’s highly unlikely anyone is typing in a full URL that is more than one category deep. Once you have filtered this guesstimate of direct traffic, the pages that remain will be your dark traffic.

2. Verify filtered direct traffic with current social campaigns:

Look at any current and recent social campaigns and link these pages to your filtered direct traffic — this is your dark social traffic.

3. Identify your dark search traffic

Once you have your bucket of dark social traffic singled out you can then filter the remaining data to show only ‘new users’. This will filter out instances of true direct visits to deep page URLs such as autocomplete generating the URL to a previously visited page. You now have your dark search traffic.

4. Split out your dark mobile traffic

Once you have identified your ‘dark search’ traffic filter this by device category and you can then split desktop versus mobile traffic. That’s your dark mobile covered.

This isn’t going to help you identify visits for past campaigns but it’s a must-do step for any future campaigns.

5. Always tag links on your campaigns using Google UTM parameters

Using Google UTM’s parameters on links for campaigns allows you to track exactly where the visit has come from. This isn’t going to help you identify visits for past campaigns but it’s a must-do step for any future campaigns. Google even gives you a tool to build the URL.

Once you have tagged your pages, you can view the results of these campaigns in Google Analytics by navigating to Behaviour > Site Content > All Pages and dimension set to ‘campaign’.

6. Set up a custom filter

All the steps above are helpful to quantify the issue, although they don’t fix it. So, what can be done? While there is no 100% fix, you can use the information garnered so far to put a filter in place to attribute this traffic appropriately. If you are unsure of how to go about setting up filters within Google Analytics, then this article is a good place to start.

7. Implement HTTPS

How will HTTPS make your analytics data better? Essentially, if your site is non-HTTPS, a visit from an HTTPS site will not pass referrer data. So, this visit will be thrown into your channels as ‘direct’ traffic. However, a visit from an HTTPS enabled site to another HTTPS site does pass referrer data. Likewise, a visit from a non-HTTPS site to an HTTPS site will also pass referrer data.

By making the switch over to HTTPS you are going to increase visibility of referral data. It was once probably a tad drastic to take this step solely to improve your Analytics data. But with many other benefits to making the change, it’s well worth considering.

Thankfully SEO is an ever-changing field with a lot of smart minds, and that’s what makes it so interesting.

Conclusion

There’s no foolproof way to correctly identify dark traffic with 100% accuracy. However, following these steps while keeping your eyes peeled for any unusual data patterns (such as the Google Android App issue mentioned previously) can help you to gain an understanding of it and try to nullify the effects as best you can.

Most importantly this information should empower you to report, as accurately as possible, on that ‘missing’ traffic and make sure your organic campaigns are getting the credit they deserve.

What’s the forecast?

The issue of dark traffic isn’t one that’s going away anytime soon, for many reasons. The increase of mobile traffic and the continuing ease with which our mobile devices allow us to share content with our social circle on an ever-increasing selection of platforms.

As Google continues to reduce free access to data, this is something we are going to have to continue to battle with. Thankfully SEO is an ever-changing field with a lot of smart minds, and that’s what makes it so interesting. With every problem that arises, a solution is never far around the corner.


Jonathan is Head of Insight at Caliber. Visit our site if you’re curious about getting top notch reporting and forecasting.