What Does Mobile First Index Actually Mean?
Ever since Google’s Gary Illyes dropped his mobile first index announcement at Pubcon, there’s been lots of theorizing, conjecture, rampant speculation, panic, and confusion about what exactly that means. Will desktop users get mobile sites? Will sites without the mobile friendly designation suffer? Do we need to change all of our canonical tags? How will Google handle the reduced token corpus? What the hell are tokens and a corpus?
Google has gone on record that they are still figuring out exactly how to handle some issues, but that’s probably not comforting to many businesses who rely on Google traffic to pay their bills. As a recovering software engineer turned SEO, I’d like to try and tackle some of these complaints head on and reason through just exactly what may change, how Google might handle them, and what exactly some of these things mean. At the very least, I will add to the theorizing, conjecture, rampant speculation, and panic.
In the spirit of TAGE (that’s TAGFEE minus the fun and empathy) I should disclose that not only did I (and a couple others) share a meal with Gary on the day of this announcement and get a chance to voice our concerns, but we also met with him again after it and had several long talks about these issues on Twitter.
So while much of what I say is technically still conjecture, it’s based on several conversations I’ve had with some Googlers. It’s also important to note that unless I specifically say “Google said this” then Google didn’t say it.
Ok, enough of the CMA-type talk. Let’s get down to it.
First, Let’s Understand What Indexing Means
If you saw my talk at SEJ Summit Chicago I went pretty in-depth into the differences between Crawling, Indexing, Retrieval, and Rankings — the core parts of any search algorithm. It’s important to distinguish them. I assumed this was SEO101 material, but I did see some confusion come up in a few Twitter chats the other day.
- Crawling is the process by which Google follows links on the web to discover pages. It’s here that they likely discover your content and where they may also apply nofollow and disavow file rules.
- Indexing is the process of turning your webpage into something more useable and storing it in their database. There’s a lot of cool stuff that happens here like word vectors, ngrams, and all kinds of other awesome computer science stuff I’d love to geek out with you about over some beers. For our purposes though, Indexing is when they make a copy of your page in a format that’s useful to the ranking algorithm. This is basically caffeine, from what I gather.
- Retrieval is the first part of the search query. It comes before ranking in my example, but there’s a good chance that the actual algorithm does retrieval and ranking at the same time. Retrieval is the part where the engine says “give me everything relevant to this query.” This is the part where those word vectors and ngrams get put to use. This is also the part where Rankbrain most likely comes into play. Hummingbird too (I think — not that it matters.)
- Ranking is the part we obsess the most over. It’s where they order the results based on whatever number of factors we’re arguing about today. Much of this probably happens during the retrieval phase. However it’s likely that some factors (like penalties, and speed, and even mobile friendly) happen after retrieval. At least that’s how I’d code it — but I’m not as smart of an engineer as some Google employees.
Ok great, what does any of this have to do with the Mobile First Index? I’m getting there, but the WordPress plugin says I need a couple hundred more words, so bear with me. (Kidding, I swear it’s useful.)
So What Is Mobile First Indexing?
Right now, Google has just one index based on the desktop site. It creates signals based on Googlebot with the desktop user agent. Google then crawls with their mobile Googlebot to gather mobile friendly and other signals — but they aren’t creating a new index based on the the mobile site.
Currently, when a user searches Google (either desktop or mobile) the retrieval part of the algorithm looks at the desktop index created by the Googlebot desktop crawler. It finds relevant results based on the desktop index, then ranks them based on the desktop index and even shows the searcher a snippet based on the desktop index. The Ranker then looks at the mobile signals collected by the mobile crawler and adjusts the rankings accordingly.
This has caused some problems. There’s way too many cases where a user sees something in a snippet, clicks the results, gets redirected to the site’s mobile homepage (which probably spawns an app store or newsletter popup) and then realizes the content they saw in the search snippet isn’t available on the stripped down mobile version of the site. This is a bad user experience but it’s pretty much the norm on too many sites.
With this new change, Google seeks to stop that. The general theory (my words not Google’s or Gary’s) is that if the content isn’t important enough to be on your mobile site, then maybe you aren’t the most authoritative or relevant result for that content.
Ok…What’s That Mean?
Many of the conversations and blog posts I see lately are all focused on Ranking. While it’s true that indexing can affect ranking, it shouldn’t be our foremost concern. We should step back and look at indexing if we truly want to be prepared for this change.
When it comes to indexing, there are only a couple of potential situations that can occur for your site. Let’s ignore the concept of “mobile friendly” because that only offers a teeny tiny ranking boost. It also may not come into play here.
It’s important to distinguish between “mobile friendly” for ranking and “mobile index” for relevance. While the weights of “mobile friendly” may change, it’s not the same concept as a mobile index. A site can be in the mobile index without being mobile friendly.
When it comes to mobile first index, there are only about three buckets of scenarios: A website is either responsive, has a separate mobile site, or doesn’t have a mobile site at all. Let’s look at what this means for those three scenarios.
The good news here. Pretty much nothing changes with regards to indexing. The same content is (mostly) seen by Googlebot mobile and Googlebot desktop. There’s still some issues here that Google needs to work out but I expect responsive sites to not see much of a change. Issues include things like changing the weights for tabbed content or drop-down menus which are probably less valued on desktop but shouldn’t (in theory) be devalued in mobile.
Separate Mobile and Desktop Sites
Here’s where things get tricky. If a site has device type redirects OR rel=alternate and canonical tags setup, then the mobile crawler will see the mobile site only, and not the desktop site. That means if some content is ONLY on the desktop site, the mobile Googlebot won’t see it and it won’t end up in the mobile first index. This is the issue Google is trying to solve, but it’s also an issue for many publishers.
I showed the below slide at Pubcon. Using SEMrush I grabbed all the keywords that Ampproject.org ranked for on both mobile and desktop. I then graphed them and highlighted the mobile keywords. What’s left un-highlighted is all the keywords where AmpProject.org is ONLY ranking on desktop. That means all of this content is likely not to be seen by the mobile crawler and may soon stop showing up after this mobile index change is made. Or does it? Let’s examine that in the next section.
No Mobile Site
The last category is pages that don’t have a mobile site. There’s still a ton of those out there. Here’s the good news: The mobile Googlebot will still see these pages! The mobile crawler doesn’t just crawl “mobile friendly” pages. It crawls everything. These pages will still be seen — they just won’t get the “mobile friendly” designation — but that’s completely OK because it has absolutely nothing to do with mobile first indexing. Sure they won’t rank as well as mobile friendly sites — but they’re already not ranking as well as mobile friendly sites. That won’t change after mobile first indexing.
So, What Will be Affected?
Basically, the only pages affected will be pages who have a mobile version that doesn’t include the same content as the desktop version. Again, it’s important to note “page level”here. If we look at our AMPproject.org example above, we see that much of the content is a case study. If that case study had a mobile version that didn’t include all of the content that the desktop page did, then it would suffer from this change. IF, however, that case study ONLY had desktop pages — it would still rank! It’d just be ugly on mobile phones. That’s an important distinction.
And that’s the distinction I’m not seeing many SEOs make. I’m seeing lots of posts looking at the number of pages on a site’s desktop and mobile sites. Unless those desktop pages that don’t have mobile equivalents are redirecting everybody to the mobile homepage, that metric doesn’t matter. Those pages will still get indexed. What we should look at is the content that’s on the desktop page but not the mobile version.
Great, Enough Theory. Just tell me what to do.
Sure, no problem. Make. A. Responsive. Site.
But what about where my mobile users are different personas than my desktop users? What if I want to tailor my mobile site to the fact that they search differently?
Great. Do it! But do it smartly. Think about your site. If your homepage will be missing a ton of content for the mobile version, then maybe that content belongs on a sub-page instead of the homepage? It’s going to take some good Information Architecture, Content Strategy, and UX teamwork, but it’s entirely possible to create a different mobile experience without losing content. It just takes forethought. Hey, nobody said SEO was easy right? It’s time to roll up our sleeves and do some RMS. Real Marketing Shit.
So you’re saying SEO is dead?
Yes. SEO is dead. We need to stop doing SEO and start doing SEO. No, it’s not. This doesn’t really change much in the long scheme of things. It’s just a bit ahead of the curve based on where websites are currently.
What if I’m not responsive? Can I do some little things?
Yup. For starters, get rid of all those device type redirects from sub-pages to the mobile homepage. Nobody likes those. If you don’t have a mobile equivalent of the page, just serve mobile users the desktop page. It’s actually a better user experience to get the content they want in an ugly format than a pretty page that doesn’t relate to what they’re looking for. Better yet, create a mobile version of the page!
What about rel=alternate and canonicals? Do I have to change them?
Probably not. Google has mentioned that it’s likely they’ll be smart enough to handle them on their own — as asking Webmasters to change them all would basically never happen.
Will this open up the doors for spamming desktop users?
Probably not (But that won’t keep us from trying right?) Just like Google visits your site with multiple user agents now to determine mobile friendly, I assume they’ll keep doing that only in reverse. If your mobile site is about Taylor Swift and your Desktop site is about Viagra, they’ll easily catch that and take appropriate action.
Start thinking about micro-formats. For some reason, many SEOs didn’t add these to their mobile sites. Make sure you’re using your schema and hreflang on all your pages.
That’s pretty much it. If you think of other questions or concerns, please share them in the comments or annoy me on Twitter and I’ll gladly take a stab at answering them.
PPT slide — my own creation
Featured image: Flickr creative commons license
Originally published at www.searchenginejournal.com on November 21, 2016.