Why does Google now restricts the search results based on the location of the user?
Is it to combat scrapers? My opinion is NO. Let me explain.
A few days ago Google announced it has started restricting their search results to local Google domain version, no matter on which Google domain you are, based on the user location. This basically means that as a user you can not compare or check how a website may be visible in two different Google domains, e.g. Google.de and Google.com.
Depending who you talk to in the industry this is either hailed as a victory for Google in their combat against scrapers of the Google search results or as a setback in transparency. I don't think it is either of these misconceptions. First, Google is known for providing on average more transparency than most other big tech giants from Silicon Valley towards their end-users. Second, scrapers will only be temporarily affected.
Why scrape the Google search results?
The answer is as diverse as can be, but in most cases it comes down to one of the following scenarios:
- Providing a historic archive of development and changes of the Google search results (similar to the Web Archive).
- Providing a tool to monitor Google search results development and provide a free of charge graphic interpretation of fluctuations, such as Algeroo.
- Comparing different websites ranking for a specific query and often compare it with own or client website. This latter is often done by SEO tools for the purpose of doing data-driven SEO. For website owners Google does provide an API to check their own rankings, something Majestic is now utilizing within their product.
Technology can be really beautiful and adaptable. I am pretty confident that this new change will not stop most scrapers of search results and they will likely just find a workaround within the next few weeks (if they have not already) and continue business as usual.
Is it illegal to scrape Google search results?
Google does not like it and the Google TOS clearly states that Google does not allow for automated queries. So I don't recommend doing this. Having said that, the legal aspect of scraping is being debated for publicly -without login- available data as was illustrated in the recent LinkedIn trial outcome.
Also as Google is such a dominant force on the internet, the development of the search results is actually something worth preserving for posterity and we should be creating an open source archive available for everyone, similar to the Web Archive and the HTTP Archive. Unfortunately we are still very very far away from this.
Why is Google limiting the search results based on the location of the user?
Is it to combat search result scrapers? As I mentioned earlier, my opinion is NO, I believe that is a misconception. I do think that combating scrapers is an added bonus for Google and likely was listed as one of the benefits when this idea was discussed internally within Google prior to making the final decision to implement this change.
No, my opinion is that the whole reason for this change is driven from a legal perspective. Since the right to be forgotten came into effect in 2014 Google has battled with the problem that it would only remove content from the local country domains. Thereby keeping the problematic search results in most/all the other Google domains, often resulting in outrage from the courts, governments and entities that requested the search results to be deleted. Google tried to accommodate European law by extending the right to be forgotten to all Google country domains in Europe, however the other Google domains -including the most used domain Google.com- was excluded from this change and as a result this change was often not seen as enough.
The change Google just made will restrict users to only access the search results for their local country domain based on their location, no matter which Google domain they are on. So when a search result has been removed because of “right to be forgotten” and has not been removed from the other Google domains, it will simply be too difficult for the average user -including most lawyers, judges and government officials- to still access the search result from the country/jurisdiction in question. It solves the visibility issue for Google in regards to "right to be forgotten" without removing specific search results from all Google domains.
Of course Google is also promoting this change as being better for users and in addition making the life of scrapers harder is probably a nice bonus for the Google engineering team, however I think legal is the primary reason why Google is now limiting the search results to the local country domain based on the user location.
Does that mean we need to be worried, I don't think so. Google search results remain relevant for users and hopefully one day we will have access to an open source historic archive of Google search results worldwide.
About the author: Fili Wiese is a renowned technical SEO expert, former Google engineer and was part of the Google Search Quality team. At SearchBrothers.com he offers SEO consulting services with SEO audits, SEO workshops and successfully recovers websites from Google penalties.
This article is a personal opinion/view on developments in the world of Search Marketing.