The long tail lockdown: how Google is quietly fencing off the internet for itself

For years, I’ve been explaining that much of the richness of the internet lies in its long tail, the millions of small, specialized pages updated by communities and authors who do not appear in rankings but who sustain its informational diversity. More than 90% of queries and keywords relate to this long tail, making this entire periphery the true connective tissue of the web.

Today, however, this raw material is being closed off for training and grounding generative models. Google has just made a subtle but very significant change: last month, it quietly removed the search parameter that allowed 100 results to be displayed on the search results page (Search Engine Results Page, or SERP), which means that you can no longer see 100 results at a time; the default maximum is now ten. Part of the closure is supposed to be to prevent abuse, but it looks more like a simple lockdown. And with it, Google is preventing third-party algorithms from accessing that long tail and, in practice, skewing the LLM market.

The first problem with this is that in most cases LLM crawlers use Google search results in addition to their own, which means that Google has just taken away 90% of the results that those LLMs rely on. This is basically the end of being able to opt-out from AI at the infrastructure…