In bed with Google? What we’ve learnt from Flexible Sampling six months on
In October 2017, Google announced that they were ending their prescriptive ‘First Click Free’ programme and replacing it with the new, more malleable, ‘Flexible Sampling’. Although The Times and Sunday Times did not participate in the former, the new initiative is far more attractive for quality subscription publishers. Six months on from implementation, here’s what we’ve noticed, and what it means for the next 12 months.
Before — First Click Free
In keeping with Google’s guidelines, websites cannot show their search engine crawler anything different to that which a user coming from their results pages would see — it’s known as ‘cloaking’, and rule number one to follow when it comes to playing Google’s game.
To adhere, subscription websites must allow Googlebot to crawl only what logged out users could see — not much, in most cases.
Google’s way around that was First Click Free, and it largely did what it said on the tin. When a user is accessing a subscription website from Google, they get their first click free and in return, Google will read all of your pages and index them in full.
The problem, though, is that Google decreed that one click per day was not enough, it must be ten. There were grumbles, and eventually it said: fine, it must be five. But still, there were fundamental issues. People found common loopholes and accessed more than their five per day by clearing their cache or using multiple devices. For many business models (particularly publishers), users rarely consumed more than five pages per day so any chance for conversion to subscription was null.
At The Times and Sunday Times there were too many flaws with the model to participate, so the business opted out. As such, Google continued to get stopped at our paywall during crawls and we became virtually non-existent in it’s search engine results pages. Our relationship with Google was frosty, and CEO Robert Thompson wrote that “Google’s commodification of content knowingly, wilfully, undermined provenance for profit” as well as describing them, along with Facebook, as having built “an ecosystem that is dysfunctional and socially destructive.”
Then — Flexible Sampling
In October, Google announced that they were replacing the First Click Free service with a new initiative designed to encourage the success of professional publishers: Flexible Sampling. They acknowledged that “publishers are in a better position to determine what specific sampling strategy works for them”. Hurrah.
Working very similarly to First Click Free, Flexible Sampling allows Googlebot to crawl entire articles and properly index the content in their results. However with this solution, the publisher is in control of how many articles a user can view without subscribing, a far more attractive proposition to many businesses.
Google gave us some lead time in advance of announcing Flexible Sampling, giving us the opportunity to decide whether to participate and if so, prepare for the changes it meant to our website. During that time, we audited our website, refreshed our SEO strategy and threw all we could into accelerating a newsroom culture change that was already underway.
There were some hiccups. Flexible Sampling is implemented at CDN level so in the process of letting Googlebot through, we had to be sophisticated enough to keep the access tight and avoid people accessing our articles for free under the guise of a search engine crawler. It involved whitelisting only specific IP addresses, and a lot of back-and-forthing with Google.
It was live for three days before we turned off caching, meaning that searchers could navigate to the place that Google stored articles as it saw them during a crawl — in their entirety. It opened us up to a readership free from barriers for 72 hours. An oversight on our side, but also a key point missing from Google’s developer guidelines.
Now — what’s the story?
We implemented Flexible Sampling at the beginning of October 2017, and results have been promising. This line shows daily visits to our website from search engines:
Cool, but particularly cool when we compare that with the same timeframe one year before:
The only day that visits from search engines were higher in the previous year was the day that Donald Trump won the US presidential election.
Although the implementation time kicks off a great graph, the results aren’t all down to Flexible Sampling. This, and the promise of a new relationship with Google, set a fire beneath the business. SEO initiatives could promise far higher benefits for our business model than they had before, and investment could offer real, concrete results in Google search pages. It made it far easier to predict positive results against our efforts.
Fresh strategic thinking, tech updates and sheer hard work in the newsroom have all contributed to search traffic in a way that Flexible Sampling can’t, increasing indexed pages and click through rate to the site.
We rolled out training at all levels from desk editor to sub-editor. We expanded the SEO team and doubled down on basic good practice.
We integrated SEO into the newsroom processes through the introduction of a morning meeting led with search data, giving editors access to general public interest in a way we hadn’t before.
But it was Flexible Sampling that opened the doors to the growth of SEO internally, making way for those things to flourish.
What’s next?
We continue to trial new initiatives, exploring the new opportunities that Flexible Sampling offers us. We’re still wary of some of Google’s practices but the future looks bright, and we’re looking for every day this year to be better than the day that Trump became president.
