We’ve given Google far too much power.
Every month Google answers 100 billion queries, we only see the results they deem relevant. Here’s a non-technical look into how they got there, what they do, and why they may have more power over us than any intelligence agency.
Occasionally a friend will ask:
— ‘What exactly do you do?’
— ‘I make educational sites that millions of people use each month, other companies advertise to my audience’.
Boring now, but just a few years back it was a radical answer.
I became a ‘Webmaster’ in my teens. In the early 90s there were perhaps only around 2,000 websites (quite a few were inexplicably green). I’d tell people about the big revolution coming, they'd momentarily appear interested, however it didn't apply to them quite yet.
By the mid-90s that number had grown to a hundred thousand or more. And somebody somewhere invented a better way to navigate them: The Search Engine was born!
1995(ish) - AltaVista, Excite, Hotbot, Infoseek, Yahoo, Lycos!
People had heard of ‘the internet’, but mostly didn't care.
Our world was filled with promise and excitement. This was the revolution we'd all been waiting for (by ‘we’, I mean ‘nerds’). I was a young nerd, with my future ahead of me. No longer did you need to be a ‘suit’, or own a fancy building to succeed. Anybody could create a business or voice their opinion. New concepts and new ideas were flying out of every country, culture and group involved.
Back then you’d pass up buying domains like ‘hotels.com’ because it “didn’t seem appropriate”.
In those very early days we saw a confused mess of creative, immature and wildly revolutionary concepts spill out of our small global community. Little business began to see the potential, big business was oblivious to it (some even called it ‘trashy’). Five years later they'd begin to wise up.
We, and the incredible engineers who had gone before us, changed the world - outright. There was excitement as the rest of the planet began to join in.
1999(ish) - In comes Google: “Don't be evil.”
A spiffy young company, full of promise and merit.
For some reason, this new company just blew many of us away. “Don't be evil” - hey great, that’s what we want “no evil”! It was tight, simple, and the results represented the best the internet had to offer. It changed our world, it was the day we found out how much had really gone on around us.
It was common for us to be in tech positions at this stage. Many sent network updates in their companies instantly installing ‘Google’ as the default homepage on thousands of computers. It was a different world; security policies for that kind of thing barely existed.
In short, if we had backed a different horse; you might never have heard of ‘Google’.
We'd tell our friends who were getting online, we'd evangelise horrifically to anyone who mistakenly asked.
We loved what Google stood for, this was the next stage of the information revolution. “Search [fit] for kings”, given to everyone, the ultimate democratiser.
We backed Sergey Brin and Larry Page because they understood the true purpose of the internet: The liberation of information.
2005(ish) - Something is ‘off’.
Nerds are split into many groups. New nerds have joined, everybody is now claiming nerd-dom, all are welcome. The media frenzy is almost over, the internet is now part of everybody’s life, and Google is front and center. Everything on television has a url. However not all these new nerds embrace the original concepts of the revolution. This is now big business and the world is looking through Google supplied glasses.
2010(ish) - Oh wait, no way, they wouldn’t.
What happens next, is what concerns many of us. I’ll attempt to take it out of its technical, industrial and secretive context for your review:
Nota bene: Everything here has been debated and re-debated by some of the smartest minds within our community. This is my, and some other fairly powerful webmasters, interpretation. Due to the secrecy surrounding search, we do not posses conclusive or definitive data. It is drawn from our limited, but considered perspective and the writings of hundreds of other webmasters.
The point of Google, according to Google, is to deliver the most relevant search result when you type a query: “what is this thing on my finger?”, “replacement washer valve” , “holiday in Malta” , “jobs for nail technicians”, “best student loans”
Google goes through the billions of pages stored in its ‘index’, and decides on a set of results. It does this in a few milliseconds; computers are fast.
Let’s pretend the internet is a city.
This city is 3,000x bigger than London, New York or Tokyo.
The groups found in this city:
1. Skyscrapers, Wall streets, Wal Marts They have deep pockets, they have media outlets, they have enormous marketing budgets. They can offer great value. The ‘Titans’ of the business world.
2. Dodgy businesses (The Bad Guys)
Flashy facade, flashy furniture, nothing behind it. Quick to build. Trying to imitate a real business. They only need visitors to milk. “Sucker born every minute.” Low cost setup, no community connection, no long term reputation to worry about, no legal or ethical concerns. Some of these people are extremely smart.
3. Small to medium businesses (SMEs)
The majority: Nail salons, schools, grocery stores, dentists, indie newspapers etc. Takes time and money to establish. Variety of funding levels, most employ reasonable ethical standards. Generally participate in the community, and have reputations to maintain.
4. Philanthropists, Event organisers, Forums, Community Outreach.
Woodstock meets TED conference meets ‘local park’ meets activist HQ. Takes time and dedication to establish and maintain. Not generally breaking rules, but there is a ‘gray’ side. It’s the stuff of revolutions, community debate, good advice, and useless crap.
5. Hobbyists, Artisans.
That ‘specialist electronics store’, ‘glass blower’ or ‘comic book place’ you like. Similar to 3 but their focus is their craft, driven by love of the subject. There are better ways to make money.
The mechanics of ‘search’ in this city.
The goal was to help interested visitors find what they wanted, while minimising trickery and crime.
Google would send out millions of spidery-robot reviewers (because a task like this had to be automated). They would take a quick look at your building, and perhaps a peek inside. The robot would record the relationships between your business and the community, guessing the type and quality of each building and the service offered.
And most importantly: If other property owners had referred some of their visitors to you, that would work in your favour. I'll refer to this as a ‘citation’.
Google would then make recommendations to city visitors based on this data.
“We can't tell you much about what the robots are looking for, as the bad guys will use that information to trick our robots.” - Google (paraphrased).
If you played dirty, the robot reviewers would catch it and you'd count for little. If you created a great service, and other owners cited you, you'd be rewarded with new visitors.
It wasn't perfect, but at the time it seemed to be mostly working well (from the outside).
2010-ish - The giant makes its first mistake.
The bad guys had begun to take over, imitating real businesses and tricking Google’s robots in more sophisticated ways.
A small team of people behind the scenes made a terrible decision. To go after the bad guys at all cost. To introduce many more ‘penalties’, and to continue to keep everyone in the dark as to what those penalties were.
The ‘War on Spam’ was born.
Overnight, the ‘Spidery-robot reviewers’ became the ‘Spidery-Techno-Active Secret Inspectors’. Let’s call them ‘STASI’ for short. They had almost complete control over city visitor traffic.
You'll never guess what happened next. Hint: Nothing good.
Pretty soon, property owners were complaining that the robots were attacking them unfairly. Most of these first offences were ‘bad air-conditioning plumbing, or rusty door handles’. We all sympathized, but agreed it was in everybody’s interest that the changes be made.
The desperate cries continued, but a few strange things had begun to happen. Properties that didn't even install air conditioners were being mis-identified by the robots, and punished for not having the correct plumbing. With no-one to complain to, many legitimate owners flooded to the ‘public help centres’ and were greeted with:
“Perhaps you should just do the plumbing anyway, we're not sure if the ‘air conditioner plumbing’ penalty is real, but it might be. It’s probably that.” — “But I don't use any air conditioning?!”
Frustrating stuff, especially considering smaller business have less resources to survive unexpected problems.
‘Penalty analysts’ began to charge high fees (often 5,000 euros a month) to interpret what these robot inspectors were looking for, how it might affect a property, and how they could help. It was all speculation. Sometimes effective, other times leading to a horrific penalty in itself.
The bad guys got hit, adjusted their strategy, set up a new shop, and carried on. The big guys paid for the best analysts. Everyone else hung on.
2011-2013-ish. Then it got worse.
Collateral damage, and the loss of many smaller owners.
The citations you had received from other services began to be called into question. The bad guys had been setting up, or breaking into, other businesses to give themselves these references. The whole thing had become a mess (obviously).
The solution: ‘More penalties’.
Suddenly, almost every citation was cause for concern. Since many good services had been misidentified by previous penalties, you were likely to be hurt by a citation from them.
As usual: The bad guys got hit, adjusted their strategy, set up a new shop, and carried on. The big guys paid for the best analysts. Others hung on.
The solution to these ‘minor issues’ was to allow citizens to report violations. This was intended to allow owners to report ‘shady’ businesses in their area. However this endowed struggling owners with the option of tattling on their neighbours ‘plumbing’ situation. Perhaps resulting in a somewhat machiavellian way of being able to harm your competition. It was a sign of more things to come.
I heard this great analogy from another webmaster regarding some of the weirder penalties affecting the early settlers:
“…for many years the speed limit was 30 mph, one day they ‘presumably’ changed it to 20, then trawled over camera archives punishing people in retrospect.”
During this time the inspectors simultaneously began looking for ‘exciting’ new things to identify quality services. Some of these new ideas were strangely irrelevant to certain groups. eg: Installing a big window sign with your ‘law degree’, was not relevant for the folks running a gym.
Closed doors and the ‘War on Spam’.
Getting help now required subjecting yourself to a soul destroying litany of abuse.
Even with legitimate concerns, you were required to shout out your case in the public forums alongside criminals, the lost and the strangely dodgy. Unable to find your voice, you'd eventually be shunned as ‘yet-another-person-whining’.
The spirit of the community was squashed against the steel face of robotic inspection, and an apparent lack of official concern. Philanthropists, artisans, small businesses and middle class, mostly unable to afford analysts were being permanently lost from the city. The decreasing visitor numbers made any hope of even ‘paying’ for recovery an impossibility. Some turned to crime.
The ‘War on Spam’ had created more of what it fought against, the city was saturated with analysts, and a wholly toxic environment for true innovation was emerging.
And as usual: The bad guys got hit, adjusted their strategy, set up a new shop, and carried on. The big guys paid for the best analysts. Some failed to hang on.
It turned out there was not much harming big business, they had access to a lot of ‘counter balance’. Bad guys easily bounced back, having no real community ties and very little to lose they simply ‘churned and burned’ until something worked.
That city, over time:
We start with philanthropy, community, small business, big business.
A little ‘heavy-handed secret policing’ later… (those little guys are ‘analysts’)
“Nature abhors a vacuum”: As time passes holes are filled.
As this process is repeated it tends to favour ‘Brands’, ‘Bad guys’ and ‘Analysts’ - filling the city disproportionality with those groups.
Your local nail-salon is less likely to survive.
I'll end the analogy here. That’s what we see, and that’s what we think you'll find more of as you search. This is not the ‘liberation of information’ so many envisioned.
A reminder: I'm not saying the above situation is everyone’s situation; such a declaration would be absurd. But it is the reality for a great many who ask only to compete fairly, and on tangible merits.
“If you remember [web browser] Netscape, people thought, oh the web is great but here’s a completely controlling web company, what are we going to do? Then one morning they weren't worried about Netscape any more, it was Microsoft. Then suddenly, wait a moment the browser wasn't the issue, it was the search engine. Then, it’s wait a moment, it’s the social network.
If you look at it broadly, yes a monopoly slows innovation, reduces competition. That’s why it’s important this is an open platform. But monopolies come and go all the time.”
Tim Berners-Lee (Inventor of the World-Wide-Web)
Are Google being evil?
No, they’re being a really big corporation. They're not listening to the little people; how could we have expected anything else? We were so naive.
The size and scope of their power makes this a remarkably poignant (yet misunderstood) issue. We gave them a huge part of the ‘free media’ with few checks or balances.
Google may be creating a terribly filtered bubble for us to gaze at.
A temporary solution.
Better than nothing.
We need open sharing of appropriate internal statistics, to allow for better analysis and auditing of the global impact all these changes make. Specifically how they impact the business, philanthropic and informational communities. Currently Google participate in no public auditing or data transparency. They are free to do as they see fit.
“… 500+ algo launches/year mean 1-2 a day.“ - Head of Webspam via Twitter.
We need an official channel to debate and question changes . The help forums they supply seem purposely ineffective - with (to the best of my knowledge) only a single staff member available to answer the occasional question. The digital equivalent of the middle finger, considering the number of people affected.
There is one other person (‘The Head of Webspam’) who posts YouTube videos, and answers questions at his own discretion.
They are in contact with certain popular bloggers and analysts in the field; but the average ‘Joe’ can't get a real foot in that door. Though if you're lucky you can get a reply via Twitter - which is nice.
This is how a multi-billion dollar company, with influence over millions of peoples lives, and the health and future of the internet should be expected to behave. It’s not a game anymore.
We continue to hand over our infrastructure.
These are not trivial concerns, these are the tools and services that will shape Earth’s future.
We're currently handing them the network infrastructure itself (Google Fibre - awesome!) and some may be giving over their day-to-day lives (Google Glass - I want it too), while browsing for what’s next (Google Chrome - big fan!) using Android (a humanoid robot).
When the NSA takes its afternoon nap, it dreams of being Google.
During this decade we'll see developers create much needed decentralised (possibly peer-to-peer) applications. Until then:
If you appreciate the need for diversity in the search results you see, consider alternatives where you can.
Disclaimer: My current website handles over 140k visitors a day, the large majority of that from Google. I am first in almost all positions for my relevant sector. I have lived and breathed Google traffic for almost 15 years now. I don’t ‘hate’ Google, I don’t ‘hate’ SEO. I simply believe this monopoly is bad for the internet as a whole.
Relevant reading:
WebmasterWorld thread that inspired this. (Technical and lengthy)
