Can you imagine a world without Search?
Twenty-five years ago this week the world’s first search engine was released
September 11th, 2015 marks the 25th Anniversary of an obscure event that changed the world and the lives of many of the 7 billion human beings living on it. You’ve probably never heard my name or even of what I did, and it was an event noticed by very few at the time and arguably none who recognized its larger significance, including myself.
On September 11, 1990 we announced Archie, the world’s first Internet Search Engine.
The Internet of the early 90s would be unrecognizable to most people today. For those relatively few who were using what would be now considered a glacially slow network, the primary functions were exchanging email, transferring files and logging on to remote servers. Then funded by the National Science Foundation its purpose was to facilitate research and education and there were strict limits on commercial use, a fact that seems nearly inconceivable now in the age of Amazon, Netflix, Apple and Google.
At the School of Computer Science at McGill University in Montreal, Canada, I was a couple of years into my Master’s Degree but, like so many graduate students my studies had been sidelined by a job, working for my own department as a system administrator. While it involved long hours and low pay, we were compensated by acquiring invaluable experience and having access to latest “toys” in the field.
As we had little funding to purchase software we took advantage of the fledgling Open Source community, programs that were written by individuals and groups and freely distributed for all to use. Through happenstance, it fell to me to locate and retrieve desired packages for the use of the faculty and staff. Like all good lazy programmers I automated the process and built a rudimentary database of the available files on hundreds of repositories sprinkled around the network, known as “archive” servers. One day my boss and fellow grad student Peter J. Deutsch asked me to locate some files that a user in an online forum was searching for, I quickly got the answers and he posted the results online.
We were soon inundated with many similar requests and it became clear to us that there was a large unmet demand for this kind of service: a central “card catalog” was a missing piece of the puzzle. So with the help fellow syadmins Bill Heelan and Mike “der Mouse” Parker, we built a public interface to the database which together became the Internet’s first search engine. It was only then, in a moment of slight panic we realized that we had no name for this new creation. Since it worked with archives, I suggested “archie” — “archive” without the “v”. (The capital-A “Archie” only came after we trademarked the name years later.)
From there things grew by leaps and bounds and soon the server Peter had commandeered for Archie was the destination for fully half of all the network traffic to Eastern Canada.
The early ‘90s were the Internet’s “Cambrian Explosion” and “Wild West” rolled into one: a frenetic period of novelty, creativity and exploration, with no precedents on how we should proceed. It had been spreading since its creation twenty years earlier and now with hardware and telecommunication costs rapidly decreasing and significant improvements in network speed, the floodgates of innovation were opened.
In 1991 came Brewster Kahle’s Wide Area Information Server (WAIS) from Thinking Machines Corporation, Mark McCahill’s Gopher from the University of Minnesota and Tim Berners-Lee’s World Wide Web from CERN in Geneva, all designed to provide human-friendly ways to browse and access information on the Internet — at least by the standards of the day.
Search engines for those systems soon followed. The first for Gopher went by the very tongue-in-cheek name of “Very Easy Rodent-Oriented Net-wide Index to Computer Archives”, or Veronica which some will recognize as the girlfriend of Archie, the Riverdale High comic strip character. In 1993, the first search engine for the World Wide Web was created and named ALIWEB — “Archie Like Indexing for the WEB”.
These were exciting times and we all knew we were working on something big, though I suspect few of us had any idea just how big. For me the environment was wholly egalitarian and the fact that I was young gay man of color, uniquely among this group of pioneers, was simultaneously embraced and irrelevant to the task at hand.
Under the auspices of the Internet Engineering Task force (IETF), the standard-setting organization of the network, a small group of us worked together to create and shape the information infrastructure that all of you are using right now to read this.
In 1992 with McGill’s help and encouragement, Peter and I split off from the university and formed Bunyip Information Systems, Inc. While there were several companies that had been created to provide Internet connectivity at the time, we believe Bunyip was the world’s first example of what would now be considered a software “Internet Startup”.
Though it may seem counter-intuitive in the current hyper-commercial era, we purposely didn’t patent the algorithms underlying Archie, algorithms that every search engine now use. We strongly believed that we were working in a nascent field whose development could be significantly held back if everyone started trying to protect their own intellectual property, locking up ideas behind licenses and limiting its availability. Thankfully, in the same spirit CERN in 1993 generously released the World Wide Web software into the public domain, a move that allowed it to be adopted, modified and deployed throughout the network in short order.
I have no regrets about those decisions. Being a small part of one of the greatest revolutions in the history of humanity has its own rewards and, of course, being first is no guarantee of either great success or wealth. The history of technology is littered with financially unsuccessful “firsts”.
In just over 20 years we have gone from nothing, through several generations of software engineering, to an industry worth hundreds of billions of dollars. Search now encompasses anything we can digitize, not just documents but images, maps, video, music, geological, climate and astronomical data and much, much more. And now, as we enter the second quarter century of Internet search it is becoming broader, deeper and more sophisticated — but many challenges remain.
But at the root I believe that the fundamental aim of search must be to return the most relevant results to your queries, as free from inappropriate bias as possible.
This can be subverted by those sites who would try to manipulate their search results for nefarious means such as spam, fraud or malware. More insidiously it can be done by the search engines themselves by promoting their own products over less revenue-generating links — a self-serving practice that needs to be vigorously policed. These engines are also poised to incorporate knowledge of your past queries and browsing habits which exposes the danger of them omitting results they consider irrelevant to you — all without your knowledge. These kind of blinkered results can produce a kind of “tunnel vision”, limiting the things you get to see and the serendipity that comes with it.
Controversial things. Unpopular things. Things you may not agree with. And of course, even more significantly as we see in places like China, things governments do not agree with.
And with search engines being the starting point for our exploration from everything from recipes to airline schedules even subtle modifications to the results, whether intentional or malicious, could have enormous real-world consequences. It may even affect our most fundamental democratic processes as one recent study suggested: Google, Bing and the others potentially have the power to swing elections of the local dog catcher to judges to the presidency itself. And all of this could be done in a way that would be extremely difficult to detect without access to the highly proprietary search algorithms — something unlikely to be willingly made available.
Another recent challenge is the so-called “Right to be Forgotten” enshrined in European Union law which has noble aims but I believe to be deeply misguided. Yes it is a complex topic that evokes much passion but the negative consequences to free speech and transparency will likely be significant when this form of “selective amnesia” is mandated. Search engines are often the sole means of discovering information and claims that requiring them to remove references to particular online information are not a pernicious form of censorship are disingenuous. Recent attempts by authorities to apply this “right” to search engine indexes worldwide, not just within the EU, are absurd and would effectively extend this flawed policy to the entire planet.
Many have talked about search coming to the “deep Web” where every kind of information, criminal and court records, property transactions, fishing licenses and gun registrations will all be returned in results as easily as today’s news. How will the conflicts with privacy be resolved? How will governments and corporations react? How far will it go?
Though I have been out of the field of Internet Search for many years, I continue to follow it with interest. The sophistication of current and proposed search technologies is nothing short of amazing and I am constantly reminded of “third law” of science fiction author Arthur C Clarke: “Any sufficiently advanced technology is indistinguishable from magic”.
Search democratizes and facilitates our path to knowledge. We truly live in an age of magic where we can access nearly the sum total of knowledge of humanity from a device that fits in our pocket. That’s something that even to us who were there in the beginning of this journey in 1990 would never have imagined.
Whether in real life or in the digital world we understand that search enables us to exist — be it in the quest for food, shelter or a mate.
Search enables Life.
