History of Search Engines (1945–1993)

Harsh Parikh
Dec 25, 2018 · 5 min read

It’s easy to forget that just a little over 15 years ago, the web looked like this. A list of links that was maintained by humans. Finding information that you wanted was a difficult process, and usually involved clicking link after link, hoping you would arrive at the right place.

Earlier View of Yahoo Search Engine

Many say the idea of the Internet was conceived in 1945, after an engineer, Vannevar Bush, wrote an article for Time titled As We May Think. In this article, Bush pushed top scientists at the time to create a virtually limitless, fast,reliable, extensible associative memory storage and retrieval system. Bush realized that technology was advancing rapidly, and with it the human race would need a way to store, catalog, and easily access the information we were accumulating.

In the 1960’s Gerard Salton, often considered the father of modern search technology, created the idea of a search engine, and developed the information retrieval system called SMART, which stood for Salton’s Magic Automatic Retriever of Text. He authored a book called A Theory of Indexing , which details many of his initial tests that search is still largely based off of, including concepts like statistical weighting, relevancy algorithms, and more.

Around the same time, Ted Nelson created a project of his own and named it Project Xanadu, which had the goal of being a computer network with a simple user interface that could solve a variety of social problems, such as attribution. Ted coined the term Hypertext, and was against complex markup code, broken links, and other problems typically associated with traditional HTML and the World Wide Web. Much of the inspiration to create the World Wide Web was actually drawn from Ted’s work.

In 1969, a service called ARPANET was born.This was created by ARPA, which stands for the Advanced Research Projects Agency Network, and is part of the US Department of Defense. ARPANET was a secure and fast computer network that allowed the transmission of information across long distances.The service utilizes rented telephone lines to convey military intelligence. It’s pretty safe to say that without the creation of ARPANET, the internet as we know it today would not exist.

In 1990 the first search engine was created. This search engine was known as Archie, which was short for archives. Archie was able to retrieve files from a database by matching a user query using regular expressions. Alan also created a template indexing method that would allow Archie to indexfreely available or public domain documents, images, sounds, and services on the network.

Archie didn’t use keywords to find related documents like modern search engines do. To use Archie effectively, one had to know the name of the file you were looking for, as Archie didn’t index the content of the files, only the titles. By 1992 Archie contained about 2.6 million files, and its service processed approximately 50,000 queries per day, generated from thousands of users around the world. As Archie’s popularity grew, two similar search engines, Veronica and Jughead were created, with the purpose of indexing plain text files.

In 1991, Tim Berners-Lee, an independent contractor at CERN, had created the World Wide Web. The World Wide Web was created based on a concept of hyper-text to facilitate sharing and updating information among researchers. In 1991 the first web site was built and put online. It provided an explanation about what the World Wide Web was, and how one could set up a web server and own a browser. This also allows you to simulate viewing the website through an old line mode browser.

In 1993, the first robot or spider was created. This bot was called the World Wide Web Wanderer, and was created as a way to measure the growth of the web.The bot was soon upgraded to capture active URLs and store them in a database called WANDEX. But the wanderer soon became more of a problem than a solution. It crawled websites and accessed pages hundreds of times a day, creating a lot of server lag, and at times causing websites to crash. This created a lot of distrust in robots among webmasters and the general public.

In response to the wanderer, ALIWEB was created. ALIWEB stood for Archie-like indexing of the Web and crawled meta information of pages.To avoid issues of bots, ALIWEB allowed owners to submit their site to be included in the search index, along with the description of their webpage. Due to this, bots were not needed to collect the data, so bandwidth issues are no longer a concern. The downside is that many people did not know they had to submit their site, or how to do so.

You should now have an understanding of why search engines were created, and how they have evolved over time. While very different than modern day search engines like Google, these early search engines still have a lot of similarities to how search engines process and analyze data today.