Unlocking the entire web

The BBC asks “What can you use instead of Google and Facebook?”, this is what AMPLYFI thinks

Louise O’Reilly
AMPLYFI
4 min readMar 23, 2020

--

Photo by NASA on Unsplash

The internet as most people know it currently contains around 6.6 billion websites. This effectively represents the total amount that standard search engines access. However, just as we now know that the universe is made up primarily of stuff we can’t see — so-called dark matter or worlds beyond the range of our current technology — so the internet is actually predominantly made up of an area called the Deep Web, a massive repository that sits beyond the range of standard search engines. These search engines, such as Google, Yahoo! and Bing operate in what is known as the Surface Web, an area estimated to be less than 1% of the entire internet. This really is just the tip of the iceberg, delve deeper and you’ll find the Deep Web, some 400–500 times larger and containing a wealth of untapped information.

Not to be confused with the infamous Dark Web, the Deep Web is fundamentally benign. It is a vast knowledge pool that most people are unaware of and have very limited access to. Some Deep Web content can sit behind paywalls that require log-on details to access. The majority, however, such as academic papers, scientific reports, patents, government resources, organisation-specific repositories etc. exists as open-source content that is found by entering queries into site-specific search boxes (e.g. searching for a show on Netflix). Consequently, they will not be indexed and returned by standard search engines. Whilst the Deep Web may sound daunting, you will have visited it on various occasions i.e. searching a news outlet like the Financial Times or purchasing train tickets online.

(Image source: https://www.deepwebtech.com/2016/08/google-just-gets-to-the-tip-of-the-iceberg/)

Consequently, by using standard search engines there is a wealth of information that many web users are missing out on. These tools barely scratch the surface of the vast sea of content that lies underneath. Of course, for simple, every-day searches that we all perform, for example quick direct questions of your local shop’s closing time or finding out a fact like the age of the oldest person, standard search engines are invaluable tools. However, for in-depth research, such as trying to uncover trends and links, these platforms are highly limited and leave users having to make multiple searches, going through the process of selecting, reading, and screening through pages of results.

This is where next-generation search engines such as AMPLYFI’s DeepResearch take internet queries to the next level. It is the world’s first real-time search engine to unlock both the Surface and Deep Web. Rather than simply skimming the surface, it utilises Artificial Intelligence (AI) algorithms to crawl and access the Deep Web in order to unlock its hidden value. For example, if you conduct a search on Boris Johnson on a standard search engine you will be returned with millions of results (i.e. on Google for this search it returned an estimated ‘229 million results’). However, this is merely an estimation of what might exist across the entire internet. Page to the end of the search results and you will see there are only around 300 actual results, massively less than the estimation promoted. And that’s before the wall of personalised advertisements and Search Engine Optimisations restrict your query, creating a problematic filter bubble where you are simply not seeing the most relevant results.

AMPYFI’s DeepResearch topic wheel

DeepResearch allows you to see the results you would otherwise have got using a standard search engine. In less than a minute and by going to the Deep Web, it is able to return 12,580 results for the same search. In doing so, it immediately outperforms standard search engines. Our AI algorithms then analyse every single site to distil key terms and topics that are then clustered into a Topic Wheel. Clicking within the wheel allows you to filter the search results to find exactly what you’re looking for, as well as discover unknown connections and information. The results can also be filtered and screened by document types such as academic, news, government, finance, patents etc. delivering a new level of depth to your analysis.

AMPLYFI’s tools enable clients to not only unlock the entire internet but access open source documents, forecast emerging trends or disruptions, establish correlations and related topics, and then make better decisions driven off unbiased machine-driven analysis. Find out more at www.amplyfi.com.

--

--