Scratching the surface: What you should know about the deep and dark web

Welcome to Threat Intel’s #WednesdayWisdom column, which aims to help improve your cybersecurity knowledge and keep you informed on important developments.

When most of us think of the web we think of sites like Google, Facebook, Twitter, and this one. We think of doing some internet shopping, and finding out what’s happening in the news.

However, what many people may not realize is that the web that most people use — the web that is accessed through search engines like Google and Bing — is only a tiny percentage of the “web” that’s out there.

Known as the “surface web”, “indexed web”, and “visible web”, the current estimated size of the surface web, according to one source, is close to 5 billion pages.

However, the surface web is believed to make up only a tiny percentage of the World Wide Web, with much of it existing on the “deep web” and the “dark web”.

No one really knows how big the deep web is, but it is certainly larger than the surface web, with some estimates saying it may be 500 times as large as the searchable web. According to a study published in Nature in 2015, Google indexes just 16 percent of the surface web, and none of the deep web, meaning that a Google search will show up less than 1 percent of the information that exists online.

The analogy of an iceberg is often used to describe the difference between the surface web and the deep web — we can see a small part of it on the surface, but a lot more lies beneath.

Much like an iceberg, only a small part of the world wide web is visible on the surface

Deep web vs dark web

While the terms are often used interchangeably, the deep web and the dark web are not the same thing.

The deep web refers to the parts of the World Wide Web that are not indexed by search engines. It includes things like databases, email services, online banking services, and other services protected by paywalls or passwords.

The dark web is a part — a relatively small part — of the deep web. Websites on the dark web are not accessible through normal browsers and can only be accessed through special software such as TOR or I2P (Invisible Internet Protocol). The appeal of the dark web is its anonymity — it has a high level of encryption that means it is difficult to identify either those visiting websites or hosting websites on the dark web.

How dark web encryption works

TOR and similar services operate by bouncing your traffic around the web so that your ISP does not know what sites you visit, and the sites you visit do not know your physical location. TOR users do not connect directly to the website or service they want to visit. Instead, they bounce through a series of nodes on the network. Each node only knows the data it receives and the node to which it is passing the data — so while the initial node knows it is being connected to by, for instance, Susan’s computer, it doesn’t know which website Susan is planning to visit, as Susan’s connection will be bounced through further nodes before connecting to its final destination — the site Susan wants to visit. This website will only know the final node that connected to it, it will not know that Susan’s computer was the one that originally sent the request to connect to the service.

This complex encryption model makes it very difficult to trace back who has visited a site using the dark web, guaranteeing users a level on anonymity they do not have on the surface web.

What is the dark web used for?

The name of the dark web has been blackened, particularly recently, with the perception existing among many people that the dark web is exclusively used by criminals, and for illicit purposes such as the sale and distribution of drugs, hacking tools, and child pornography.

Certainly, some of the activities that take place on the dark web are illegal. Symantec’s Internet Security Threat Report, which was released earlier this year, details the items and services our researchers found for sale on dark web marketplaces. It includes things like credit cards, passports, and gift cards, as well as ransomware toolkits and banking Trojans.

The prices for various services on dark marketplaces (as observed by Symantec researchers)

The jailing for life of Silk Road founder Ross Ulbricht, and the takedowns of dark web marketplaces AlphaBay and Hansa, which were used to sell drugs, guns, and other illicit items, served to put the dark web in the spotlight for the wrong reasons recently.

However, while its illicit uses are what garner attention, the dark web is not, in fact, solely used by criminals or those looking to make illegal purchases.

Light in the dark

It is difficult to say what percentage of dark web sites are used for illegal activities: estimates vary widely, and the nature of dark web sites, which can often disappear suddenly, mean it is difficult to put a figure on it.

However, despite its current reputation, the origins of TOR actually lie with the U.S. government, which funded its creation as a way to protect and anonymize communications from U.S. intelligence agents.

And while it cannot be denied that this dark web technology is used for nefarious activities, it is also used by human rights activists, journalists, and others who may be exchanging sensitive information on the web. It is also used to bypass restrictions in places like China, where sites like Facebook and Twitter cannot be accessed on the surface web. However, countries like China and Russia are cracking down on the use of anonymizing software like TOR and the use of virtual private networks (VPNs), and have made recent attempts to make them illegal.

Some people also use TOR software to simply increase their privacy and add a layer of anonymity to their regular browsing habits. With the introduction of new laws, such as the law in the U.S. allowing ISPs to sell information about your browsing history, TOR may become more popular among “ordinary” people.

Much like the surface web, the dark web is a place of pros and cons, with some people using it to stand up to repressive regimes, and others for illegal activities. Whether or not the pros of services like TOR outweigh the cons is likely something that will be debated for some time to come.

Check out the Security Response blog and follow Threat Intel on Twitter to keep up-to-date with the latest happenings in the world of threat intelligence and cybersecurity.

Like this story? Recommend it by hitting the heart button so others on Medium see it, and follow Threat Intel on Medium for more great content.

One clap, two clap, three clap, forty?

By clapping more or less, you can signal to us which stories really stand out.