Evaluating Threat-Blocking DNS Providers

Ask any CISO and they will tell you that one of the biggest challenges they face is measuring the ROI of a given security product or service. It is very difficult, if not impossible, for a customer to understand a product’s coverage and quantify the protection that they are paying a vendor for.

Defense and risk mitigation should be a technical exercise (either a security control is effective or it isn’t), but it has become a political exercise instead. CISOs look to their peers and security vendors to get a feel for how much money they should be spending, and on which products, without measuring their efficacy or understanding the gaps in their visibility.

In-turn, the security controls of many organizations aren’t effective, and when a technical adversary (whose attack is successful or it isn’t) compromises their environment they remark that the adversary was sophisticated.


Attacker economics and market forces are already shifting defenders from taking political measures to technical ones to protect their networks. To get ahead of this trend, you can evaluate your countermeasures to understand their shortcomings. This post focuses on threat-blocking DNS providers, but let me first talk about some previous work I did to assess security vendors.

Evaluating Penetration Testing Vendors

A number of years ago, I oversaw a program within the financial services sector called Sentinel. We set up networks of vulnerable systems and evaluated penetration testing vendors to understand their strengths and weaknesses. During one test involving 10 vendors, we found that:

  • Two failed to scan all 65,536 TCP ports
  • Five failed to report the MySQL service root password of “password”

The main takeaway from the test was, for a given bank to consistently identify all of the high severity issues across their mission critical technologies (e.g. Oracle Database, PostgreSQL, and Microsoft SharePoint), they would have to retain a handful of testing vendors — as no single one provided sufficient coverage. Before evaluating and actually measuring their efficacy, the banks had assumed that the vendors were generally good.

Evaluating Threat-Blocking DNS Providers

In recent months, I’ve spent time with security teams who swear by the coverage that Cisco Umbrella provides them, and others that prefer Quad9. Cisco mentions threat protection like no other against C2 callbacks and phishing attack vectors on their sign-up page, and Quad9 ingests threat intelligence material from IBM and other partners, as shown below.

There exist many other threat-blocking DNS services, including those operated by Comodo, Neustar, Norton, SafeDNS, Safesurfer, WatchGuard, and Yandex. Usefully, for security teams wishing to evaluate these vendors, CryptoAUSTRALIA published a utility called DiNgoeS which resolves domains against threat-blocking DNS servers to provide deep insight into the individual threats that are blocked by each.

Preparing a list of malicious destinations

DiNgoeS uses the hpHosts malicious domain lists by default. While the hpHosts phishing list is decent, many C2 domains are missing from the others. To evaluate threat-blocking DNS service coverage, we combined our own threat intelligence with open source materials to create a list of malicious domains for the week of 30 April 2018, including:

Traffic to each of these 140 FQDNs indicates severe infection, or a user clicking a live phishing link. We would expect threat-blocking DNS service providers to protect their users from many of these threats.

As mentioned, the DiNgoeS utility pulls phishing, malware distribution, and exploit site domains from hpHosts by default. We patched the software so that it would process the list of domains we’d prepared, and executed it.

C2 blocking statistics

The DiNgoeS blocking statistics for the 100 C2 domains are found below. Neustar, SafeDNS, Safesurfer, and WatchGuard DNSWatch performed well, providing around 50% coverage. We found that Quad9 underperformed by blocking only 24% of the requests.

DiNgoeS blocking statistics for 100 C2 domains

The Cisco Umbrella public resolvers blocked two of the C2 domains pulled from CyberCrime-Tracker, and did not block any of the DNS requests associated with Hancitor, ZeuS, Cobalt Strike, or the malicious domains observed in AlphaSOC customer environments last week.

The raw DiNgoeS C2 test results are available here >

Phishing blocking statistics

Next, we ran a test using the shorter list of live phishing sites (blocking statistics found below). WatchGuard performed well and clearly tracks phishing domains, and both Neustar and Quad9 provide some coverage.

DiNgoeS blocking statistics for 40 phishing domains

The raw DiNgoeS phishing test results are available here >

Winners and losers

The combined coverage statistics across the C2 and phishing sites are listed below. There is an unexpected victor in WatchGuard, and surprises around Cisco Umbrella and Quad9 coverage considering their pedigree.

  • WatchGuard DNSWatch — 61%
  • Neustar Free Recursive DNS — 40%
  • SafeDNS — 39%
  • Safesurfer — 34%
  • Quad9 — 27%
  • Comodo SecureDNS — 5%
  • Norton ConnectSafe — 5%
  • Cisco Umbrella — 2%
Update: Dan Hubbard (ex-CTO of OpenDNS) has been in-touch and reports a higher coverage figure when using the paid version of Cisco Umbrella. We’re working to test the commercial version and will publish the results.

Patching the Holes and Increasing Coverage

If you do use a threat-blocking DNS provider to protect your users from malware and phishing threats, this research indicates that around half of the malicious requests will go unblocked at best. These services offer a good base level of protection from which you can build upon, but are by no means a silver bullet.

Google SafeBrowsing

We took a look at the Google SafeBrowsing API during this test and it provided no coverage on the C2 side, but did block 28% of the phishing sites (outperforming all but WatchGuard, Quad9, and Neustar). Moving users to Google Chrome therefore provides some phishing protection.

DNS Analytics

Many security teams use DNS Analytics for Splunk to flag emerging threats, phishing attacks, and anomalies. If you are indexing DNS logs within Splunk, you can use the app to uncover callbacks to the C2 destinations found here, and solve a number of use cases that go beyond basic correlation with threat indicators, including:

  • DNS tunneling and data exfiltration
  • Phishing attacks using homoglyphs (Unicode and others)
  • Infected hosts using contemporary DGA algorithms
  • Infected hosts beaconing to unknown C2 destinations
  • Suspicious traffic patterns indicating infection (e.g. dynamic DNS events)
Anomalies and emerging threats identified by DNS Analytics for Splunk

Network Flight Recorder is a non-Splunk option which supports scoring of DNS event data from the network or disk using the same analytics engine. NFR supports processing of Bro IDS, Suricata, and other logs, providing visibility and deep scoring of malicious domains without known indicators.

Further Reading and Prior Art

Threat-blocking DNS service providers have been the focus of previous research undertaken by CryptoAUSTRALIA and Nykolas Z (as below). Norton ConnectSafe performed well using the hpHosts dataset, which demonstrates the importance of selecting malicious destinations to evaluate coverage with.