Open in app

Sign in

Write

Sign in

Building a secure DNS infrastructure like SecureDNS.eu

Rick Lahaye
15 min readApr 28, 2020

As can be seen on my LinkedIn, SecureDNS will shutdown as of 30th of April 2020. Many people have asked me if they could buy SecureDNS, or get my help to build one alike. This technical write-up is for those people. For it to make sense, its best if you checkout my Github SecureDNS repository as well

SecureDNS
SecureDNS Diagram

Introduction

The infrastructure exists out of the following components:

  • Front-end Loadbalancer
  • Back-end DNS services

As front-end loadbalancer/reverse proxy it uses HaProxy. HaProxy handles the TLS part, and the SNI routing to the correct service.

As back-end DNS services it uses the following:

Not all components or configured options will be described in detail, as some are straight forward or just best-practice. If you do want to know, check the RFC.

Front-end Loadbalancer HaProxy

HaProxy is a TCP/HTTP loadbalancer with many configurable features. Only disadvantage of this loadbalancer is that it has no support for UDP. This shouldn’t be a problem with DNS over TLS or DNS over HTTPS, though with DNS over QUIC (DoQ) or DNSCrypt it will.

This loadbalancer is amazing as it has support for SNI routing. The advantage is that you can host multiple services on the same port, and use routing based on the SNI extension in TLS. This ensures for example that one can serve DNS over TLS, DNS over HTTPS, and a webserver on port 443!

Configuration

Below sections contain the content of haproxy.conf. It is split up in the following sections:

  • global and defaults
  • front-end
  • back-end

Global and defaults

global
  daemon
  user haproxy
  group haproxy
  maxconn 55000
  external-check
  tune.ssl.default-dh-param 2048
  tune.ssl.cachesize 30000
  tune.ssl.lifetime 600
  ssl-default-bind-options no-sslv3
  nbproc 1
  nbthread 4
  cpu-map auto:1/1-4 0-3
  log /dev/log local0 noticedefaults
  timeout http-request 10s
  timeout queue 30s
  timeout check 3s
  timeout connect 5000ms
  timeout client 30000ms
  timeout server 15000ms
  balance source
  log global

The maxconn has been set to 55000 which is above the default to be able to handle more connections. Also tune.ssl.cachesize and tune.ssl.lifetime are set to decrease the CPU overhead from TLS handshakes.

As this server has 4 logical cores, we would like to use multi-threading. Therefore, it has set nbproc to 1, to ensure 1 master process, and nbthread to 4 to ensure 4 working threads. Next we assign each thread to 1 logical core by setting cpu-map to auto:1/1–4 0–3.

Next I have used log to ensure that notice messages are sent to systemd-journal (as journal listens to /dev/log). This is usefull to keep track of online and offline backends.

As last, I have set the external-check parameter to enable the feature for using custom health checks to validate if a back-end is online or offline. By default when the parameter check is used in a back-end, HaProxy will do a TCP handshake to validate if the service is listening on the back-end node. By using this feature, you can use Bash scripts with exit codes for health checks.

The default section contains all defaults for the front-ends and back-ends. Advantage is that if you add them to default, you only have to add them once, in contrary with noting them in the front or back-end section itself.

Front-ends
The front-end sections are the configuration that are public internet facing, and is where users connect to. These front-ends obviously have a relation to the back-end sections.

It uses 3 different front-end sections:

  • Port 80: serves HTTP for webservers
  • Port 443/TLS: serves HTTPS for webservers, and DNS over HTTPS (and if you would want to you can also use this for DNS over TLS)
  • Port 853: servers DNS over TLS
frontend http
  bind :::80 v4v6 tfo
  mode http
  maxconn 100
  redirect scheme https code 301 if !{ ssl_fc }
  acl securedns.eu hdr(host) -i securedns.eu
  acl test.securedns.eu hdr(host) -i test.securedns.eu
  acl ads-test.securedns.eu hdr(host) -i ads-test.securedns.eu
  use_backend http-securedns.eu if securedns.eu
  use_backend http-test.securedns.eu if test.securedns.eu
  use_backend http-ads-test.securedns.eu if ads-test.securedns.eu
  default_backend http-securedns.eufrontend https
  bind :::443 v4v6 ssl tfo crt /etc/letsencrypt/live/securedns.eu-0001/haproxy.pem alpn h2,http/1/1
  mode http
  maxconn 20000
  acl securedns.eu ssl_fc_sni securedns.eu
  acl doh.securedns.eu ssl_fc_sni doh.securedns.eu
  acl ads-doh.securedns.eu ssl_fc_sni ads-doh.securedns.eu
  acl test.securedns.eu ssl_fc_sni test.securedns.eu
  acl ads-test.securedns.eu ssl_fc_sni ads-test.securedns.eu
  use_backend http-securedns.eu if securedns.eu
  use_backend dns-doh.securedns.eu if doh.securedns.eu
  use_backend dns-ads-doh.securedns.eu if ads-doh.securedns.eu
  use_backend http-test.securedns.eu if test.securedns.eu
  use_backend http-ads-test.securedns.eu if ads-test.securedns.eu
  default_backend http-securedns.eufrontend dot
  bind :::853 v4v6 ssl tfo crt /etc/letsencrypt/live/securedns.eu-0001/haproxy.pem
  mode tcp
  maxconn 30000
  acl dot.securedns.eu ssl_fc_sni dot.securedns.eu
  acl ads-dot.securedns.eu ssl_fc_sni ads-dot.securedns.eu
  use_backend dns-dot.securedns.eu if dot.securedns.eu
  use_backend dns-ads-dot.securedns.eu if ads-dot.securedns.eu
  default_backend dns-dot.securedns.eu

It uses the bind parameter to configure the following:

  • :::443 v4v6: to listen on any interface on port 443
  • ssl: only accept TLS connections
  • tfo: use TCP Fast Open (sets kind of a cookie to enable faster handshake when connecting again)
  • crt: set certificate (take note that that 1 file must contain the cert, CA chain, and the private key. So if you are using LetsEncrypt you need to combine these). You can use multiple crt statements on this line to let HaProxy serve multiple certs for different domains.

We use the mode parameter to set the front-end to a HTTP or TCP proxy. Keep in mind that if you set it to HTTP, and the traffic is not HTTP, it won’t be routed. Using HTTP has the advantage to make the loadbalancer more aware of the traffic, in which some optimizations can be used for HTTP traffic (e.g. forwarded for headers etc.)

Next it has the maxconn set above the default to be able to serve more connections, and we will be configuring ACL to route traffic based on SNI. It is kind of based on the following conditions:

  • if SNI is X, use ACL Y
  • if ACL is Y, use backend Z
  • if ACL is none, use default_backend

Back-ends

The back-ends are a relationship to the front-ends. Depending on the ACL list, one of the back-ends will be used to proxy the incoming connection to.

It uses the following backends:

  • Port 8080: this hosts the https://securedns.eu website
  • Port 8081: this hosts the https://test.securedns.eu website that is used to test if users are using SecureDNS
  • Port 8082: this hosts the https://ads-test.securedns.eu website that is used to test if users are using SecureDNS with Adblocker
  • Port 53: this hosts the DNS server which is used for DNS over TLS
  • Port 54: this hosts the DNS server which is used for DNS over TLS with Adblocker
  • Port 8053: this hosts the DNS over HTTPS server (which uses port 53 as upstream)
  • Port 8054: this hosts the DNS oer HTTPS with Adblocker (which uses port 54 as upstream)
  • Port 9000: this hosts the HaProxy Stats website for internal use
backend http-securedns.eu
  mode http
  server nginx 127.0.0.1:8080 check proto h2 source 0.0.0.0:1025-65535backend http-test.securedns.eu
  mode http
  server nginx 127.0.0.1:8081 check proto h2 source 0.0.0.0:1025-65535backend http-ads-test.securedns.eu
  mode http
  server nginx 127.0.0.1:8082 check proto h2 source 0.0.0.0:1024-65535backend dns-dot.securedns.eu
  mode tcp
  server dns0 127.0.0.1:53 check weight 1 source 0.0.0.0:1025-65535
  server dns1 10.129.15.242:53 check weight 4 source 0.0.0.0:1025-65535
  server dns2 10.129.16.228:53 check weight 4 source 0.0.0.0:1025-65535backend dns-ads-dot.securedns.eu
  mode tcp
  server dns0 127.0.0.1:54 check weight 1 source 0.0.0.0:1025-65535
  server dns1 10.129.15.242:54 check weight 4 source 0.0.0.0:1025-65535
  server dns2 10.129.16.228:54 check weight 4 source 0.0.0.0:1025-65535backend dns-doh.securedns.eu
  mode http
  option external-check
  external-check path "/bin"
  external-check command /usr/local/etc/haproxy_check_doh.sh
  server dns0 127.0.0.1:8053 check weight 1 source 0.0.0.0:1025-65535
  server dns1 10.129.15.242:8053 check weight 4 source 0.0.0.0:1025-65535
  server dns2 10.129.16.228:8053 check weight 4 source 0.0.0.0:1025-65535backend dns-ads-doh.securedns.eu
  mode http
  option external-check
  external-check path "/bin"
  external-check command /usr/local/etc/haproxy_check_ads_doh.sh
  server dns0 127.0.0.1:8054 check weight 1 source 0.0.0.0:1025-65535
  server dns1 10.129.15.242:8054 check weight 4 source 0.0.0.0:1025-65535
  server dns2 10.129.16.228:8054 check weight 4 source 0.0.0.0:1025-65535listen stats
  # stats != logging
  bind 127.0.0.1:9000
  bind 10.0.2.1:9000
  mode http
  stats enable
  stats uri /haproxy_stats

The server parameter is used to configure a backend node that is able to handle the trafic. Multipe can be configured for loadbalancing purposes. You can use the weight option to configure how much traffic goes to what node. For webservers it has set the option proto h2 to use HTTP/2.

For some back-ends a custom health check has been configured. This is useful for DNS over HTTPS, as each node is running a DNS over HTTPS server that is dependent on another port/plain DNS Named server. If a default heath check was used, the node would stay online if the plain DNS server is offline, as the DNS over HTTPS server still has an open port/is still listening. The content of the custom health check script set by external-check command is as following (HaProxy passes down 5 parameters including the node IP as $3):

#!/bin/bash
real_ip=$3
/bin/nc -vz $real_ip 53
named=$?/bin/nc -vz $real_ip 8053
doh=$?echo $named
echo $doh
if [[ $named == 0 ]] && [[ $named == 0 ]]; then
  exit 0
else
  exit 1
fi

The custom health check basically checks if port 8053 (DNS over HTTPS) is listening on the node, and port 53 (DNS Named) on which the DNS over HTTPS server is dependent as upstream.

Back-end DNS over TLS

The backend for DNS over TLS is a Named/Bind9 instance. The traffic flow goes directly from HaProxy to the DNS server. Configuration can be seen in later section. It has 2 Named/Bind9 instances; 1 main on port 53, and 1 with Adblocker on port 54.

Back-end DNS over HTTPS

As DNS over HTTPS server it uses a DoH server created by m13253. Traffic flows from HaProxy, to the DNS over HTTPS server, to Named.

Configuration

The configuration is very straight forward. It has 2 instances per node; 1 main, and 1 for Adblocker.

listen = [
    "127.0.0.1:8053",
]
cert = ""
key = ""
path = "/dns-query"
upstream = [
    "udp:127.0.0.1:53",
]
timeout = 10
tries = 3
verbose = false

Listens on port 8053 for main, and port 8054 for Adblocker. For the main it will forward/upstream to Named on port 53, and for the Adblocker it will forward to Named on port 54. Cert and key are left empty as the TLS part is done with HaProxy.

Back-end DNS Named (Bind9)

As DNS server it uses Named/BIND9 by ISC. I think this is one of the best DNS software as I deem it very stable and customizable. This section will be long and complex due to the many customizations that are done.

Configuration

The configuration will contain the following sections:

  • Options
  • Logging
  • Zones and Masters

Options
This section is used as global options in Named.

options {
 auth-nxdomain yes;
 directory "/usr/local/etc/";
 listen-on { any; };
 dnssec-validation auto;
 statistics-file "/var/log/named.stats";
 recursion yes;
 allow-query     { any; };
 allow-recursion { any; };
 allow-query-cache { any; };
 minimal-responses yes;
 max-cache-size 3000m;
 tcp-clients 1500;
 minimal-any true;
 response-padding { any; } block-size 468;
 qname-minimization relaxed;
};

minimal-response has been set to improve performance. This ensure that the DNS server won’t return any other query types other then NS, A, and AAAA, unless specifically requested.

max-cache-size has been set as well for optimization. If not set, it can use up to 90% of available memory. As these servers contain multiple services, I have set it manually to ensure some memory for e.g. Named Adblocker, DNS over HTTPS server etc.

minimal-any has been set to ensure that queries with the type ANY do not return all types, but only NS. This decreases bandwidth, and is just considered best practice.

response-padding is used as well, to prevent analysis of encrypted downstream packets in correlation with unencrypted upstream queries to the DNS root servers (in regards to packet length). Just best practice as well.

qname-minimization is set to ensure privacy when querying authoritative DNS servers. Basically this means: if I query a.b.c.d.com, only request NS of 1 subdomain e.g. ask .com for d.com, and ask d.com for c.d.com., and ask c.d.com for b.c.d.com. This prevents leaking the actual request which is a.b.c.d.com.

Logging
Section is to prevent error messages from requests in the logging. It ensures that only critical messages are sent to systemd-journal. Not to special:

logging {
  channel default_syslog {
      print-time yes;
      print-category yes;
      print-severity yes;
      syslog daemon;
      severity critical;
  };
  category default { default_syslog; };
};

Zones and Masters
This section contains the zones that are used to let a user know on https://securedns.eu if they are connected to SecureDNS or not, and the pairing with Emercoin Core and OpenNIC.

zone "." {
 type hint;
 file "/usr/local/etc/named.root";
};zone "test.securedns.eu" IN {
 type master;
 file "/usr/local/etc/zone/test.securedns.eu.zone";
 allow-query { any; };
 notify no;
};

First zone is the root zone which forwards to the root hints. Second zone is a master authoritative zone for test.securedns.eu. The zone contains 1 A record for test.securedns.eu as well. This means that if you can resolve test.securedns.eu, or be able to reach https://test.securedns.eu hosted by Nginx, you are using SecureDNS (keep in mind that SecureDNS is not the authoritative server for its own securedns.eu domain as this is hosted with Cloudflare. This is why test.securedns.eu is not known with exception from the SecureDNS DNS server).

$(document).ready(() =>
    $.ajax({type: 'HEAD', url: 'https://test.securedns.eu'})
    .done(() => {
      const $text = $('#status');
      $text.html("You are using SecureDNS! <i class='fas fa-lock'></i>");
    })
  );

The website https://securedns.eu actually does a Ajax jQuery HEAD request to https://test.securedns.eu. If this query is succesful, then it means that you were able to resolve test.securedns.eu as well. This is how SecureDNS is able to check if you are using SecureDNS or not.

Next follows the pairing with OpenNIC TLDs:

masters opennicPeers {
 185.121.177.177;
 2a05:dfc7:5::53;
};masters opennicNS {
 161.97.219.84;
 2001:470:4212:10:0:100:53:10;
 163.172.168.171;
 94.103.153.176;
 2a02:990:219:1:ba:1337:cafe:3;
 207.192.71.13;
 178.63.116.152;
 2a01:4f8:141:4281::999;
 51.77.227.84;
 188.226.146.136;
 2001:470:1f04:ebf::2;
 51.75.173.177;
 79.124.7.81;
 144.76.103.143;
 2a01:4f8:192:43a5::2; 
};zone "dns.opennic.glue" in {
 type slave;
 file "zone/dns.opennic.glue.zone";
 allow-transfer { any; };
 notify yes;
 masters { opennicNS; opennicPeers; };
};zone "bbs" in {
 type slave;
 file "zone/bbs.zone";
 allow-transfer { any; };
 notify yes;
 masters { opennicNS; opennicPeers; };
};zone "cyb" in {
 type slave;
 file "zone/cyb.zone";
 allow-transfer { any; };
 notify yes;
 masters { opennicNS; opennicPeers; };
};zone "bit" in {
 type slave;
 file "zone/bit.zone";
 allow-transfer { any; };
 notify yes;
 masters { opennicNS; opennicPeers; };
};zone "chan" in {
 type slave;
 file "zone/chan.zone";
 allow-transfer { any; };
 notify yes;
 masters { opennicNS; opennicPeers; };
};zone "dyn" in {
 type slave;
 file "zone/dyn.zone";
 allow-transfer { any; };
 notify yes;
 masters { opennicNS; opennicPeers; };
};zone "free" in {
 type slave;
 file "zone/free.zone";
 allow-transfer { any; };
 notify yes;
 masters { opennicNS; opennicPeers; };
};zone "fur" in {
 type slave;
 file "zone/fur.zone";
 allow-transfer { any; };
 notify yes;
 masters { opennicNS; opennicPeers; };
};zone "geek" in {
 type slave;
 file "zone/geek.zone";
 allow-transfer { any; };
 notify yes;
 masters { opennicNS; opennicPeers; };
};zone "gopher" in {
 type slave;
 file "zone/gopher.zone";
 allow-transfer { any; };
 notify yes;
 masters { opennicNS; opennicPeers; };
};zone "indy" in {
 type slave;
 file "zone/indy.zone";
 allow-transfer { any; };
 notify yes;
 masters { opennicNS; opennicPeers; };
};zone "libre" in {
 type slave;
 file "zone/libre.zone";
 allow-transfer { any; };
 notify yes;
 masters { opennicNS; opennicPeers; };
};zone "neo" in {
 type slave;
 file "zone/neo.zone";
 allow-transfer { any; };
 notify yes;
 masters { opennicNS; opennicPeers; };
};zone "null" in {
 type slave;
 file "zone/null.zone";
 allow-transfer { any; };
 notify yes;
 masters { opennicNS; opennicPeers; };
};zone "opennic.glue" in {
 type slave;
 file "zone/opennic.glue.zone";
 allow-transfer { any; };
 notify yes;
 masters { opennicNS; opennicPeers; };
};zone "oss" in {
 type slave;
 file "zone/oss.zone";
 allow-transfer { any; };
 notify yes;
 masters { opennicNS; opennicPeers; };
};zone "oz" in {
 type slave;
 file "zone/oz.zone";
 allow-transfer { any; };
 notify yes;
 masters { opennicNS; opennicPeers; };
};zone "parody" in {
 type slave;
 file "zone/parody.zone";
 allow-transfer { any; };
 notify yes;
 masters { opennicNS; opennicPeers; };
};zone "pirate" in {
 type slave;
 file "zone/pirate.zone";
 allow-transfer { any; };
 notify yes;
 masters { opennicNS; opennicPeers; };
};zone "o" in {
 type slave;
 file "zone/o.zone";
 allow-transfer { any; };
 notify yes;
 masters { opennicNS; opennicPeers; };
};

On top you have the master servers that hold the zones for the OpenNIC TLDs. Then you have the slave zones that actually replicate the full zone locally.

Next follows the pairing with the Emercoin Core TLDs:

zone "emc" in  {
 type forward;
 forward first;
 forwarders {
  10.129.6.52 port 5335; // Local Emercoin wallet
 };
};zone "coin"  in {
 type forward;
 forward first;
 forwarders {
  10.129.6.52 port 5335; // Local Emercoin wallet
 };
};zone "lib" in {
 type forward;
 forward first;
 forwarders {
  10.129.6.52 port 5335; // Local Emercoin wallet
 };
};zone "bazar" IN {
 type forward;
 forward first;
 forwarders {
  10.129.6.52 port 5335; // Local Emercoin wallet
 };
};

These are forwarding zones that forward to port 5335.

As DNSSEC is enabled for Named, you won’t be able to actually resolve these TLDs from Emercoin and OpenNIC as they do not have any RRSIG record. Therefore you have to create a negative trust anchor for these TLDs by using the following:

/usr/local/sbin/rndc nta -lifetime 604800 -force <TLD>

Back-end DNS Named (Bind9) with Adblocker

Previous sections are the configuration for the main Named instance. Though it also has a Named instance with an Adblocker. This configuration is the same for the options and logging part, apart from the listening port which is 54 instead of 53 (as can be seen in previous sections as Adblocking services forward to 54).

Zones and Masters
The zones are different though. This instance uses the Named main instance as forwarder.

zone "." {
  type forward;
  forward first;
  forwarders {
    127.0.0.1 port 53; // Main Bind instance
  };
};zone "test.securedns.eu" IN {
  type master;
  file "/usr/local/etc/zone/test.securedns.eu.zone";
  allow-query { any; };
  notify no;
};zone "ads-test.securedns.eu" IN {
  type master;
  file "/usr/local/etc/zone/ads-test.securedns.eu.zone";
  allow-query { any; };
  notify no;
};

As you can see it also contains an extra zone that hosts the ads-test.securedns domain. This zone file contains again an A record for ads-test.securedns.eu. Same as previously, if you are able to resolve this domain, and therefore be able to browse to https://ads-test.securedns.eu hosted on Nginx, you are using SecureDNS with Adblocker.

Then the last part of this configuration is the actual blocking of domains. SecureDNS uses the domain block list from Notracking. Every night a script runs that iterates over each domain, rewrites it to the syntax below, and ads it to the Named Adblocker configuration file.

zone "blocked.domain.com" { type master; notify no; file "zone/ads-null.zone"; };

This zone sets ads-null.zone as authoritative for the domain. This zone file contains a wildcard record that resolves everything to 127.0.0.1 (and therefore it is blocked).

Back-end Emercoin Core

It runs the Emercoin Core to get the Blockchain database. So you need to install Emercoin Core and run Emercoind.

Configuration

It uses uses the following configuration:

emcdns=1 
emcdnsport=5335

Back-end DNSCrypt-Wrapper

It uses a DNSCrypt-wrapper to run the DNSCrypt instances. As DNSCrypt uses UDP as well, it cannot use the loadbalancer. This means that this service has no redundancy/failover.

To start DNSCrypt it uses 2 scripts:

  • Daemon: starts the actual 2 wrapper below, and kill already running wrappers. Does this twice a day to ensure daily key re-roll.
  • Wrapper: starts the DNSCrypt instances (1 for IPv4 and 1 for IPv6) and rerolls keys if older than 24 hours, forwards to Named
  • Wrapper Adblock: same as above, but forwards to Named Adblocker

LetsEncrypt

SecureDNS uses LetsEncrypt certificates. These certs expire every 90 days. One could you certbot to automatically renew the certificates.

Keep in mind that if you renew certificates with certbot, it will automatically generate a new public key. As the DNS over TLS standard actually validates the public key, one need to make sure that it stays the same. Therefore, I always renew the certificates manually.

I won’t go to much into detail as there is enough written on the internet about this, but its basically done by doing the following:

  • create ssl configuration file (cnf) with subjectAltName
  • generate private key with openssl
  • generate a csr file from private key with openssl
  • generate LetsEncrypt chain by using certbot with — csr command
  • combine/cat the chains, cert, and privkey into haproxy.pem
  • load haproxy.pem with HaProxy

I would like to thank everyone for using SecureDNS. Especially the people who donated as they made it possible to keep it up and running for almost 3 years.

Feel free to contact me for any questions.

With kind regards,
Rick Lahaye | https://www.linkedin.com/in/ricklahaye/

DNS
Dns Over Https
Dns Over Tls
Dnscrypt
Tld

--

--

Rick Lahaye

Written by Rick Lahaye

5 Followers

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams