Anycast DNS — Resilient Scalability for Critical Network Infrastructure Software, Part 2 — Implementation

9 min readJun 18, 2019

This is the second and final part in a series on Anycast DNS where I’ll describe the technical details of deployment and migration. If you haven’t already read Part 1, you can check it out here where I explained how Anycast works and why it’s a game changer for your DNS infrastructure.

Scope

I’ll be talking specifically about implementing Anycast DNS on Infoblox DNS servers peering with Nexus Switches (NX-OS mode) using eBGP. For the purpose of this walkthrough each Nexus switch represents a vPC pair but for simplification I’m only showing the configuration for one switch in the pair — just know that the configuration would be similar for each member of the vPC. Also, be aware that Infoblox supports BFD for BGP, but some Nexus platforms do not support BFD over a vPC — check with support for your specific platform.

In the example below I’ll be using the following IPs:

Anycast IP — 10.10.10.10/32

DNS Servers — 192.168.1.5/24, 192.168.1.6/24, 192.168.2.5/24, 192.168.2.6/24

Nexus Switches — 192.168.1.2/24, 192.168.2.2/24

For the example network we have two data centers with two DNS servers and a pair of Nexus core switches in each location. Both Nexus switches at each location peer with the two local Infoblox DNS servers. Per best practices outlined in Dinesh Dutt’s BGP in The Data Center, the Infoblox servers share the same ASN and each pair of Nexus switches has its own ASN.

I will not be covering redistribution into the EIGRP AS shown in the diagram as there are plenty of examples of this on the internet. However, if your network uses OSPF you’re in luck because Infoblox supports OSPF — no eBGP or redistribution required. However, since Infoblox uses Quagga, one caveat to consider is that Quagga’s BGP implementation is much more mature than OSPF.

Logging

I highly recommend logging DNS queries before, during, and after the migration to Anycast — this is essential and one of the best ways to gauge user experience throughout the process. The graphs are also very helpful in conveying the effectiveness of the solution to management and other teams.

To configure query logging in Infoblox, go to the Grid Manager -> Grid DNS properties and under the Logging tab select ‘queries’. You may need to toggle Advanced Mode in the upper left of the DNS Properties box to make the Logging tab available.

NOTE: I’ve been told by other engineers that Infoblox shys away from recommending logging queries as it can have an adverse performance impact. Use caution and keep an eye on server performance if you make this change. The default ‘Status’ dashboard in Grid Manager -> Dashboards -> Status shows performance stats for all Grid Members.

If you use Splunk for logging in your organization, check out the Splunk Add-on for Infoblox, which will parse out the fields in Infoblox log messages for more meaningful reports. Also, if you use Splunk, there’s no need for the extra Infoblox reporting features to report on query data.

Once query logs are sent to Splunk you can create a report that shows server load using this Splunk search:

sourcetype=infoblox:dns | chart count(query) by dns_request_name_serverIP

This report shows the DNS IP used by clients to resolve names, and once Anycast is successfully implemented we’ll see another server in this report showing queries sent to the Anycast IP address.

While you’re in the Splunk dashboard you can create a couple more reports that will be handy throughout the migration:

Top 10 misconfigured clients (using NON-anycast IPs):

sourcetype=infoblox:dns dns_request_name_serverIP!=10.10.10.10 | top limit=10 dns_request_client_ip

This report will be used after the Anycast implementation to show which clients are still using the old DNS IPs. The report shows the top 10 misconfigured clients by number of queries, so you can focus your efforts on the most active misconfigured clients.

Anycast server load distribution — used to verify ECMP (where applicable):

sourcetype=infoblox:dns dns_request_name_serverIP=”10.10.10.10" | chart count(query) by infoblox_host

The first place you would verify ECMP is the routing table, but it’s always helpful to see the fruits of your labor from another angle.

Anycast Routing Configuration: Infoblox

In this section we’ll add a loopback interface, enable Anycast and BGP, and configure the loopback as a DNS source interface for each Infoblox DNS member. It’s important to note that BGP will not attempt to peer with neighbors until the loopback is set to listen for DNS queries. This is a key aspect of the design and emphasizes how the DNS and routing processes are tightly coupled in Infoblox; if DNS is not working for any reason, the anycast route will not be advertised. More details on this entire section can be found in the Infoblox Admin Guide: Configuring IP Routing Options.

First, add a loopback interface to each Infoblox member in the Grid Manager by going to Grid -> Grid Manager tab -> Members tab -> select the Grid Member -> Edit. In the Grid Member Properties window under the Network tab add a loopback interface in the Additional Ports and Addresses section and assign your anycast IP with a /32 subnet mask.

Next, enable anycast addressing on each Infoblox member. In the same Grid Member Properties Editor as above, toggle Advanced Mode and select the Anycast tab. Under Anycast Interfaces select and add the loopback interface you configured in step 1. Also, select BGP so the loopback address gets advertised to BGP neighbors as a /32 network.

To configure BGP stay in the Anycast tab and scroll down to BGP Neighbor Configuration. Dinesh Dutt recommends making the BGP Keepalive timer 3 seconds and the Hold Down 9 seconds in his BGP Guide. Next we’ll configure the local ASN and BGP neighbors for our Nexus switches. Remember, if you have a pair of Nexus switches there will be neighbor configuration for each switch. Optionally configure BFD and BGP authentication for each BGP neighbor.

Next we’ll set the loopback interface as a DNS source port, which lets Infoblox respond to DNS queries sent to the loopback/anycast IP.

Edit the properties for each DNS server/member in Data Management -> DNS tab -> Members tab. In the General tab in Member DNS Properties, select Advanced and add the loopback IP address to the list of additional IP addresses.

At this point Infoblox is ready to peer with our core switch and begin advertising the anycast route and listen for DNS queries sent to the anycast address.

Anycast Routing Configuration: NX-OS

The Nexus configuration is fairly simple: configure prefix-lists to control what routes are sent and received via BGP, enable the BGP feature for NX-OS, and configure each local Infoblox DNS server as a BGP peer. For our current example, we’ll also ensure ECMP.

First we need to configure a couple prefix lists to make sure we’re not sending a lot of superfluous routes to Infoblox or receiving anything other than the anycast route from Infoblox.

For outbound route filtering we’ll only allow a default route sent to Infoblox:

Nexus Core A(config)# ip prefix-list DEFAULT seq 5 permit 0.0.0.0/0

And for routes coming from Infoblox we’ll only allow the anycast route:

Nexus Core A(config)# ip prefix-list DNS-ANYCAST seq 5 permit 10.10.10.10/32

Next, we need to enable the BGP feature in NX-OS and configure each Infoblox server as a BGP peer.

To enable the BGP feature in NX-OS:

Nexus Core A(config)# feature bgp

The BGP configuration for both Infoblox peers from Nexus Core A’s prospective:

router bgp 64601
  router-id 192.168.1.2
  timers bgp 3 9
  bestpath as-path multipath-relax
  address-family ipv4 unicast
    maximum-paths 4
  neighbor 192.168.1.5
    remote-as 64550
    log-neighbor-changes
    password 3 [md5 authentication password goes here]
    timers 3 9
    address-family ipv4 unicast
      prefix-list DNS-ANYCAST in
      prefix-list DEFAULT out
  neighbor 192.168.1.6
    remote-as 64550
    log-neighbor-changes
    password 3 [md5 authentication password goes here]
    timers 3 9
    address-family ipv4 unicast
      prefix-list DNS-ANYCAST in
      prefix-list DEFAULT out

A few things to note about the above configuration:

timers bgp 3 9 — we’re using the same timer configuration we used in Infoblox.
Although not a necessity for our example, bestpath as-path multipath-relax relaxes the same AS-Path requirement BGP has by default for ECMP routing.
maximum-paths 4 enables BGP to install up to 4 equal cost paths to the destination network.
prefix-list DNS-ANYCAST in and prefix-list DEFAULT out applies the prefix lists we configured earlier, keeping the core from sending superfluous routes to Infoblox and vice versa.

Finally, let’s verify Nexus Core A is receiving two routes for 10.10.10.10/32 from our Infoblox servers:

Nexus Core A# show ip route bgp10.10.10.10/32, ubest/mbest: 2/0
    *via 192.168.1.5, [20/0], 1m32s, bgp-64601, external, tag 64550
    *via 192.168.1.6, [20/0], 2m24s, bgp-64601, external, tag 64550

At this point you should have a functioning anycast solution for DNS, but there is still one more thing to do — migrate clients to use 10.10.10.10 for DNS lookups.

Client Migration

One of the convenient aspects of implementing anycast for DNS in this way is that both the old configuration and the new configuration run in parallel. Since the DNS servers are still listening for queries on the old IPs you can take as much time as you need migrating clients to the new anycast IP.

The main way of migrating clients will be through DHCP. I recommend adding the anycast IP to the beginning of the list of DNS IPs you hand out to clients. What I’ve found is that most clients use the first DNS server in their list and maintaining the old IPs ensures clients have other servers to fall back to if for some reason the anycast IP is unreachable.

In Infoblox either as a per-member configuration setting, or under DNS Grid Properties, add the anycast address as an IPv4 Option and move it to the top of the list.

Gradually, as clients renew their leases they’ll grab the new IP and begin querying anycast for DNS. In the ‘Anycast Server Load Distribution’ Splunk report I mentioned above, you’ll begin to see the anycast IP taking on more and more queries.

There will be outliers, but that’s what the ‘Misconfigured Clients’ Splunk report is for. If you begin to see clients from a particular subnet consistently using the old DNS IPs, this may be an indication the anycast IP is being blocked for that network.

Once you’re certain all clients can successfully query the anycast IP, pull the old servers from the DHCP options list.

Going back to the splunk reports, we can see that server load distribution is much more balanced per site.

We can also confirm that a majority of the DNS queries are being sent to the anycast address.

Congratulations, you’ve successfully implemented Anycast for DNS!

Don’t Forget to Lab It Up

If you’re a network engineer reading this you probably already realize how easy it would be to get anycast DNS running in a lab, and this really should be done before you set up anycast in the wild. In its simplest form you only need a couple routers and the Infoblox DDI OVA, which is available with a 90 day trial license.

Special Thanks

To Jason Murray and Joe Marentette from Washington University in St. Louis for introducing me to Anycast DNS and for hosting the local Infoblox User Group.

And to my friends and colleagues, Iain Leiter and Garrett Holthaus for making valuable suggestions and editing both articles in this series.

Contact Me

If you have any questions or comments, feel free to reach out to me on LinkedIn or shoot me an email at bowles.thomas@gmail.com.