Hybrid DNS for the Enterprise in AWS

Learnings and challenges from extending a hybrid DNS solution from an existing enterprise DNS environment into AWS.

Matt Adorjan

Published in

Statuscode

9 min readApr 17, 2017

Update: 19-Nov-2018

On November 19, AWS released a new offering called ‘Route 53 Resolver for Hybrid Cloud’. This offering removes most of the undifferentiated heavy lifting outlined in this article below and offers a solution for extending your enterprise hybrid DNS solution in to your VPCs without any extra DNS services or servers. Click here to read more about this offering.

AWS’ DNS offering, Route 53, is a great option for managing the basics of name resolution when resources exist solely within the AWS ecosystem. Route 53 integrates with most all of AWS’ services, offers great alias record functionality, and is easy to automate. However, for enterprises looking to make the move to AWS, especially in a hybrid deployment, Route 53 will most likely not be enough to provide all of the name resolution capabilities required. Because of this, hybrid DNS solutions need to be explored and implemented.

Through the rest of this post, I will go through some of the key areas to consider when planning for the extension and integration of internal DNS into AWS. This post will focus on internal, private DNS zones.

Route 53 Background and Limitations

For those not familiar, Route 53 is Amazon’s scalable DNS service (more information here). It is quite powerful and very cost effective. It offers great integration with AWS services, has support for private and public zones, and supports all of the most commonly used record types. For enterprises who are new to AWS, it might seem like the obvious choice.

An example of how one might expect Route 53 hybrid DNS integration to work. Unfortunately, this is not the case.

Many who are new to AWS expect to be able to use Route 53 as expressed in the above diagram, by forwarding on-premise systems to Route 53 for DNS resolution, and adding forwarders to Route 53 for resolving on-premise names. This will not work.

This brings us to some of the key downsides to Route 53:

Route 53 private endpoints are not available over VPN/DirectConnect:

Whenever you create a private zone in Route 53, and associate that zone with a VPC, Route 53 creates a DNS endpoint address for that VPC. It’s just the +2 of the VPC subnet (e.g. if the VPC is 172.31.0.0/16, the Route 53 address is 172.31.0.2).
In order to allow on-premise clients resolve records in a Route 53 hosted zone, you have to somehow add a forwarder to your on-premise DNS environment to point requests for that specific domain to Route 53.
In a traditional example, you’d go into your on-premise DNS, add a conditional forwarder pointing cloud.acmecorp.com -> 172.31.0.2, and be done with it.
Unfortunately, that isn’t going to work. The Route 53 private endpoint address for the VPC is not routable across VPN or DirectConnect! So this means that, even if you can get to any services running in AWS over your VPN within the 172.31.0.0/16 subnet, you will never be able to reach 172.31.0.2 over that same VPN connection.

Route 53 provides no forwarding or conditional forwarding options:

This means you cannot just point everything in AWS to Route 53 and set up a forwarder for the domains you use in your on-premise network, expecting it to get there. You will have to plan around this and implement a solution to enable this resolution.

Route 53 does not support private zone transfers:

Many enterprises will want to extend an existing domain name to their AWS environment by adding another sub-domain, such as “cloud.acmecorp.com”. This is a great idea because you can easily identify where resources are located, and you can be comfortable there will be no conflicts with on-premise name records.
The downside to this is that, Route 53 does not support the notion of private zone transfers - meaning that, if you have the root level domain “acmecorp.com” registered somewhere, and you want to appoint Route 53 as the authoritative source for “cloud.acmecorp.com”, this is not possible.

Forwarding Name Resolution Requests

Knowing what we know about Route 53 private endpoints not being available over VPN, we need another solution for clients to resolve Route 53 addresses, and for AWS servers requiring resolution of on-premise addresses. The recommended solution here is the implementation of a forwarding service within AWS which provides a routable address over the VPN. You can then point your on-premise DNS services to this new service.

The following diagram shows the problem we need to solve. Where the ? indicates some solution that needs to be implemented to help us facilitate name resolution between AWS and our existing on-premise network.

AWS has written posts on 3 options for doing this, so rather than reiterate everything they have already said, I will post them here:

DNS Resolution Between On-Premises Networks and AWS Using AWS Directory Service and Amazon Route 53

Very simple to set up and maintain — no configuration required of the AWS service.
Highly available without configuration.
Can not configure specific forwarders — locked down to the forwarding rules built into the service.

DNS Resolution Between On-Premises Networks and AWS Using AWS Directory Service and Microsoft Active Directory

AWS service — no servers to manage.
Highly available without configuration.
No built in forwarders, like Simple AD — must configure them.

DNS Resolution Between On-Premises Networks and AWS by Using Unbound

Allows for more complex configuration.
Familiar to enterprises already and is the best option not requiring AD DNS.
Complex to set up and maintain.
EC2 instances required — not a service.
Must configure the service to be highly available.
This can be accomplished using something like BIND, as well.

The biggest downside that comes with any of these solutions is the need to configure and manage an additional DNS service, in addition to Route 53. Of course, choosing one of the managed directory services helps alleviate some of this concern. However, you are still adding an additional layer of complexity and you may experience resistance from an already overwhelmed operations group.

One other option I’d like to bring up, though it only addresses half of the problem, is a blog post from AWS on Powering Secondary DNS in a VPC using AWS Lambda and Amazon Route 53 Private Hosted Zones. This post explains how to build a process that does a zone transfer from all of your on-premise zones to Route 53 at a regular interval. This essentially provides full name resolution capabilities for services running in AWS to access resources in your on-premise network, without the need for any additional DNS infrastructure.

The limitation here is that, for any services on-premise which need to talk to a private zone in Route 53, you still require a DNS forwarding server in AWS which can be accessed over VPN or DirectConnect.

Additional notes on DNS for services running in AWS

Every VPC comes with a default “DHCP Option Set” assigned. The default configuration assigns Route 53 as the only DNS server for the machines in that VPC. This is fine for basic environments, but if you need to allow your services running in AWS to talk to services running elsewhere in your network, you need to change your DHCP option sets to point to your newly configured DNS forwarding service (discussed above).

The following diagram shows the name resolution process for systems in AWS which need to resolve internal and external, on-premise DNS names. Where the ? indicates a solution that needs to be implemented to help us facilitate this name resolution.

One item to note when changing DHCP option sets - there are some AWS services that provide internal DNS names which can only be resolved by Route 53.

For example:

When creating an EFS mount, I will be provided a DNS name for a mount point, such as: fs-1234567d.efs.us-east-1.amazonaws.com.
Even though the DNS name above is part of the “amazonaws.com” domain, it is only resolvable inside your VPC.
If in your DHCP option set, you have a DNS server above the internal Route 53 endpoint (VPC address plus 2), your EC2 instances will try to resolve the EFS mount-point DNS name from the public “amazonaws.com” DNS records and it will fail.
This is partially detailed in the EFS documentation, under “Mounting on On-Premises Servers with a DNS Name”

Some options for addressing the above:

Use only the IP address of the mount point when referring to EFS — this way, DNS is taken completely out of the picture. This also has downsides, such as if the IP address changes for some reason.
Add a record to your own private DNS zone to point to the EFS IP address, such as efs.cloud.acmecorp.com. This way, the EFS mount point address can be resolved via the same DNS resolution process you already have in place.
Add a forwarder for any *.amazonaws.com addresses into your DNS forwarder that sends requests to Route 53 for name resolution.

Automated DNS Record Management

As I mentioned earlier, the real benefit of sticking with Route 53 over another solution is the fact that it’s fully integrated with AWS and can be automated via the same methods as other AWS resources.

CloudFormation

For example, say you’re creating a bunch of servers and a load balancer via CloudFormation:

You can add Route 53 resources into your CloudFormation template:
1 for each server as an A record
1 record for your Load Balancer as an Alias record
You’re then just using References to the other objects being created within the stack to populate the Route 53 records. Simple!

This also makes it simple to manage the lifecycle of those records — because the records are associated with the same CloudFormation stack as the actual servers, when you delete the stack and the servers, the Route 53 records are deleted at the same time.

Lambda

Route 53 records can also be automatically added and removed via a Lambda function which is called each time a new resource is created or deleted. This option removes the requirement for specifying any DNS records in a CloudFormation template, as the DNS record lifecycle is now managed via this Lambda function and its interaction with the Route 53 API.

For more information on this approach, head over to this AWS blog post: Building a Dynamic DNS for Route 53 using CloudWatch Events and Lambda.

Reverse Lookups

One other piece about Route 53 — reverse lookup zones are not automatically created or maintained. With some DNS services, when you create a DNS record in a zone, it will maintain a record in a reverse lookup zone, as well. This is not the case with Route 53. In order to offer reverse lookup on Route 53, this is another private hosted zone you will need to create and then associate with your VPC.

Manually maintaining reverse lookup records would be easy enough when there is a rigid process for performing DNS record updates. However, in an environment that is dynamic and changing often, it is more cumbersome.

The method I have implemented for reverse zones is to:

Let your initial provisioning process (e.g. CloudFormation) handle the creation of the forward records (e.g. host1.cloud.acmecorp.com -> 172.31.20.45).
Create a Lambda function which will monitor your forward zones in Route 53, and create new reverse records whenever a new record is added. It’s a lot easier to create new Route 53 forward records in CloudFormation than it is to create reverse records.

Conclusion

I hope this compilation of information about private DNS zones on AWS will be helpful for someone else. I’ve gone through a couple of these hybrid DNS integrations, so I wanted to share some learnings and pain points I’ve come across. I also found there wasn’t much comprehensive information available on hybrid DNS on AWS. I’m hoping this can fill a small part of that gap.

Thanks for reading!