Recently I published an article for using Custom DNS Servers with AKS Pods (and Nodes).
There, the approach I took was to modify the DNS Settings of the VNet and set a DNS forwarder to Azure DNS.

In this article, I am going to cover how to modify the CoreDNS to use Custom DNS Servers.

Some theory of-course …

Modifying CoreDNS in AKS requires creation of a ConfigMap with a specific name ‘coredns-custom’ in the kube-system namespace.

The configuration from this custom ConfigMap is read by CoreDNS and the associated Corefile is modified to incorporate the new settings.

To setup usage of Custom DNS Server as a resolver for my custom domain (testabc.com), the ‘forward’ plugin for CoreDNS can be used.

Note that the earlier used ‘proxy’ plugin in now replaced by ‘forward’ plugin.

Architecture:

Request from within the Pods:

• to custom domain testabc.com will go to the Custom DNS Server.
• to all other domains will use the default configuration, i.e. use the /etc/resolv.conf file, which is derived from the host. In this specific case, since I have not modified the DNS Settings in the VNet where my AKS Nodes are present, the nodes use Azure DNS as the default DNS resolver.

The Custom DNS Server sitting in another VNet should be reachable from the AKS Node.

Setup

• My AKS VNet settings point to the Azure DNS. That means the Nodes will use Azure DNS as the default resolver.
• AKS VNet is peered to another VNet that contains the Custom DNS Server and a VM (VM1) whose hostname we will resolve.
• CoreDNS version is 1.6.6. This is important as the plugin ‘forward’ was introduced in version 1.6.x and the earlier used ‘proxy’ plugin was removed.

Steps

Assuming you already have an AKS Cluster and the AKS Nodes are able to connect to the Custom DNS Server:

1. Write the ConfigMap to ‘forward’ queries to the Custom DNS Server (172.16.0.4 in my case)*if* the requested domain for DNS resolution from the Pod is my custom domain (testabc.com).
2. Update the custom ConfigMap that the CoreDNS refers to create additional configurations in the CoreFile
3. Restart the CoreDNS Pods to take in the new configuration
   For all other queries, the default configuration of CoreDNS will be used which uses the DNS configuration of the Node.

Detailed Steps:

• Write the ConfigMap named ‘mycoredns.yaml’

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  yourdns.server: |
    testabc.com:53 {
      errors
      cache 1
      forward . 172.16.0.4
    }

• Apply the ConfigMap

kubectl apply -f <path_to_file/mycoredns.yaml>

• Restart the CoreDNS Pods to take the new settings

kubectl delete pods -n kube-system -l k8s-app=kube-dns

Once the reboot is successful, the CoreDNS Pods will be updated with the new settings. The request flow will look something like this:

Verification

In order to verify the settings, I created an httpd pod, installed dnsutils on it, and tried to resolve both external and custom domain.

$ kubectl exec -it myapp /bin/bash

root@myapp:/usr/local/apache2# nslookup google.com

Server:  10.0.0.10

Address: 10.0.0.10#53

Non-authoritative answer:

Name: google.com

Address: 172.217.13.238

Name: google.com

Address: 2607:f8b0:4004:808::200e

root@myapp:/usr/local/apache2# nslookup vm1.testabc.com

Server:  10.0.0.10

Address: 10.0.0.10#53

Name: vm1.testabc.com

Address: 172.16.0.5

As can be noted, both custom and external domains are getting resolved as expected.

Footnotes

For reference, here is the default ConfigMap for CoreDNS:

kubectl describe cm -n kube-system coredns
Name:         coredns
Namespace:    kube-system
Labels:       addonmanager.kubernetes.io/mode=Reconcile
              k8s-app=kube-dns
              kubernetes.io/cluster-service=true
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"v1","data":{"Corefile":".:53 {\n    errors\n    health\n    kubernetes cluster.local in-addr.arpa ip6.arpa {\n      pods in...

Data
====
Corefile:
----
.:53 {
    errors
    health
    kubernetes cluster.local in-addr.arpa ip6.arpa {
      pods insecure
      upstream
      fallthrough in-addr.arpa ip6.arpa
    }
    prometheus :9153
    forward . /etc/resolv.conf
    cache 30
    loop
    reload
    loadbalance
    import custom/*.override
}
import custom/*.server

Events:  <none>

These changes, made to the DNS Settings for Pods through CoreDNS modification, will not modify the DNS Settings in the Nodes. The Nodes will still refer to the DNS Server configured in the VNet.

There are various plugins that are available with CoreDNS and they have a specific order of processing. More details can be found below:

• https://coredns.io/manual/plugins/
• https://github.com/coredns/coredns/blob/master/plugin.cfg

I have worked with a lot of clients who want to use their Custom DNS Server to resolve names from Pods hosted on Azure Kubernetes Service.

These DNS Servers can be on-prem, or in a different VNet, or in the same VNet, and are used to resolve custom domains.

I have noticed many people having difficulty with the configuration, whereas this is a fairly simple one if the flow of the traffic is clear.

In this post, I will talk about one of the approaches that can be taken to enable the AKS Pods and resolve custom domain names. One other popular approach is to modify the CoreDNS configuration, which is covered in another post here: https://medium.com/@rishasi/custom-dns-configuration-with-coredns-aks-kubernetes-599ecfb46b94

For the solution to work, in very simple terms, forward all DNS queries to the Custom DNS Server, and:

• resolve the custom domain using this Custom DNS Server, and
• for every other domain, forward request to Azure DNS using DNS forwarder.

A simple diagram:

Approach:

Pod requests DNS resolution, which will then be forwarded to the DNS Server configured in VNet settings, say 10.x.x.x.

This DNS Server can be on-prem, or in the same VNet, or a peered VNet.

The DNS Server will see the request, and:

a. If the DNS query is for it’s own custom domain,
It will itself resolve the query and return the IP.

b. If it is any other domain, for example ubuntu.com,
Then the DNS Server will forward the request to the Azure DNS.

For it to work, you will have to setup the DNS Forwarder to route traffic for external domains to a public DNS Server. However, it should be noted that the AKS Nodes and the Pods may need to resolve other external domain names for various reasons, which also include Azure’s Internal Domains like cloudapp.net. Thus it is recommended to forward external queries to Azure DNS Server (168.63.129.16).

Edit:
The above approach works as long as the Custom DNS Server is in Azure. So, setting Azure DNS as the forwarder is straightforward using the Azure DNS IP: 168.63.129.16.

However, if the Custom DNS Server is on-prem, or outside Azure, the Azure DNS cannot be reached from there.
So setting Azure DNS as the forwarder will not be that straightforward.

In that case, the Custom DNS, which is on-prem, can set the forwarder as:
- another DNS Server configured on Azure VM,
- or Azure Firewall
- or a DNS Proxy on Azure
.. and then that device can use Azure DNS as it's DNS Server.

Basically, we cannot reach Azure DNS IP directly from outside Azure, so we first forward queries to a VM/Firewall/DNS Proxy, which is on Azure, and then that device can forward requests to Azure DNS.

A little tricky, but hopefully this helps.

This is it. If you understood the setup, great. If not, don’t worry, there is more to come.

I prepared a simple setup to *mimic* the discussed architecture, where the AKS VNet is connected to the on-premise environment using VPN/ExpressRoute.

For this setup,

• Launched an AKS Cluster
• Created a DNS Server in another VNet
• Created a VM in the DNS Server’s VNet (VM1) whose name we will resolve
• Created a peering connection between AKS VNet and the custom VNet
• Updated AKS VNet to use the custom DNS Server instead of the default Azure Provided DNS Server

Instead of setting up a VPN/ExpressRoute, I used VNet Peering. The basic idea is to have connectivity to the Custom DNS Server. Thus, here, the custom VNet can be thought of as the on-prem network.

Creating a DNS Server was the roadblock, as I had done it a long long time ago. I used a Windows 2016 VM and referred this simple article to configure a DNS Server on top of it: https://www.hostwinds.com/guide/how-to-setup-and-configure-dns-in-windows-server-2016/

The details for the setup:

AKS VNet CIDR: 10.0.0.0/8

AKS Node IP: 10.240.0.4

Custom VNet CIDR: 172.16.0.0/16

Custom Domain: testabc.com

DNS Server IP: 172.16.0.4

VM1 IP: 172.16.0.5

Hostname for VM1: vm1.testabc.com

The DNS Server settings:

DNS Forwarder set to Azure DNS IP:

Once setup, I first ensured that the hostname for VM1 is getting resolved from the DNS Server itself. From CMD inside the DNS Server:

Then I checked if the DNS is getting resolved from the AKS Node. SSH’d to the AKS Node and tested through CLI:

Here the screenshot validates two things:

1. The DNS Server in other VNet is reachable via the private IP, so the peering is working fine,
2. and that the hostname for the VM1 is getting resolved to the correct IP

So far so good. However, currently I have to specify the DNS Server explicitly in the nslookup command. I want the custom DNS Server to be referred by default.

Thus, I changed the AKS VNet’s DNS Property to refer Custom DNS Server in another VNet (reachable via Private IP due to VNet peering):

The AKS Node had to be restarted for the new settings to propagate. Once the machine restarted, I verified

1. That the DNS Settings for the AKS Nodes are updated

2. The AKS Node is able to resolve the hostname for VM1:

Not only the internal customer domain of ‘testabc.com’ is getting resolved, but also the external domains.

If you look closely, the url google.com has been resolved by a non-authoritative server, suggesting that the Custom DNS forwarded the query to some other dns server, which validates that the forwarding to Azure DNS is working fine.

All working as expected!!

Now the main question, are my pods able to resolve the domain name, both custom and external?

To test, I launched up a simple httpd pod, installed dnsutils on it and performed the test:

Voilà!, as simple as that.

From inside the pod, I was able to resolve VM1 hostname to the correct IP and was also able to resolve external domains.

Footnotes:

It should be noted that I did not specify any dnsPolicy for the Pod. Thus, the default policy was applied, in which the Pod inherits the DNS configuration from the node it is running on.

More details about dnsPolicy for pods can be found here: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-dns-policy

As a side note, the benefit of forwarding external domain names to Azure DNS is that the custom DNS server will only be responsible for resolving the custom domain, and all other domain names are resolved by Azure DNS, reducing the load on the Custom DNS Server.

Lastly, this setup can work in any environment, with minor tweaks, be it Azure, AWS, Google Cloud, or others. The requirement is to ensure that:

a. Custom DNS Server(s) is/are reachable from the Kubernetes Nodes,

b. The Nodes have the Custom DNS Servers as the default DNS resolver, and

c. the Pods have the default dnsPolicy.

Hopefully, this explains the configuration required to use Custom DNS Server with Kubernetes Cluster.

Thank you!

Custom DNS Configuration: AKS/Kubernetes was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.

While working with a client, they wanted zero downtime deployment (updates) for Windows nodes.​ The Deployment resource in Kubernetes, relies on Readiness and Liveness probes for zero downtime deployment.​

However, at the time of the discussion, there was a known issue with Windows nodes, where the Liveness probes and Readiness probes do not work.​

• https://github.com/Azure/AKS/issues/1014#issuecomment-519866944​
• https://github.com/kubernetes/kubernetes/issues/81938​

There were certain workarounds, but none worked for all. Different behavior was observed for each case. The suggestion was not to rely on these probes until the Windows nodes are released in GA which happened recently: https://azure.microsoft.com/en-us/blog/announcing-the-general-availability-of-windows-server-containers-and-private-clusters-for-azure-kubernetes-service/

​Hence, I suggested the client to use Blue/Green Deployment for zero-downtime.​

Problem:

The problem is that, everywhere, the approach utilizes Readiness probes. Even with using Jenkins or Istio, these agent based or agent-less integration tools rely on Readiness probes instead of directly checking the health of the application, and why not? When there is an inbuilt feature provided by Kubernetes, why try to re-invent the wheel?​ However, sadly in this case, the wheels were punctured 😜 , so, Readiness probes were not to be used.​​

As another workaround, the manual solutions just rely on giving a minimum amount of time for the Pods to be ready, and switching the Service to point to New Pods. This is not a permanent solution and is much like a shot in the dark.

What if the application fails to come up in the given amount of time? No entity is checking the status of the application. The solution only relies on the “status” of the “Pod”, but no entity checks the health of the application.​

The big question: How to check health of the application without Readiness probes?​

Do we use a temporary Service? and check the HTTP return code of that service?​ Yes, this could work if implementing a Load Balancer type of service. But this again lead to some problems:

• With many updates, again the Node can have SNAT exhaustion, which the customer had already faced, and had to replace all his nodes.​https://github.com/Azure/azure-container-networking/issues/195​
• Also, the Load balancer IP would take time to come up, causing unnecessary delay in the schedule​
• The biggest problem: Creating a new Load Balancer type Service every time when there is an update made. Like really? This did not seem to be the best approach.​

Another approach could have been creating a Cluster IP type service and run the automation script inside the same VNet as the Nodes.

• But configuring this would again need us to include some Runtime, either creating a new VM, or using something like Azure Functions.
• Configuring kubectl, managing contexts, providing least required access was again too much of overhead.​

Solution​:

Here comes init container into the picture.​The best thing about init container for this solution was “the application container does not start initialization until the init container has completed processing”.​

Thus came an idea, why not to use a Pod with init container as a health checking entity. Here is a brief overview of the workflow:​

Phase 1: Deployment Phase​

The main Deployment (D1) and the main Service (Svc1) are deployed. This is the Blue Deployment.​ All good here.​

Phase 2: Update Phase​

From the script which is used for update, create the following resources:​

1. Deployment 2 (D2), which has the new version of the application, i.e. the Green Deployment.​

2. A temporary service of type ClusterIP. Yeah, we don’t need a Load Balancer or NodePort type service. Let’s call this Svc2. This service will only be needed to check if the application inside the Pods are responding.​

3. A temporary Pod which has the init container. The init container will perform checks on the Svc2. Once desired HTTP status code is received, the init container can complete execution, and then the app container in this Pod can start. The app container in this Pod can be any simple docker container, which just needs to come up and mark the status of the Pod as Ready.​

4. Instead of checking the status of the application from the script, which is very complex, the script can just check the status of the Temporary Pod. Once it is in ready state, it means that the init container completed execution, which means, the Svc2 returned HTTP 200 (or 2XX/ 3XX depending on the use case), which in turn means that the D2 pods are ready to serve traffic.​

Phase 3: Delete Phase​

Once the script determines that the Temp Pod is ready, it can move further in the processing by updating Svc1 to point to D2 Pods by modifying the labels, and then delete the temporary resources: Svc2, Temp Pod, and D1. ​

A simple diagram for the approach:

• for checking the status of the Pod, you could use something like:​

-       $podState = kubectl get pod <temp_pod_name>​

-       $status = $podState[1].SubString(17, 1)​

-       if $status == 0, sleep for 5 sec and keep looping,​

-       if $status == 1, break out of the loop and continue   processing further steps​

For the init containers, health checks can be performed using wget. You can launch any simple windows image that has wget installed. One example is stefanscherer/busybox-windows docker image, and then can pass a script to the Pod, where the status code is checked. Note, that this image supports shell scripts and not Powershell scripts. It has wget and curl pre-installed.​Alternatively, you could launch a Powershell based docker image, and check the availability of the service using wget.​

- $response = wget google.come​

- if $response.StatusCode == 200, break; else sleep 5, keep checking the state. ​

- for a shell script: (for i in {1..100}; do sleep 5; if dig myservice; then exit 0; fi; done; exit 1)​

You can create a script which checks for the status code, and if the output returned is 200, it suggests the service is ready, else, keep checking after some interval.​

If the docker images are heavy, you could use a Windows NanoServer image, and install your own components in the docker image, just like how the busybox image has been created.

NOTE: ​This is not a complete solution, but more of an architectural guidance, which was more than enough for the client to implement the solution.

I am not good with scripts. If anyone would be interested in developing a complete solution, I would be more than happy to collaborate.