Custom DNS Configuration with CoreDNS: AKS/Kubernetes

4 min readMay 8, 2020

Using on-premise DNS Servers for name resolution on AKS Pods by modifying CoreDNS

Recently I published an article for using Custom DNS Servers with AKS Pods (and Nodes).
There, the approach I took was to modify the DNS Settings of the VNet and set a DNS forwarder to Azure DNS.

In this article, I am going to cover how to modify the CoreDNS to use Custom DNS Servers.

Some theory of-course …

Modifying CoreDNS in AKS requires creation of a ConfigMap with a specific name ‘coredns-custom’ in the kube-system namespace.

The configuration from this custom ConfigMap is read by CoreDNS and the associated Corefile is modified to incorporate the new settings.

To setup usage of Custom DNS Server as a resolver for my custom domain (testabc.com), the ‘forward’ plugin for CoreDNS can be used.

Note that the earlier used ‘proxy’ plugin in now replaced by ‘forward’ plugin.

Architecture:

Request from within the Pods:

to custom domain testabc.com will go to the Custom DNS Server.
to all other domains will use the default configuration, i.e. use the /etc/resolv.conf file, which is derived from the host. In this specific case, since I have not modified the DNS Settings in the VNet where my AKS Nodes are present, the nodes use Azure DNS as the default DNS resolver.

The Custom DNS Server sitting in another VNet should be reachable from the AKS Node.

Setup

My AKS VNet settings point to the Azure DNS. That means the Nodes will use Azure DNS as the default resolver.
AKS VNet is peered to another VNet that contains the Custom DNS Server and a VM (VM1) whose hostname we will resolve.
CoreDNS version is 1.6.6. This is important as the plugin ‘forward’ was introduced in version 1.6.x and the earlier used ‘proxy’ plugin was removed.

Steps

Assuming you already have an AKS Cluster and the AKS Nodes are able to connect to the Custom DNS Server:

Write the ConfigMap to ‘forward’ queries to the Custom DNS Server (172.16.0.4 in my case)*if* the requested domain for DNS resolution from the Pod is my custom domain (testabc.com).
Update the custom ConfigMap that the CoreDNS refers to create additional configurations in the CoreFile
Restart the CoreDNS Pods to take in the new configuration
For all other queries, the default configuration of CoreDNS will be used which uses the DNS configuration of the Node.

Detailed Steps:

Write the ConfigMap named ‘mycoredns.yaml’

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  yourdns.server: |
    testabc.com:53 {
      errors
      cache 1
      forward . 172.16.0.4
    }

Apply the ConfigMap

kubectl apply -f <path_to_file/mycoredns.yaml>

Restart the CoreDNS Pods to take the new settings

kubectl delete pods -n kube-system -l k8s-app=kube-dns

Once the reboot is successful, the CoreDNS Pods will be updated with the new settings. The request flow will look something like this:

Verification

In order to verify the settings, I created an httpd pod, installed dnsutils on it, and tried to resolve both external and custom domain.

$ kubectl exec -it myapp /bin/bashroot@myapp:/usr/local/apache2# nslookup google.comServer:  10.0.0.10Address: 10.0.0.10#53Non-authoritative answer:Name: google.comAddress: 172.217.13.238Name: google.comAddress: 2607:f8b0:4004:808::200eroot@myapp:/usr/local/apache2# nslookup vm1.testabc.comServer:  10.0.0.10Address: 10.0.0.10#53Name: vm1.testabc.comAddress: 172.16.0.5

As can be noted, both custom and external domains are getting resolved as expected.

Footnotes

For reference, here is the default ConfigMap for CoreDNS:

kubectl describe cm -n kube-system coredns
Name:         coredns
Namespace:    kube-system
Labels:       addonmanager.kubernetes.io/mode=Reconcile
              k8s-app=kube-dns
              kubernetes.io/cluster-service=true
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"v1","data":{"Corefile":".:53 {\n    errors\n    health\n    kubernetes cluster.local in-addr.arpa ip6.arpa {\n      pods in...Data
====
Corefile:
----
.:53 {
    errors
    health
    kubernetes cluster.local in-addr.arpa ip6.arpa {
      pods insecure
      upstream
      fallthrough in-addr.arpa ip6.arpa
    }
    prometheus :9153
    forward . /etc/resolv.conf
    cache 30
    loop
    reload
    loadbalance
    import custom/*.override
}
import custom/*.serverEvents:  <none>

These changes, made to the DNS Settings for Pods through CoreDNS modification, will not modify the DNS Settings in the Nodes. The Nodes will still refer to the DNS Server configured in the VNet.

There are various plugins that are available with CoreDNS and they have a specific order of processing. More details can be found below: