reCap: Kube vrs Cloud DNS in GKE

6 min readOct 15, 2022

When deciding on how to use DNS with GKE, what are the available native Kubernetes options, which options exist on Google Cloud for GKE, and how do these two things play together?

As we know every Service defined in the kubernetes cluster (including the DNS server itself) is assigned a DNS name. Kubernetes schedules a ‘DNS Pod’ and ‘Service’ on the cluster, and configures the kubelets to tell individual containers to use the DNS Service’s IP to resolve DNS names.

What objects get DNS records?

Services, “busy-service.a-subdomain.my-namespace.svc.cluster-domain.example" and
Pods 172-17-0-3.namespace.pod.cluster.local(cluster domain being local) (btw that is when you not set setHostnameAsFQDN: true in the Pod spec)

A Resolver and a Policy

Say if you are running CoreDNS as a Deployment, as a DNS resolver, it will typically be exposed as a Kubernetes Service with a static IP address.

**Name spaces** used by kube-dns service, and how it /etc/resolv.conf in a POD, in this example 10.100.0.10 is IP address of CoreDNS Pod aka **resolver IP address**. `And <namespace>.svc.cluster.local, svc.cluster.local, and cluster.local are the domains it resolves in your search.`

and a DNS policy (how to sequence the resolution) is specified in the dnsPolicy field of a Pod Spec.

“Default": The Pod inherits the name resolution configuration from the node that the Pods run on.
“ClusterFirst": Any DNS query that does not match the configured cluster domain suffix, such as "www.example.io", is forwarded to the upstream nameserver inherited from the node. In CoreDNS deployment a ‘forward plugin’ is used.
“ClusterFirstWithHostNet": For Pods running with hostNetwork, you should explicitly set its DNS policy "ClusterFirstWithHostNet".
“None": It allows a Pod to ignore DNS settings from the Kubernetes environment. All DNS settings are supposed to be provided using the dnsConfig field in the Pod Spec. See Pod's DNS config.

Kube DNS

Since Kubernetes deployments are abstracting kube-dns service object, lets review this service first. Btw kube-dns is the default DNS provider for GKE clusters, and it’s deployed as shown below.

GKE you can follow this guide for configuring custom autoscaling.

You can configure custom domain resolvers. Say, api.company.com, you can configure kube-dns to forward the resolving request to the external DNS Server using subdomains. Also, if you modify the ConfigMap for kube-dns to include upstreamNameservers, kube-dns forwards all DNS requests except *.cluster.local to those servers.

If you are experiencing high latency with DNS lookups, or frequent failures, you can choose one of the following options:

Enabling NodeLocal DNSCache.
Scaling up kube-dns.
Switching from kube-dns to Cloud DNS (Preview).

Cloud DNS

Cloud DNS provides Pod and Service DNS resolution without a cluster-hosted DNS provider like kube-dns.

The Pod `frontend` sends a DNS request for the IP address of the Service `backend` to the Compute Engine local metadata server at `169.254.169.254`. The metadata server runs locally on the node, sending cache misses to Cloud DNS data plane runs locally on a Compute instance.

The Cloud DNS controller automatically provisions DNS records for pods and services in Cloud DNS for ClusterIP, headless and external name services.

Now, this is where things get confusing a bit. Only Services with ClusterIPs (including headless or external name Services) are registered with Cloud DNS. The load balancer IP (Private or Public) or Ingress is NOT registered with Cloud DNS (as of this writing).

For resolve an existing private domains in Cloud DNS, you don’t have to configure anything extra. Make sure your private zone is configured to allow the VPC on which the GKE cluster is created to query it.

Using Cloud DNS as a DNS provider does not enable clients outside of a cluster to resolve and reach Kubernetes Services directly. You still need to expose your Services externally using a Load Balancer and register their cluster external IP addresses on your DNS infrastructure.

Before you use Cloud DNS with GKE you have to decide if you want to go with the VPC or Cluster Scope:

VPC Scope: DNS records are resolvable within the entire VPC. This option is GA.
Cluster Scope: DNS records are resolvable only within the Cluster. This option is in Preview for now.

above as well as Pre-GA Offerings Terms for Cloud DNS service bothers you, see Setting up a custom kube-dns Deployment, like Core DNS.

There are a couple of caveats you have to keep in mind when making the decision:

With VPC Scope, Services can overlap across clusters, so you have to set a custom domain for each cluster. Your Services FQDN will become svc.cluster.cluster1 and svc.cluster.cluster2 respectively.
Cluster Scope allows you to seamlessly migrate from kube-dns to Cloud DNS without worrying about overlapping Services.
Enabling Cloud DNS is a one way-operation, there is no way to go back to kube-dns. And you have to perform an upgrade on the node pools which will force the nodes to re-create.

Other patterns possible are,

Say if you have a StatefulSet, because VPC native clusters are routable inside the VPC, you can technically call the pod using its Cluster IP without the need for a load balancer (saving LB resources) And the DNS record will be updated if the pod changes IP.
Is same true with a Headless Service? Kubernetes Headless Services doesn’t create a ClusterIP. So what will happen?

You can also perform this type of resolution from other networks when connected to the VPC through Cloud Interconnect or Cloud VPN.

Other Options

You can also run a custom Deployment either CoreDNS or any other DNS provider that follows the Kubernetes DNS specifications.

External DNS

ExternalDNS synchronizes exposed Kubernetes Services and Ingresses with DNS providers. Unlike KubeDNS, however, it’s not a DNS server itself, but merely configures other DNS providers accordingly — e.g. AWS Route 53 or Google Cloud DNS.

External-dns is an open source controller for GKE. You can check the list of compatible providers on this page. The Google Cloud DNS support is stable.

Service Directory

Service Directory is a service discovery tool in Google Cloud. It allows you to publish and discover services across multiple environments and query them using gRPC, HTTP, or DNS. In Service Directory terminology, an endpoint is an individual IP/Port pair or an optional URL. endpoints are groups in Services that are grouped in namespaces.

Service Directory has many advantages, its independent & globally available service that can be reached from any compute environment (not just that VPC) and also from onPrem. Being global, it provides a single view of services across all of your Kubernetes deployments.

Refer to how to register your services with Service Directory for GKE.

Conclusions

Hope recap is helping to provide insight on options to be used in your GKE deployments based on your service discoverability needs. Kube-DNS is provided as simple and default option in GKE, addressing needs of single cluster and moderate query workloads. CloudDNS integrated with either Service Directory as an External DNS as more global and scalable alternatives for your enterprise.

Appendix

Kube-dns vrs CoreDNS

Although CoreDNS and Kube-dns ultimately perform the same task, there are some key differences in implementation that affect resource consumption and performance. At a high level, some of these differences are:

CoreDNS is a single container per instance, vs kube-dns which uses three.
Kube-dns uses dnsmasq for caching, which is single threaded C. CoreDNS is multi-threaded Go.
CoreDNS enables negative caching in the default deployment. Kube-dns does not.

These differences affect performance in various ways. The larger number of containers per instance in kube-dns increases base memory requirements, and also adds some performance overhead (as requests/responses need to be passed back and forth between containers). For kube-dns, dnsmasq may be highly optimized in C, but it’s also single threaded so it can only use one core per instance. CoreDNS enables negative caching, which aids in handling external names searches.

NodeLocal DNSCache

NodeLocal DNSCache is a Kubernetes native feature that is available also on GKE. Its runs a pod that caches DNS requests on the node, making DNS lookup faster and reducing the number of DNS queries to kube-dns or Cloud DNS.