how istio dns proxy improve dns performance, capabilities to resolve dns inter mesh cluster or outside mesh cluster, and distinguish multiple TCP services on the same port

Problem

1. istio has a limitation unable to distinguish multiple TCP services on the same port.
2. by default mesh cluster can’t resolve dns to other mesh cluster or outside mesh cluster(need workaround to configure kube-dns or resolv.conf on node).
3. high load kube-dns with increasing number of services and too many number of DNS lookup queries.

Goal

kubernetes cluster with service mesh istio can resolve dns inter mesh cluster and outside mesh cluster. Then improve dns query performance and reduced load on kube-dns.

Prerequisite

Prior to reading this article, it is recommended to read the article on Istio multicluster with istio-csr + cert-manager + vault PKI to deploy istio multicluster.

DNS Proxy Introduction

In addition to capturing application traffic, Istio can also capture DNS requests to improve the performance and usability of your mesh. When proxying DNS, all DNS requests from an application will be redirected to the sidecar, which stores a local mapping of domain names to IP addresses. ref

https://istio.io/v1.12/blog/2020/dns-proxy/dns-interception-in-istio.png

for a deeper understanding of DNS Proxy in istio, i recommend visiting the istio website or istio blog. in this article, i’m simply focusing on explaining the conclusion of DNS Proxy:

1. all DNS requests from an application will be redirected to the sidecar.
2. sidecar stores a local mapping of domain names to ip addresses.
3. can avoiding a roundtrip to the upstream DNS Server.
4. ServiceEntry addresses can be resolved without requiring custom configuration of a DNS Server, and will reduced load on kube-dns and increased performance.
5. increase performance dns resolution because the sidecar agent will detect the real hostname being queried within the first query and return a CNAME record to specific dns that the service want to query, as well as the A/AAA record for specific dns that the service want to query.
6. can distinguishing between multiple external TCP services same port with automatically allocate non-routable VIPs (Class E subnet) using ISTIO_META_DNS_AUTO_ALLOCATE configuration.
7. can resolve kubernetes services across cluster (istio multicluster)
8. VMs within mesh can DNS lookup queries to kubernetes services

Enabling istio DNS Proxy

This part is how to enable istio DNS Proxy. enable ISTIO_META_DNS_CAPTURE to enable DNS Proxy. if you want to enable automatically allocate non-routable VIPs, you can enable ISTIO_META_DNS_AUTO_ALLOCATE.

change line 203 to:

meshConfig:  
  defaultConfig:
    proxyMetadata:
      ISTIO_META_DNS_CAPTURE: "true"
      ISTIO_META_DNS_AUTO_ALLOCATE: "true"

Use Case

This Part explain 3 Use Case for istio DNS Proxy which include how to resolve dns inter and outside mesh cluster and multiple TCP service.

resolve kubernetes service inter cluster

step 1. create namespace sample with injected istio sidecar on both clusters.

kubectl create ns sample
kubectl label ns sample istio-injection=enabled

step 2. Create HelloWorld V1 service to istio-cluster1

apiVersion: v1
kind: Service
metadata:
  name: helloworld-v1
  namespace: sample
  labels:
    app: helloworld
    service: helloworld
spec:
  ports:
  - port: 5000
    name: http
  selector:
    app: helloworld

step 3. Create HelloWorld V2 service to istio-cluster2

apiVersion: v1
kind: Service
metadata:
  name: helloworld-v2
  namespace: sample
  labels:
    app: helloworld
    service: helloworld
spec:
  ports:
  - port: 5000
    name: http
  selector:
    app: helloworld

step 4. Deploy HelloWorld V1 apps to istio-cluster1

kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/helloworld/helloworld.yaml \
-l version=v1 -n sample

step 5. Deploy HelloWorld V2 apps to istio-cluster2

kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/helloworld/helloworld.yaml \
-l version=v2 -n sample

step 6. Deploy sleep apps to both clusters

kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/sleep/sleep.yaml \
-n sample

step 7. Send multiple request from sleep pod istio-cluster1 to helloworld-v2 service istio-cluster2.

for i in $(seq 100); do kubectl exec "$(kubectl get pod -n sample -l app=sleep -o jsonpath='{.items[0].metadata.name}')" -c sleep -n sample -- curl -s helloworld-v2.sample.svc.cluster.local:5000/hello; done

make sure that the helloworld version should v2

Hello version: v2, instance: helloworld-v2-848cbd7dc9-cc8tp
Hello version: v2, instance: helloworld-v2-848cbd7dc9-cc8tp

step 8. Send multiple request from sleep pod istio-cluster2 to helloworld-v1 service istio-cluster1.

for i in $(seq 100); do kubectl exec "$(kubectl get pod -n sample -l app=sleep -o jsonpath='{.items[0].metadata.name}')" -c sleep -n sample -- curl -s helloworld-v1.sample.svc.cluster.local:5000/hello; done

make sure that the helloworld version should v1

Hello version: v1, instance: helloworld-v1-5d868b5577-6bznq
Hello version: v1, instance: helloworld-v1-5d868b5577-6bznq

multiple external service with TCP

step 1. create ServiceEntry for external database

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
 name: database-1
spec:
 hosts:
 - database-1.xxxxxxxxxxxx.ap-southeast-1.rds.amazonaws.com
 ports:
 - number: 3306
   name: tcp
   protocol: TCP
 resolution: DNS

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
 name: database-2
spec:
 hosts:
 - database-2.xxxxxxxxxxxx.ap-southeast-1.rds.amazonaws.com
 ports:
 - number: 3306
   name: tcp
   protocol: TCP
 resolution: DNS

step 2. running istioctl command to list listeners

istioctl proxy-config listeners deploy/sleep | grep database

240.240.0.1 3306  ALL                                                     Cluster: outbound|3306||database-1.xxxxxxxxxxxx.ap-southeast-1.rds.amazonaws.com
240.240.0.2 3306  ALL                                                     Cluster: outbound|3306||database-2.xxxxxxxxxxxx.ap-southeast-1.rds.amazonaws.com

you’ll notice that port 3306 supports multiple IPs without any conflicts, allowing you to avoid add multiple ports in the ServiceEntryfor external services using TCP. this is because database-1 and database-2 IPs are automatically assigned VIPs within the class E Subnet.

discovery Virtual Machines

step 1. add WorkloadEntry to vm want to join mesh.

apiVersion: networking.istio.io/v1beta1
kind: WorkloadEntry
metadata:
 name: vm1-10.128.15.211
 namespace: vm
spec:
 address: 10.134.10.252
 labels:
   app: vm1

step 2. add ServiceEntry for adding vm to istio service discovery.

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
 name: vm1
spec:
 hosts:
 - vm1.com
 ports:
 - number: 80
   name: http
   protocol: HTTP
 resolution: STATIC
 workloadSelector:
   labels:
     app: vm1

step 3. Send multiple request from sleep pod to vm1.com

for i in $(seq 100); do kubectl exec "$(kubectl get pod -n sample -l app=sleep -o jsonpath='{.items[0].metadata.name}')" -c sleep -n sample -- curl -s vm1.com; done

conclusion

1. Using DNS Proxy istio can reduce load kube-dns and improve queries every time a lookup is made. this results improve resolution times because all DNS requests from an application will be redirected to the sidecar.
2. Application can resolve dns inter mesh cluster because sidecar stores a local mapping of domain names to ip addresses all mesh clusters. By default istio watches all namespaces, services, endpoints, and pods in a cluster whether they have a sidecar or not.
3. If the request can be handled by the sidecar, it will directly return a response to the application. if not, it forwards the query to the upstream name servers defined in /etc/resolv.conf.
4. Regarding points 2 and 3, when you enable discovery selector only to namespace with label injected istio sidecar, istio can’t watch service lacking istio sidecar or service/vm residing outside the mesh cluster. The impact is service within cluster can still communicate, but DNS Query forwarded to upstream name servers defined in /etc/resolv.conf.Not directly to sidecar. If service-to-service inter cluster, service can’t communicate because local mapping service not listed on sidecar, or name servers not defined in /etc/resolv.conf.