Making Istio Work with Kubernetes StatefulSet and Headless Services
Here at Airy, we use Istio on Kubernetes to monitor and control the traffic of our microservice workloads. Istio is a tool to manage Service Meshes in Kubernetes. In Kubernetes, we can deploy stateful workloads such time-series databases like Prometheus. Stateful workloads often need to reach individual machines directly without load balancing.
Using the standard Kubernetes services will lead to problems for these use cases, as Kubernetes will load-balance the traffic to the stateful machines. We typically use Headless Services, so that the clients can reach individual machines without going through Kubernetes’ load balancing. But, Istio by default is incompatible with Headless Services.
In this article, we will explain how we tune our Istio resources to work with Kubernetes’ Headless Services, with Thanos as an example. We encountered the problem when we were integrating Thanos with our Prometheus monitoring service. Thanos has a component called thanos-querier, which needs to be able to reach individual Prometheus pods running in multiple clusters both for internal-internal (East-West) and external-internal (North-South) traffics.
East-West Traffic
The key resource for Ease-West Traffic is Istio’s ServiceEntry resource. We have to manually set up a ServiceEntry resource to let Istio’s Service Registry know where and how to reach a defined service. Here are the Headless Service and ServiceEntry manifests:
Apply the manifest with `kubectl apply -f FILE.yaml`
In the ServiceEntry specification, we specify the spec.resolution with DNS, to tell Istio to resolve the service’s IP using the cluster’s internal DNS, and spec.location with MESH_INTERNAL to specify that these are cluster’s internal services. We then specify the ports and how to access services in each port.
North-South Traffic
Fundamentally, North-South Traffic is traffic that goes from outside the cluster to the Istio Ingress Gateway Service. Traffic will then proxied from Ingress Gateway to the actual services. In this point, the traffic is actually just East-West traffic, going from your Ingress Gateway Pods to your actual service’s pods, so the tasks we have done above applies to the North-South traffic too. Besides that, we also have to specify VirtualServices and Gateways for outside traffic to reach the individual pods. To do that, we create VirtualService and Gateway manifests:
After applying the manifest, we can reach individual pods through the “ingressgateway” with the specified hosts, cluster-A-prometheus-0.example.com and cluster-A-prometheus-1.example.com. Make sure you register them to your DNS registry, pointing the Ingress’ ELB/LoadBalancer, or to your own reverse proxies proxying to the Ingress if you use them. This method works on any other Stateful workloads that need to be reached individually using headless services!
Caveats
A tradeoff with this solution is operational scalability, we have to manually (or just automate it FTW) create the unique manifests (ServiceEntry, VirtualService, and Gateway) for each additional pod of the service. Istio also doesn’t seem to support this in the near future, as this would affect Istio’s scale, so we’ll stick with this method!
