Connect to an external service on a different AKS cluster through private network
My goal is to call a service on an AKS cluster (aks1/US) from a pod on a second AKS cluster (aks2/EU).
These clusters will be on different regions and should communicate over a private network.
For the cluster networking I’m using the Azure CNI plugin.
Above you can see a schema of the two possible ending architectures. ExternalName or ExternalIP service on the US AKS pointing to a private EU ingress.
So, after some reading and some video listening, it seemed for me that the best option was to use an externalName service on AKS2 calling a service defined in a custom private DNS zone (ecommerce.private.eu.dev), being these two VNets peered before.
Address space for aks services:
- dev-vnet 10.0.0.0/14
- dev-test1-aks: v1.22.4, 1 node, dev-test1-vnet 11.0.0.0/16
- dev-test2-aks v1.22.4, 1 node dev-test2-vnet 11.1.0.0/16
After some trials I can get connectivity between pods networks but I was never able to reach the service network from the other cluster.
- I don’t have any active firewall
- I’ve peered all three networks: dev-test1-vnet, dev-test2-vnet, dev-vnet (AKS services network)
- I’ve create a Private DNS zones private.eu.dev where I’ve put the “ecommerce” A record (10.0.129.155) that should be resolved by the externalName service
I’ve also created inbound and outbound network policies for the AKS networks:
- on dev-aks (10.0/16) allow all incoming from 11.1/16 and 11.0/16
- on dev-test2-aks allow any outbound
Let’s do some scripting!
Are you able to see what is missing here? :)
If you deploy an nginx ingress controller a standard load balancer will be created (line 6). This LB will expose a public IP to the internet and an internal ClusterIP to the pods network.
But this internal IP is not routable! Meaning that it is not reachable from the Azure subnet, but only from the kubernetes control plane.
So to complete this architecture what you need is to expose this load balancer as an internal load balancer, adding an AKS annotation.
kubectl — context=dev-test1-aks patch service -n ingress-nginx ingress-nginx-controller — patch ‘{“metadata”: {“annotations”: {“service.beta.kubernetes.io/azure-load-balancer-internal”: “tr ue”}}}’
Here’s the results:
Sources:
