A Deep Dive into Azure Kubernetes Service and Istio Ambient for sidecar-less microservices integration (part 1)

Published in

AT&T Israel Tech Blog

9 min readMay 18, 2023

Short introduction to Service Mesh:

“A Service Mesh is a dedicated infrastructure layer for making service-to-service communication safe, fast, and reliable”

Ref: https://linkerd.io/2017/04/25/whats-a-service-mesh-and-why-do-i-need-one/

I’ll add to this great definition also operational observability and ability to provide different traffic scenarios.

Sidecar or Sidecar-less?

There are a variety of service mesh solutions running on Kubernetes and previously they were all based on the sidecar approach.

Recently with the rise of popularity of eBPF , the latest announcements made by Solo.io and Google in September 2022 at the latest Kubecon 2023 regarding sidecar-less service mesh — Ambient from Istio and Cilium from Isovalent, I decided to give a try and run Istio Ambient sidecar-less service mesh on Azure AKS private cluster.

The pros of using sidecar-less are to set the connection between sidecar and its app container deployment free, save compute resources, upgrading service mesh do not require app pod restart and more.

Currently we are running another service mesh based on sidecars and if there an option to have the same or even better performance, and save costs it worth a try.

There is a short documentation provided by Solo.io about Istio Ambient hands-on implementation and additional topology articles, and on MSFT site there is only reference to sidecar solution, so this an interesting challenge was accepted.

Let’s switch back to service mesh architecture — in general we can say that service mesh comprises of two main planes — Data and Control planes.

While Control plane used to propagate configurations and security settings/policies (by various Kubernetes CRDs) through authentication, authorization to a data plane for establishing encrypted (by mTLS) communication between microservices, controlling traffic routing, monitoring, discovery, and registration of services.

Control plane typically comes with a management interface UI, API and CLI and that the operations/admins use to configure the service mesh.

Data plane is essentially the execution components such as all proxy components which do the actual work on the compute nodes like traffic routing, cert exchange, metrics collection and tracing.

Let’s look at how it appears in Istio Ambient

The Data plane (proxy layer) on Istio Ambient is separated into a 2-layer approach by taking care of different OSI model layers: Layer 4 (TCP) and Layer 7 (HTTP/HTTPS).

By using the 2-layer approach we can easily try to and implement each part of the new proxy components step-by-step.

The first part consists of “ztunnel” daemonsets deployed on each node and provide us with Layer4 secured connectivity include mTLS with authentication and authorization polices, metrics for observability and all that without terminating HTTP traffic or HTTP headers.

This agent is a zero-trust tunnel or “ztunnel” and its responsibility is to securely connect and authenticate microservices running within the service mesh.

Written in Rust, it should be lightweight and fast with a low footprint (initially Istio used Envoy but due to the xDS settings size it was replaced by a Rust component).

The configuration provided by the Istiod Control plane component to ztunnel is pushed using xDS(discovery service) API configuration settings.

After L4 connectivity was established using ztunnel and Istio-cni daemonsets on each node in the cluster, we can talk about L7 capabilities that are much more interesting for us as application owners.

The Istio-cni component is responsible to continuously detect which workloads pods are added to ambient mode and create and update iptables rules to run redirects for both directions (in and out) from these “ambient mode enabled” pods to node’s ztunnel agent.

The tunnel between “ztunnel” agents implement HBONE (HTTP Based Overlay Network Encapsulation) running on dedicated TCP port 15008 and uses mTLS and identity and authorization policies to send and receive traffic.

The identities of workloads are verified by using SPIFFE service identity format based on their service accounts (SPIFFE and SPIRE are also open-source projects under CNCF landscape):

spiffe://<trust-domain>/ns/<namespace>/sa/<service-account>

Using L7 features we can get by creating a “waypoint” proxy (based on Envoy) component for each workload service we want to expose to other services.

The “waypoint” proxy provides us with the following things: L7 authorization policy, HTTP observability (metrics/logs/traces) and rich set of traffic management features like canary and more.

So, if we want to see the whole picture of Layer 4 and Layer 7 components:

While there are 2 scenarios:

App A pod can reach App B pod directly via “ztunnel” (with HBONE encapsulation) on Layer 4, or if App B haves a “waypoint” proxy created then it will be routed there, get the routing policies settings, and reach App B pod accordingly.

After we have understood all Istio Ambient components, let’s start implementing it on Azure Kubernetes service.

What does our Azure environment look like?

As we are running on private AKS clusters, that means the Kube API and our misc web/tcp services are exposed to internal network via Azure Private-Endpoints (PLE for short).

In general, our Ingress network topology usually looks like:

The so-called “network” subscription contains different network components (proxies, firewalls and WAFs) to implement routing policies to and from the corporate network and Azure workload resources.

This layer holds significance for us only in terms of PLE connectivity toward our Kubernetes resources exposed to end-users such as Kube API and web services ingress.

We will start by creating new AKS cluster on an existing resource group and vnet range of 10.224.0.0/16.

$ az aks create -n aks-ambient -g test-rg -l eastus2 \
  --max-pods 80 \
  --network-plugin azure \
  --generate-ssh-keys \
  --enable-private-cluster \
  --disable-public-fqdn \
  --node-vm-size "Standard_D4s_v3" \
  --node-osdisk-size "40" \
  --node-count "2" \
  --enable-managed-identity \
  --enable-cluster-autoscaler \
  --max-count 5 \
  --min-count 1 \
  --node-osdisk-type Managed \
  --os-sku Ubuntu \
  --nodepool-name systempool

Then we’ll get the credentials and implemented PLE to our AKS Kube API and now we can reach our cluster via the kubectl command line.

These are our AKS nodes:

$ kubectl get nodes -o wide
NAME                                 STATUS   ROLES   AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
aks-systempool-14515445-vmss000000   Ready    agent   9m55s   v1.25.6   10.224.0.5    <none>        Ubuntu 22.04.2 LTS   5.15.0-1035-azure   containerd://1.6.18+azure-1
aks-systempool-14515445-vmss000001   Ready    agent   9m51s   v1.25.6   10.224.0.84   <none>        Ubuntu 22.04.2 LTS   5.15.0-1035-azure   containerd://1.6.18+azure-1

Our AKS namespaces:

$ kubectl get ns
NAME              STATUS   AGE
default           Active   12m
kube-node-lease   Active   12m
kube-public       Active   12m
kube-system       Active   12m

Following Istio Ambient docs and checking its github for latest version, we run the following command:

$ kubectl get crd gateways.gateway.networking.k8s.io &> /dev/null ||   { kubectl kustomize "github.com/kubernetes-sigs/gateway-api/config/crd/experimental?ref=v0.6.2" | kubectl apply -f -; }

The following resources were added to CRDs:

customresourcedefinition.apiextensions.k8s.io/gatewayclasses.gateway.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/gateways.gateway.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/grpcroutes.gateway.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/httproutes.gateway.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/referencegrants.gateway.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/tcproutes.gateway.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/tlsroutes.gateway.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/udproutes.gateway.networking.k8s.io created

These CRDs are added as Istio includes beta support for K8S Gateway API (created by K8S SIG Network community) and trying to make it the default API for traffic management in the future. Istio also supports using CRDs from different APIs (Istio and K8S Gateway) together.

Download the istioctl cli command tool:

$ wget https://github.com/istio/istio/releases/download/1.18.0-alpha.0/istioctl-1.18.0-alpha.0-linux-amd64.tar.gz
$ tar zxvfp istioctl-1.18.0-alpha.0-linux-amd64.tar.gz 
$ cp istioctl /usr/local/bin/
$ istioctl version
no ready Istio pods in "istio-system"
1.18.0-alpha.0

We now want to add Istio Ambient components with an ingress controller, as we generally do with Nginx ingress controller installed with Helm.

On the Azure side it creates the “kubernetes-internal” load balancer in the “MC_<rg_name>_<aks_name>_<region>” resource group (in our case MC_test-rg_aks-ambient_eastus2) which exposes HTTP/HTTPS ports 80,443 for our web microservices.

Install Istio Ambient on AKS

$ istioctl install --set profile=ambient --set components.ingressGateways[0].enabled=true --set components.ingressGateways[0].name=istio-ingressgateway --skip-confirmation --set components.ingressGateways[0].k8s.service."loadBalancerIP="10.224.0.180 --set components.ingressGateways[0].k8s.service.serviceAnnotations."service\.beta\.kubernetes\.io/azure-load-balancer-internal"=true

We provided an internal free IP address 10.224.0.180 from our range as frontend IP for the "kubernetes-internal" load balancer and the serviceAnnotations will create the load-balancer resource on Azure side.

The Istioctl command runs the Istio Operator on background and we can verify our settings provided in the command line on the IstioOperator.install.istio.io CRD named “installed-state”.

...
spec:
  components:
    base:
      enabled: true
    cni:
      enabled: true
    egressGateways:
      - enabled: false
        name: istio-egressgateway
    ingressGateways:
      - enabled: true
        k8s:
          service:
            loadBalancerIP: 10.224.0.180
            type: LoadBalancer
          serviceAnnotations:
            service.beta.kubernetes.io/azure-load-balancer-internal: true
        name: istio-ingressgateway
...

If we see that there is a problem to create it, or to run the istioctl command with this service annotation, we can run a shorter command and modify the service manifest of “istio-ingressgateway” later.

$ istioctl install --set profile=ambient --set components.ingressGateways[0].enabled=true --set components.ingressGateways[0].name=istio-ingressgateway --skip-confirmation --set components.ingressGateways[0].k8s.service."loadBalancerIP="10.224.0.180

The final service manifest of “istio-ingressgateway” should look like this one:

kind: Service
apiVersion: v1
metadata:
  name: istio-ingressgateway
  namespace: istio-system
...
  labels:
    app: istio-ingressgateway
...
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: 'true'
...
spec:
  ports:
    - name: status-port
      protocol: TCP
      port: 15021
      targetPort: 15021
      nodePort: 31893
    - name: http2
      protocol: TCP
      port: 80
      targetPort: 8080
      nodePort: 30672
    - name: https
      protocol: TCP
      port: 443
      targetPort: 8443
      nodePort: 30978
  selector:
    app: istio-ingressgateway
    istio: ingressgateway
  clusterIP: 10.0.228.152
  clusterIPs:
    - 10.0.228.152
  type: LoadBalancer
  sessionAffinity: None
  loadBalancerIP: 10.224.0.180
  externalTrafficPolicy: Cluster
  ipFamilies:
    - IPv4
  ipFamilyPolicy: SingleStack
  allocateLoadBalancerNodePorts: true
  internalTrafficPolicy: Cluster
status:
  loadBalancer:
    ingress:
      - ip: 10.224.0.180

Please check that your “kubernetes-internal” load balancer was created and contains 3 rules that use port 80,443 and 15021 (Istio-ingress health-check port).

Also check that the pods are up and running quickly:

$ kubectl get pods -n istio-system
NAME                                    READY   STATUS    RESTARTS   AGE
istio-cni-node-cfjjp                    1/1     Running   0          9m
istio-cni-node-q7gpp                    1/1     Running   0          9m
istio-ingressgateway-748bdc4999-gdddr   1/1     Running   0          9m
istiod-7596fdbb4c-hvd22                 1/1     Running   0          9m
ztunnel-g58zp                           1/1     Running   0          9m
ztunnel-k47q4                           1/1     Running   0          9m

# same for our services
$ kubectl get svc -n istio-system
NAME                   TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)                                      AGE
istio-ingressgateway   LoadBalancer   10.0.228.152   10.224.0.180   15021:31893/TCP,80:30672/TCP,443:30978/TCP   9m
istiod                 ClusterIP      10.0.156.231   <none>         15010/TCP,15012/TCP,443/TCP,15014/TCP        9m

After the “kubernetes-internal” load balancer (Istio Ingress controller) has been created successfully we can create a PLS (Private Link Service) and attach it to our load balancer “kubernetes-internal” frontend IP (in our case 10.224.0.180). Then create a PLE to this PLS and approve it on PLS side.

The whole network chain should look as follows:

Azure Load Balancer “kubernetes-internal” settings:

Let’s check our Istio Ingress Gateway listener status:

# istioctl proxy-config listener istio-ingressgateway-748bdc4999-gdddr -n istio-system
ADDRESS PORT  MATCH                     DESTINATION
        0     ALL                       Cluster: connect_originate
0.0.0.0 15021 ALL                       Inline Route: /healthz/ready*
0.0.0.0 15090 ALL                       Inline Route: /stats/prometheus*

When it created the Istio Ingress Gateway, the only port that really listens to connections is TCP 15021 (health-check port); other ports 80,443 are not activated until Gateway resource is applied.

Note: if you want to clear all Istio resources, run the following command:

$ istioctl uninstall -y –purge

Part 2 of this article: