Testing the Kubernetes Gateway API with the Istio-based service mesh add-on for Azure Kubernetes Service

Back in July 2022 the Kubernetes Gateway API graduated to beta. The Istio control plane supports Gateway API resources, and this support is eventually going to be the default API in the future.

Gateway API resource types, image from the Kubernetes Blog https://kubernetes.io/blog/2021/04/22/evolving-kubernetes-networking-with-the-gateway-api/

This article assumes you installed the Istio Service Mesh AKS add-on. A detailed example is available in my previous medium article.

Note: you can skip the classic Istio ingress gateway installation because it is not necessary when using Gateway API. However, if you install also the classic Istio ingress gateway you will have some extra pods but it will not prevent the Gateway API from working correctly.

Install the Gateway API custom resource definitions

To test Gateway API you need to install the necessary custom resource definitions:

kubectl get crd gateways.gateway.networking.k8s.io &> /dev/null || \
  { kubectl kustomize "github.com/kubernetes-sigs/gateway-api/config/crd?ref=v0.6.1" | kubectl apply -f -; }

This command will install the following crds:

gatewayclasses.gateway.networking.k8s.io
gateways.gateway.networking.k8s.io 
grpcroutes.gateway.networking.k8s.io 
httproutes.gateway.networking.k8s.io
referencegrants.gateway.networking.k8s.io 
tcproutes.gateway.networking.k8s.io 
tlsroutes.gateway.networking.k8s.io 
udproutes.gateway.networking.k8s.io

Deploy a workload

You will deploy the echoserver container to have a simple workload to test:

kubectl apply -f - <<EOF
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echoserver
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      run: echoserver
  template:
    metadata:
      labels:
        run: echoserver
    spec:
      containers:
      - name: echoserver
        image: gcr.io/google_containers/echoserver:1.10
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: echoserver
  namespace: default
spec:
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    run: echoserver
EOF

Expose the workload with Gateway API

With the classic Istio ingress gateway you would create a gateways.networking.istio.io resource with a spec.selector to match an existing Istio ingress gateway deployment. The information in the Gateway resource would be used to create a piece of Envoy configuration in the already running Istio ingress gateway pods.

Now you will use instead the gateways.gateway.networking.k8s.io resource. This will create a dedicated Deployment and Service for this gateway:

kubectl apply -f - <<EOF
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: echoserver
  namespace: default
  labels:
    istio.io/rev: asm-1-17
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: "/healthz/ready"
    service.beta.kubernetes.io/port_80_health-probe_protocol: http
    service.beta.kubernetes.io/port_80_health-probe_port: "15021"
spec:
  gatewayClassName: istio
  listeners:
  - name: http
    port: 80
    protocol: HTTP
    allowedRoutes:
      namespaces:
        from: All
EOF

To write the above Gateway definition, I started from the Istio bookinfo sample, and I added some important Azure specific details:

• The label istio.io/rev: asm-1-17 enables Gateway injection correctly for the Istio based service mesh add-on. The Deployment template comes with the annotation sidecar.istio.io/inject: “true” to make sure the pods are injected regardless of the labels on the namespace. However, the add-on uses Istio revisioned control plane so the istio.io/rev label is necessary to enable the pod injection. The Istio Pod injection is necessary because in the Deployment template the container image is set to image: auto and the Istio Pod injector adds the correct image reference.
• The Gateway annotations are propagated to the Kubernetes Serviceof type: LoadBalancer that will expose the Gateway pods with an Azure Load Balancer. I added the necessary Service annotations from the Azure LoadBalancer documentation to point the health checks, to the /healthz/ready Envoy endpoint on port 15021, that is the endpoint meant to check the health status of the Gateway.

After applying the Gateway you will see a new Deployment and a new Service both called echoserver-istio . This is a key difference from the classic Istio ingress gateway operating model:

• Classic Istio ingress gateway: the istio ingress gateway Deployment and Service are provisioned at the installation time and the pods run with an empty configuration until Gateway and VirtualService resources are created to generate the necessary Envoy configuration
• Gateway API: when the Gateway resource is created, the Istio control plane creates the Deployment and Service .

When using the classic Istio ingress gateway the VirtualService resource describes how a request is routed to a service. When using Gateway API you must use instead the HTTPRoute resource. Here the simplest configuration to send all the requests to the backend:

kubectl apply -f - <<EOF
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: echoserver
  namespace: default
spec:
  parentRefs:
  - name: echoserver
    namespace: default
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: echoserver
      port: 8080
EOF

The configuration is now complete and you can use curl to test if you can reach the echoserver Pod exposed correctly with Gateway API:

curl $(kubectl get service echoserver-istio -o=jsonpath='{.status.loadBalancer.ingress[0].ip}')

Conclusion

Istio supports Kubernetes Gateway API, and to use it on AKS you just need to add the necessary custom resource definitions.

You can use Kubernetes Gateway API to create Istio Gateways, or some of the other Gateway API implementations.

The Gateway resource needs the correct istio.io/rev label and the Azure Load Balancer annotations to configure the health checks.

Because Gateway API will be the default API for traffic management in Istio in the long term future, and it is also used in Istio Ambient for the Waypoint proxies, I strongly suggest to learn how it works and to get familiar with the Kubernetes Gateway API concepts.