The new Kubernetes Gateway API with Istio and Anthos Service Mesh (ASM)

Update on April 22nd, 2024 — the Kubernetes Gateway API version 1.0.0 (GA) is now supported by GKE Gateway API! 🎉 — officially announced on May 2nd.

Update on November 2nd, 2023 — the Kubernetes Gateway API is now GA with its version 1.0.0 just announced! 🎉 While the GKE Gateway API is still using the version 0.7.0.

Introduction — Kubernetes Gateway API (k8s.io)

Gateway API is an open source project managed by the SIG-NETWORK community. It is an API (collection of resources) that model service networking in Kubernetes. These resources — GatewayClass, Gateway, HTTPRoute, TCPRoute, etc., as well as the Kubernetes Service resource - aim to evolve Kubernetes service networking through expressive, extensible, and role-oriented interfaces that are implemented by many vendors and have broad industry support.

The new Gateway APIs aim to take the learnings from various Kubernetes ingress implementations, including Istio, to build a standardized vendor neutral API.

Google Kubernetes Engine Gateway controller is now GA | Google Cloud Blog

The GKE Gateway controller, already GA, is Google Cloud’s implementation of the Gateway API for GKE clusters:

It is a fully managed service with deep integration with GKE, enabling platform operators to manage internal and external HTTP(S) load balancing using an expressive and extensible API.

In your cluster, you may now have 2 different type of Gateways:

• Istio Gateway :

kubectl get gateways.networking.istio.io -A
kubectl get gw -A

• Kubernetes Gateway:

kubectl get gateways.gateway.networking.k8s.io -A
kubectl get gtw -A

The latter is the one we will use throughout this blog post, that’s the new Kubernetes Gateway API.

What we will cover in this blog post

In this blog post we will:

• Create a GKE cluster with the managed ASM and the Gateway API enabled.
• Option 1: Deploy a Google Service Mesh Cloud Gateway (asm-l7-gxlb) to automatically deploy an Istio ingress gateway.
• Deploy the Online Boutique sample apps in our Mesh exposed by this ASM Gateway.
• Option 2: Deploy a GKE Gateway (gke-l7-global-external-managed) and an Istio Gateway (istio), and manually deploy an Istio ingress gateway.
• Secure the GKE Gateway using TLS/HTTPS
• Protect the GKE Gateway behind Cloud Armor (DDOS and WAF protection)

Note: this Option 2 is now the one taken by the official Google Cloud doc: From edge to mesh: Deploy service mesh applications through GKE Gateway | Cloud Architecture Center | Google Cloud.

The show must go on!

Let’s do the usual GKE and ASM setup:

PROJECT_ID=FIXME-WITH-YOUR-PROJECT_ID
CLUSTER=cluster-with-gtw
ZONE=northamerica-northeast1-a

PROJECT_NUMBER=$(gcloud projects describe ${PROJECT_ID} --format='get(projectNumber)')

gcloud services enable container.googleapis.com \
    gkehub.googleapis.com \
    anthos.googleapis.com \
    mesh.googleapis.com

gcloud container clusters create ${CLUSTER} \
    --zone ${ZONE} \
    --machine-type=e2-standard-4 \
    --workload-pool ${PROJECT_ID}.svc.id.goog \
    --gateway-api=standard \
    --labels mesh_id=proj-${PROJECT_NUMBER}

gcloud container fleet memberships register ${CLUSTER} \
    --gke-cluster ${ZONE}/${CLUSTER} \
    --enable-workload-identity

gcloud container fleet mesh enable

gcloud container fleet mesh update \
    --management automatic \
    --memberships ${CLUSTER}

The only difference to notice is the use of the --gateway-api=standard parameter to install the GKE Gateway controller when creating the GKE cluster. If you create new Autopilot clusters on GKE 1.26 and later, Gateway API is enabled by default.

We could see that the version of the Gateway API used is 1.0.0 (which is the latest and GA version upstream as we speak).

kubectl get crd gateways.gateway.networking.k8s.io -o yaml | grep "gateway.networking.k8s.io/bundle-version"

You need to have your GKE cluster upgraded to the 1.29.3-gke.1093000 version. With earlier version you will be with version 0.8.0 and you will need to use v1beta1 instead of v1 for the following resources: GatewayClass, Gateway, HTTPRoute.

If you are just using Istio, not ASM, as an alternative, the Gateway API CRDs could be installed manually.

If we are running this command kubectl get gatewayclasses.gateway.networking.k8s.io, we could see all the different GatewayClasses in our cluster:

gke-l7-global-external-managed networking.gke.io/gateway
gke-l7-gxlb networking.gke.io/gateway
gke-l7-regional-external-managed networking.gke.io/gateway
gke-l7-rilb networking.gke.io/gateway
istio istio.io/gateway-controller

Let’s use the Google Service Mesh Cloud Gateway (asm-l7-gxlb)!

Instead of using one of the gke-l7-* GatewayClasses , we will use the Google Service Mesh Cloud Gateway (asm-l7-gxlb). It is still in Preview, not GA yet, but worth a try!

cat << EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: asm-l7-gxlb
spec:
  controllerName: mesh.cloud.google.com/gateway
EOF

Let’s now create an actual Istio ingress gateway!

Create a dedicated asm-gateway Namespace:

kubectl create namespace asm-gateway

Deploy a Kubernetes Gateway resource attached to the gatewayClassName: asm-l7-gxlb:

cat << EOF | kubectl apply -f -
kind: Gateway
apiVersion: gateway.networking.k8s.io/v1
metadata:
  name: asm-gateway
  namespace: asm-gateway
spec:
  gatewayClassName: asm-l7-gxlb
  listeners:
  - name: http
    protocol: HTTP
    port: 80
    allowedRoutes:
      namespaces:
        from: All
EOF

Note: In this setup, we are using the shared Gateway topology: Gateway | Google Kubernetes Engine (GKE) | Google Cloud.

Like you can see, no need to deploy our own Deployment or Service for the Istio ingress gateway, they are automatically deployed for us because we use the gatewayClassName: asm-l7-gxlb.

That’s based on the Istio implementation like described here: Istio / Kubernetes Gateway API.

Let’s see what’s deployed by running this command: kubectl get all -n asm-gateway:

NAME
pod/asm-gw-istio-asm-gateway-7f7795554-dgdmm

NAME                               TYPE
service/asm-gw-istio-asm-gateway   ClusterIP

NAME
deployment.apps/asm-gw-istio-asm-gateway

NAME
replicaset.apps/asm-gw-istio-asm-gateway-7f7795554

Note: The asm-gateway Namespace doesn’t have the istio-injection=enabled label, instead we could see that the generated Deployment asm-gw-istio-asm-gateway has the sidecar.istio.io/inject=true label in order to inject the Istio proxy.

We could also see what the asm-l7-gxlb Gateway is generating by running this command kubectl get gtw -n asm-gateway:

NAME                       CLASS         ADDRESS                                                     PROGRAMMED
asm-gateway                asm-l7-gxlb                                                               Unknown
asm-gw-gke-asm-gateway     gke-l7-gxlb   34.36.35.79                                                 True
asm-gw-istio-asm-gateway   istio         asm-gw-istio-asm-gateway.asm-gateway.svc.cluster.local:80   Unknown

• A gke-l7-gxlb Kubernetes Gateway which is a Classic Application Load Balancer (EXTERNAL) instead of the new Global External Application Load Balancer (EXTERNAL_MANAGED).

gcloud compute backend-services describe \
    gkegw1-pp3a-asm-gateway-asm-gw-istio-asm-gatewa-80-ttuj7gahk852 \
    --global \
    --format='get(loadBalancingScheme)'

• An istio Kubernetes Gateway has been generated pointing to the Istio ingress gateway Service automatically deployed in the asm-gateway Namespace.

Let’s deploy the Online Boutique sample apps in our Mesh exposed by this ASM Gateway!

Deploy the Online Boutique in its own Namespace:

kubectl create namespace onlineboutique
kubectl label namespace onlineboutique istio-injection=enabled
helm upgrade onlineboutique oci://us-docker.pkg.dev/online-boutique-ci/charts/onlineboutique \
    --install \
    -n onlineboutique \
    --set frontend.externalService=false

Expose it via an HTTPRoute to bind it to the asm-gateway Kubernetes Gateway:

cat << EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: frontend
  namespace: onlineboutique
spec:
  parentRefs:
  - kind: Gateway
    name: asm-gateway
    namespace: asm-gateway
  rules:
  - backendRefs:
    - name: frontend
      port: 80
EOF

Let’s hit the public IP of the Kubernetes Gateway:

INGRESS_IP=$(kubectl get gtw asm-gw-gke-asm-gateway \
    -n asm-gateway \
    -o=jsonpath="{.status.addresses[0].value}")

echo -e "http://${INGRESS_IP}"

🎉That’s it, congrats! 🎉

You just exposed the Online Boutique sample apps behind the new Kubernetes Gateway API via the gatewayClassName asm-l7-gxlb!

Once we generate enough traffic, we could also see the associated Service Mesh topology:

This Google Service Mesh Cloud Gateway is still in Preview as we speak. It’s very promising. And there are some limitations to be aware of: Configure external HTTP(S) Load Balancing for managed Anthos Service Mesh | Google Cloud.

What about deploying manually the Istio ingress gateway?

Now, what if you want to customize the Istio ingress gateway generated by the Google Service Mesh Cloud Gateway? For example, the default Deployment doesn’t have a seccompProfile nor a dedicated ServiceAccount. No Role, RoleBinding, HPA, etc. are generated either. So you may want to have control over this by manually deploying this setup.

You may also want to use the Global External Application Load Balancer instead of the Classic Application Load Balancer used by default today.

Let’s see in action how to achieve this!

First, manually deploy an Istio ingress gateway:

INGRESS_NAMESPACE=gke-gateway
kubectl create namespace ${INGRESS_NAMESPACE}
kubectl label namespace ${INGRESS_NAMESPACE} istio-injection=enabled
cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: asm-ingressgateway
  namespace: ${INGRESS_NAMESPACE}
spec:
  selector:
    matchLabels:
      asm: ingressgateway
  template:
    metadata:
      annotations:
        inject.istio.io/templates: gateway
      labels:
        asm: ingressgateway
    spec:
      containers:
      - name: istio-proxy
        image: auto
        env:
        - name: ISTIO_META_UNPRIVILEGED_POD
          value: "true"
        ports:
        - containerPort: 8080
          protocol: TCP
        resources:
          limits:
            cpu: 2000m
            memory: 1024Mi
          requests:
            cpu: 100m
            memory: 128Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
              - all
          privileged: false
          readOnlyRootFilesystem: true
      securityContext:
        fsGroup: 1337
        runAsGroup: 1337
        runAsNonRoot: true
        runAsUser: 1337
EOF
cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: asm-ingressgateway
  namespace: ${INGRESS_NAMESPACE}
  labels:
    asm: ingressgateway
spec:
  ports:
  - name: http
    port: 80
    targetPort: 8080
  selector:
    asm: ingressgateway
  type: ClusterIP
EOF

Second, create a GKE Kubernetes Gateway:

cat << EOF | kubectl apply -f -
kind: Gateway
apiVersion: gateway.networking.k8s.io/v1
metadata:
  name: gke-gateway
  namespace: ${INGRESS_NAMESPACE}
spec:
  gatewayClassName: gke-l7-global-external-managed
  listeners:
  - name: http
    protocol: HTTP
    port: 80
    allowedRoutes:
      namespaces:
        from: All
EOF

Third, create an Istio Kubernetes Gateway pointing to the Istio ingress gateway’s Service by using the Hostname type:

cat << EOF | kubectl apply -f -
kind: Gateway
apiVersion: gateway.networking.k8s.io/v1
metadata:
  name: istio-gateway
  namespace: ${INGRESS_NAMESPACE}
spec:
  gatewayClassName: istio
  addresses:
  - value: asm-ingressgateway.${INGRESS_NAMESPACE}.svc.cluster.local
    type: Hostname
  listeners:
  - name: http
    protocol: HTTP
    port: 80
    allowedRoutes:
      namespaces:
        from: All
EOF

Fourth, create an HTTPRoute to bind the 2 Gateways:

cat << EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: asm-ingressgateway
  namespace: ${INGRESS_NAMESPACE}
spec:
  parentRefs:
  - kind: Gateway
    name: gke-gateway
    namespace: ${INGRESS_NAMESPACE}
  rules:
  - backendRefs:
    - kind: Service
      name: asm-ingressgateway
      port: 80
EOF

Finally, we need to bind our Online Boutique frontend app to the istio-gateway Gateway via an HTTPRoute:

cat << EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: frontend
  namespace: onlineboutique
spec:
  parentRefs:
  - kind: Gateway
    name: istio-gateway
    namespace: ${INGRESS_NAMESPACE}
  rules:
  - backendRefs:
    - kind: Service
      name: frontend
      port: 80
EOF

Let’s hit the public IP of the gke-gateway Gateway:

INGRESS_IP=$(kubectl get gtw gke-gateway \
    -n ${INGRESS_NAMESPACE} \
    -o=jsonpath="{.status.addresses[0].value}")

echo -e "http://${INGRESS_IP}"

🎉 That’s it, congrats! 🎉

You just exposed the Online Boutique sample apps behind the new Kubernetes Gateway API via a manual deployment of the Istio ingress gateway with two gatewayClassNames gke-l7-global-external-managed and istio!

Once we generate enough traffic, we could also see the associated Service Mesh topology:

Now, what about securing your Gateway!?

In this section, we will secure our GKE Gateway by exposing it via HTTPS. In the below example, we will store the TLS certificate as Kubernetes Secret, but you could also use Certificate Manager or SSL certificate too.

Create a static IP address:

GATEWAY_IP_NAME=gke-gateway-ip
gcloud compute addresses create ${GATEWAY_IP_NAME} \
    --global

Get the newly created static IP address:

GATEWAY_IP=$(gcloud compute addresses describe ${GATEWAY_IP_NAME} \
    --global \
    --format "value(address)")
echo ${GCLB_IP}

Create a DNS, attached to the static IP address:

DNS="frontend-onlineboutique.endpoints.${PROJECT_ID}.cloud.goog"
cat <<EOF > dns-spec.yaml
swagger: "2.0"
info:
  description: "Cloud Endpoints DNS"
  title: "Cloud Endpoints DNS"
  version: "1.0.0"
paths: {}
host: "${DNS}"
x-google-endpoints:
- name: "${DNS}"
  target: "${GATEWAY_IP}"
EOF
gcloud endpoints services deploy dns-spec.yaml

Note: here we are using Google Cloud Endpoints to provision our DNS, but you can bring your own DNS. Also, we are reusing the IP address generated earlier by the Kubernetes Gateway, you may want to have a static IP address instead here.

Create a TLS cert for this DNS:

openssl genrsa -out ${DNS}.key 2048
openssl req -x509 \
    -new \
    -nodes \
    -days 365 \
    -key ${DNS}.key \
    -out ${DNS}.crt \
    -subj "/CN=${DNS}"

Create a Secret with this TLS cert:

kubectl create secret tls frontend-onlineboutique \
    -n asm-gateway \
    --key=${DNS}.key \
    --cert=${DNS}.crt

Update the GKE Gateway with the static IP address and this TLS cert Secret:

cat << EOF | kubectl apply -f -
kind: Gateway
apiVersion: gateway.networking.k8s.io/v1
metadata:
  name: gke-gateway
  namespace: ${INGRESS_NAMESPACE}
spec:
  gatewayClassName: gke-l7-global-external-managed
  listeners:
  - name: https
    protocol: HTTPS
    port: 443
    tls:
      mode: Terminate
      certificateRefs:
      - name: frontend-onlineboutique
  addresses:
  - type: NamedAddress
    value: ${GATEWAY_IP_NAME}
EOF

Update the frontend HTTPRoute:

cat << EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: frontend
  namespace: onlineboutique
spec:
  parentRefs:
  - kind: Gateway
    name: istio-gateway
    namespace: ${INGRESS_NAMESPACE}
  hostnames:
  - "${DNS}"
  rules:
  - backendRefs:
    - name: frontend
      port: 80
EOF

Create a dedicated HealthCheckPolicy to target the actual Istio ingress gateway proxy’s port on 15021 instead of 443:

cat << EOF | kubectl apply -f -
apiVersion: networking.gke.io/v1
kind: HealthCheckPolicy
metadata:
  name: asm-ingressgateway
  namespace: ${INGRESS_NAMESPACE}
spec:
  default:
    config:
      httpHealthCheck:
        port: 15021
        requestPath: /healthz/ready
      type: HTTP
  targetRef:
    group: ""
    kind: Service
    name: asm-ingressgateway
EOF

Let’s hit the DNS to test our website:

echo -e "https://${DNS}"

🎉 That’s it, congrats! 🎉

You just exposed the Online Boutique sample apps behind the new Kubernetes Gateway API via an HTTPS endpoint!

Protect the GKE Gateway behind Cloud Armor (WAF and DDOS protection)!

We could do more. We could be more secure. Let’s add a WAF and a DDOS protection on this HTTPS endpoint!

Create and configure a Cloud Armor policy with a WAF rule and DDOS protection:

gcloud compute security-policies create gke-gateway-security-policy

gcloud compute security-policies update gke-gateway-security-policy \
    --enable-layer7-ddos-defense

gcloud compute security-policies rules create 1000 \
    --security-policy gke-gateway-security-policy \
    --expression "evaluatePreconfiguredExpr('xss-v33-stable')" \
    --action "deny-403" \
    --description "XSS attack filtering"

Note: we are just using the Cross-site scripting rule here, you may want to use other rules in addition to that too: Google Cloud Armor preconfigured WAF rules overview.

Assign this security policy to the GKE Gateway:

cat << EOF | kubectl apply -f -
apiVersion: networking.gke.io/v1
kind: GCPBackendPolicy
metadata:
  name: asm-ingressgateway
  namespace: ${INGRESS_NAMESPACE}
spec:
  default:
    securityPolicy: gke-dev-security-policy
  targetRef:
    group: ""
    kind: Service
    name: asm-ingressgateway
EOF

🎉 That’s it, congrats! 🎉

You just secured the Online Boutique sample apps behind the new Kubernetes Gateway API via an HTTPS endpoint and Cloud Armor with a WAF and a DDOS protection!

In addition to that, more security features could be added:

• HTTP to HTTPS redirect
• Configure SSL Policies to secure client-to-load-balancer traffic
• Secure load balancer to application traffic using SSL or TLS version

That’s a wrap!

In this blog post we saw how you can use the new Kubernetes Gateway in the context of Istio, Anthos Service Mesh (ASM) and GKE. GKE Gateway is now GA while the Istio and ASM Gateways are still in beta/Preview. The latters allow to generate the Istio ingress Gateway (Deployment and Service) automatically, very convenient. We also how we can manually deploy our own Istio ingress gateway with the 2 Gateways: Istio and GKE. Finally, we were able to secure the public endpoint exposed by our Gateway via HTTPS and protect it behind Cloud Armor (WAF and DDOS protection).

Further resources

• Gateway API v1.0: GA Release | Kubernetes
• One API To Rule Them All? What the Gateway API Means For Service Meshes
• KubeCon + CloudNativeCon Europe 2023: Gateway API Project Update
• GatewayClass capabilities | Google Kubernetes Engine (GKE) | Google Cloud
• Multi-cluster Gateway controller for GKE is now GA | Google Cloud Blog
• Gateway API: Can I replace my Ingress Controller with Cilium? | Ogenki
• Introducing ingress2gateway; Simplifying Upgrades to Gateway API | Kubernetes
• From edge to mesh: Deploy service mesh applications through GKE Gateway | Cloud Architecture Center | Google Cloud

Hope you enjoyed this one! Stay safe out there and happy sailing! ⛵️