The new Kubernetes Gateway API with Istio and Anthos Service Mesh (ASM)
Update on April 22nd, 2024 — the Kubernetes Gateway API version 1.0.0 (GA) is now supported by GKE Gateway API! 🎉 — officially announced on May 2nd.
Update on November 2nd, 2023 — the Kubernetes Gateway API is now GA with its version 1.0.0 just announced! 🎉 While the GKE Gateway API is still using the version 0.7.0.
Gateway API is an open source project managed by the SIG-NETWORK community. It is an API (collection of resources) that model service networking in Kubernetes. These resources —
GatewayClass
,Gateway
,HTTPRoute
,TCPRoute
, etc., as well as the KubernetesService
resource - aim to evolve Kubernetes service networking through expressive, extensible, and role-oriented interfaces that are implemented by many vendors and have broad industry support.The new Gateway APIs aim to take the learnings from various Kubernetes ingress implementations, including Istio, to build a standardized vendor neutral API.
The GKE Gateway controller, already GA, is Google Cloud’s implementation of the Gateway API for GKE clusters:
It is a fully managed service with deep integration with GKE, enabling platform operators to manage internal and external HTTP(S) load balancing using an expressive and extensible API.
In your cluster, you may now have 2 different type of Gateways
:
- Istio
Gateway
:
kubectl get gateways.networking.istio.io -A
kubectl get gw -A
- Kubernetes
Gateway
:
kubectl get gateways.gateway.networking.k8s.io -A
kubectl get gtw -A
The latter is the one we will use throughout this blog post, that’s the new Kubernetes Gateway API.
What we will cover in this blog post
In this blog post we will:
- Create a GKE cluster with the managed ASM and the Gateway API enabled.
- Option 1: Deploy a Google Service Mesh Cloud Gateway (
asm-l7-gxlb
) to automatically deploy an Istio ingress gateway. - Deploy the Online Boutique sample apps in our Mesh exposed by this ASM Gateway.
- Option 2: Deploy a GKE Gateway (
gke-l7-global-external-managed
) and an Istio Gateway (istio
), and manually deploy an Istio ingress gateway. - Secure the GKE Gateway using TLS/HTTPS
- Protect the GKE Gateway behind Cloud Armor (DDOS and WAF protection)
Note: this Option 2 is now the one taken by the official Google Cloud doc: From edge to mesh: Deploy service mesh applications through GKE Gateway | Cloud Architecture Center | Google Cloud.
The show must go on!
Let’s do the usual GKE and ASM setup:
PROJECT_ID=FIXME-WITH-YOUR-PROJECT_ID
CLUSTER=cluster-with-gtw
ZONE=northamerica-northeast1-a
PROJECT_NUMBER=$(gcloud projects describe ${PROJECT_ID} --format='get(projectNumber)')
gcloud services enable container.googleapis.com \
gkehub.googleapis.com \
anthos.googleapis.com \
mesh.googleapis.com
gcloud container clusters create ${CLUSTER} \
--zone ${ZONE} \
--machine-type=e2-standard-4 \
--workload-pool ${PROJECT_ID}.svc.id.goog \
--gateway-api=standard \
--labels mesh_id=proj-${PROJECT_NUMBER}
gcloud container fleet memberships register ${CLUSTER} \
--gke-cluster ${ZONE}/${CLUSTER} \
--enable-workload-identity
gcloud container fleet mesh enable
gcloud container fleet mesh update \
--management automatic \
--memberships ${CLUSTER}
The only difference to notice is the use of the --gateway-api=standard
parameter to install the GKE Gateway controller when creating the GKE cluster. If you create new Autopilot clusters on GKE 1.26 and later, Gateway API is enabled by default.
We could see that the version of the Gateway API used is 1.0.0
(which is the latest and GA version upstream as we speak).
kubectl get crd gateways.gateway.networking.k8s.io -o yaml | grep "gateway.networking.k8s.io/bundle-version"
You need to have your GKE cluster upgraded to the 1.29.3-gke.1093000
version. With earlier version you will be with version 0.8.0
and you will need to use v1beta1
instead of v1
for the following resources: GatewayClass
, Gateway
, HTTPRoute
.
If you are just using Istio, not ASM, as an alternative, the Gateway API CRDs could be installed manually.
If we are running this command kubectl get gatewayclasses.gateway.networking.k8s.io
, we could see all the different GatewayClasses
in our cluster:
gke-l7-global-external-managed networking.gke.io/gateway
gke-l7-gxlb networking.gke.io/gateway
gke-l7-regional-external-managed networking.gke.io/gateway
gke-l7-rilb networking.gke.io/gateway
istio istio.io/gateway-controller
Let’s use the Google Service Mesh Cloud Gateway (asm-l7-gxlb)!
Instead of using one of the gke-l7-*
GatewayClasses
, we will use the Google Service Mesh Cloud Gateway (asm-l7-gxlb). It is still in Preview, not GA yet, but worth a try!
cat << EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: asm-l7-gxlb
spec:
controllerName: mesh.cloud.google.com/gateway
EOF
Let’s now create an actual Istio ingress gateway!
Create a dedicated asm-gateway
Namespace:
kubectl create namespace asm-gateway
Deploy a Kubernetes Gateway
resource attached to the gatewayClassName: asm-l7-gxlb
:
cat << EOF | kubectl apply -f -
kind: Gateway
apiVersion: gateway.networking.k8s.io/v1
metadata:
name: asm-gateway
namespace: asm-gateway
spec:
gatewayClassName: asm-l7-gxlb
listeners:
- name: http
protocol: HTTP
port: 80
allowedRoutes:
namespaces:
from: All
EOF
Note: In this setup, we are using the shared Gateway topology: Gateway | Google Kubernetes Engine (GKE) | Google Cloud.
Like you can see, no need to deploy our own Deployment
or Service
for the Istio ingress gateway, they are automatically deployed for us because we use the gatewayClassName: asm-l7-gxlb
.
That’s based on the Istio implementation like described here: Istio / Kubernetes Gateway API.
Let’s see what’s deployed by running this command: kubectl get all -n asm-gateway
:
NAME
pod/asm-gw-istio-asm-gateway-7f7795554-dgdmm
NAME TYPE
service/asm-gw-istio-asm-gateway ClusterIP
NAME
deployment.apps/asm-gw-istio-asm-gateway
NAME
replicaset.apps/asm-gw-istio-asm-gateway-7f7795554
Note: The asm-gateway
Namespace doesn’t have the istio-injection=enabled
label, instead we could see that the generated Deployment asm-gw-istio-asm-gateway
has the sidecar.istio.io/inject=true
label in order to inject the Istio proxy.
We could also see what the asm-l7-gxlb
Gateway is generating by running this command kubectl get gtw -n asm-gateway
:
NAME CLASS ADDRESS PROGRAMMED
asm-gateway asm-l7-gxlb Unknown
asm-gw-gke-asm-gateway gke-l7-gxlb 34.36.35.79 True
asm-gw-istio-asm-gateway istio asm-gw-istio-asm-gateway.asm-gateway.svc.cluster.local:80 Unknown
- A
gke-l7-gxlb
Kubernetes Gateway which is a Classic Application Load Balancer (EXTERNAL
) instead of the new Global External Application Load Balancer (EXTERNAL_MANAGED
).
gcloud compute backend-services describe \
gkegw1-pp3a-asm-gateway-asm-gw-istio-asm-gatewa-80-ttuj7gahk852 \
--global \
--format='get(loadBalancingScheme)'
- An
istio
Kubernetes Gateway has been generated pointing to the Istio ingress gatewayService
automatically deployed in theasm-gateway
Namespace.
Let’s deploy the Online Boutique sample apps in our Mesh exposed by this ASM Gateway!
Deploy the Online Boutique in its own Namespace:
kubectl create namespace onlineboutique
kubectl label namespace onlineboutique istio-injection=enabled
helm upgrade onlineboutique oci://us-docker.pkg.dev/online-boutique-ci/charts/onlineboutique \
--install \
-n onlineboutique \
--set frontend.externalService=false
Expose it via an HTTPRoute
to bind it to the asm-gateway
Kubernetes Gateway
:
cat << EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: frontend
namespace: onlineboutique
spec:
parentRefs:
- kind: Gateway
name: asm-gateway
namespace: asm-gateway
rules:
- backendRefs:
- name: frontend
port: 80
EOF
Let’s hit the public IP of the Kubernetes Gateway
:
INGRESS_IP=$(kubectl get gtw asm-gw-gke-asm-gateway \
-n asm-gateway \
-o=jsonpath="{.status.addresses[0].value}")
echo -e "http://${INGRESS_IP}"
🎉That’s it, congrats! 🎉
You just exposed the Online Boutique sample apps behind the new Kubernetes Gateway API via the gatewayClassName
asm-l7-gxlb
!
Once we generate enough traffic, we could also see the associated Service Mesh topology:
This Google Service Mesh Cloud Gateway is still in Preview as we speak. It’s very promising. And there are some limitations to be aware of: Configure external HTTP(S) Load Balancing for managed Anthos Service Mesh | Google Cloud.
What about deploying manually the Istio ingress gateway?
Now, what if you want to customize the Istio ingress gateway generated by the Google Service Mesh Cloud Gateway? For example, the default Deployment
doesn’t have a seccompProfile
nor a dedicated ServiceAccount
. No Role
, RoleBinding
, HPA
, etc. are generated either. So you may want to have control over this by manually deploying this setup.
You may also want to use the Global External Application Load Balancer instead of the Classic Application Load Balancer used by default today.
Let’s see in action how to achieve this!
First, manually deploy an Istio ingress gateway:
INGRESS_NAMESPACE=gke-gateway
kubectl create namespace ${INGRESS_NAMESPACE}
kubectl label namespace ${INGRESS_NAMESPACE} istio-injection=enabled
cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: asm-ingressgateway
namespace: ${INGRESS_NAMESPACE}
spec:
selector:
matchLabels:
asm: ingressgateway
template:
metadata:
annotations:
inject.istio.io/templates: gateway
labels:
asm: ingressgateway
spec:
containers:
- name: istio-proxy
image: auto
env:
- name: ISTIO_META_UNPRIVILEGED_POD
value: "true"
ports:
- containerPort: 8080
protocol: TCP
resources:
limits:
cpu: 2000m
memory: 1024Mi
requests:
cpu: 100m
memory: 128Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- all
privileged: false
readOnlyRootFilesystem: true
securityContext:
fsGroup: 1337
runAsGroup: 1337
runAsNonRoot: true
runAsUser: 1337
EOF
cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
name: asm-ingressgateway
namespace: ${INGRESS_NAMESPACE}
labels:
asm: ingressgateway
spec:
ports:
- name: http
port: 80
targetPort: 8080
selector:
asm: ingressgateway
type: ClusterIP
EOF
Second, create a GKE Kubernetes Gateway
:
cat << EOF | kubectl apply -f -
kind: Gateway
apiVersion: gateway.networking.k8s.io/v1
metadata:
name: gke-gateway
namespace: ${INGRESS_NAMESPACE}
spec:
gatewayClassName: gke-l7-global-external-managed
listeners:
- name: http
protocol: HTTP
port: 80
allowedRoutes:
namespaces:
from: All
EOF
Third, create an Istio Kubernetes Gateway
pointing to the Istio ingress gateway’s Service
by using the Hostname
type:
cat << EOF | kubectl apply -f -
kind: Gateway
apiVersion: gateway.networking.k8s.io/v1
metadata:
name: istio-gateway
namespace: ${INGRESS_NAMESPACE}
spec:
gatewayClassName: istio
addresses:
- value: asm-ingressgateway.${INGRESS_NAMESPACE}.svc.cluster.local
type: Hostname
listeners:
- name: http
protocol: HTTP
port: 80
allowedRoutes:
namespaces:
from: All
EOF
Fourth, create an HTTPRoute
to bind the 2 Gateways
:
cat << EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: asm-ingressgateway
namespace: ${INGRESS_NAMESPACE}
spec:
parentRefs:
- kind: Gateway
name: gke-gateway
namespace: ${INGRESS_NAMESPACE}
rules:
- backendRefs:
- kind: Service
name: asm-ingressgateway
port: 80
EOF
Finally, we need to bind our Online Boutique frontend
app to the istio-gateway
Gateway
via an HTTPRoute
:
cat << EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: frontend
namespace: onlineboutique
spec:
parentRefs:
- kind: Gateway
name: istio-gateway
namespace: ${INGRESS_NAMESPACE}
rules:
- backendRefs:
- kind: Service
name: frontend
port: 80
EOF
Let’s hit the public IP of the gke-gateway
Gateway
:
INGRESS_IP=$(kubectl get gtw gke-gateway \
-n ${INGRESS_NAMESPACE} \
-o=jsonpath="{.status.addresses[0].value}")
echo -e "http://${INGRESS_IP}"
🎉 That’s it, congrats! 🎉
You just exposed the Online Boutique sample apps behind the new Kubernetes Gateway API via a manual deployment of the Istio ingress gateway with two gatewayClassNames
gke-l7-global-external-managed
and istio
!
Once we generate enough traffic, we could also see the associated Service Mesh topology:
Now, what about securing your Gateway!?
In this section, we will secure our GKE Gateway
by exposing it via HTTPS. In the below example, we will store the TLS certificate as Kubernetes Secret
, but you could also use Certificate Manager or SSL certificate too.
Create a static IP address:
GATEWAY_IP_NAME=gke-gateway-ip
gcloud compute addresses create ${GATEWAY_IP_NAME} \
--global
Get the newly created static IP address:
GATEWAY_IP=$(gcloud compute addresses describe ${GATEWAY_IP_NAME} \
--global \
--format "value(address)")
echo ${GCLB_IP}
Create a DNS, attached to the static IP address:
DNS="frontend-onlineboutique.endpoints.${PROJECT_ID}.cloud.goog"
cat <<EOF > dns-spec.yaml
swagger: "2.0"
info:
description: "Cloud Endpoints DNS"
title: "Cloud Endpoints DNS"
version: "1.0.0"
paths: {}
host: "${DNS}"
x-google-endpoints:
- name: "${DNS}"
target: "${GATEWAY_IP}"
EOF
gcloud endpoints services deploy dns-spec.yaml
Note: here we are using Google Cloud Endpoints to provision our DNS, but you can bring your own DNS. Also, we are reusing the IP address generated earlier by the Kubernetes Gateway
, you may want to have a static IP address instead here.
Create a TLS cert for this DNS:
openssl genrsa -out ${DNS}.key 2048
openssl req -x509 \
-new \
-nodes \
-days 365 \
-key ${DNS}.key \
-out ${DNS}.crt \
-subj "/CN=${DNS}"
Create a Secret
with this TLS cert:
kubectl create secret tls frontend-onlineboutique \
-n asm-gateway \
--key=${DNS}.key \
--cert=${DNS}.crt
Update the GKE Gateway
with the static IP address and this TLS cert Secret
:
cat << EOF | kubectl apply -f -
kind: Gateway
apiVersion: gateway.networking.k8s.io/v1
metadata:
name: gke-gateway
namespace: ${INGRESS_NAMESPACE}
spec:
gatewayClassName: gke-l7-global-external-managed
listeners:
- name: https
protocol: HTTPS
port: 443
tls:
mode: Terminate
certificateRefs:
- name: frontend-onlineboutique
addresses:
- type: NamedAddress
value: ${GATEWAY_IP_NAME}
EOF
Update the frontend
HTTPRoute
:
cat << EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: frontend
namespace: onlineboutique
spec:
parentRefs:
- kind: Gateway
name: istio-gateway
namespace: ${INGRESS_NAMESPACE}
hostnames:
- "${DNS}"
rules:
- backendRefs:
- name: frontend
port: 80
EOF
Create a dedicated HealthCheckPolicy
to target the actual Istio ingress gateway proxy’s port on 15021
instead of 443
:
cat << EOF | kubectl apply -f -
apiVersion: networking.gke.io/v1
kind: HealthCheckPolicy
metadata:
name: asm-ingressgateway
namespace: ${INGRESS_NAMESPACE}
spec:
default:
config:
httpHealthCheck:
port: 15021
requestPath: /healthz/ready
type: HTTP
targetRef:
group: ""
kind: Service
name: asm-ingressgateway
EOF
Let’s hit the DNS to test our website:
echo -e "https://${DNS}"
🎉 That’s it, congrats! 🎉
You just exposed the Online Boutique sample apps behind the new Kubernetes Gateway API via an HTTPS endpoint!
Protect the GKE Gateway behind Cloud Armor (WAF and DDOS protection)!
We could do more. We could be more secure. Let’s add a WAF and a DDOS protection on this HTTPS endpoint!
Create and configure a Cloud Armor policy with a WAF rule and DDOS protection:
gcloud compute security-policies create gke-gateway-security-policy
gcloud compute security-policies update gke-gateway-security-policy \
--enable-layer7-ddos-defense
gcloud compute security-policies rules create 1000 \
--security-policy gke-gateway-security-policy \
--expression "evaluatePreconfiguredExpr('xss-v33-stable')" \
--action "deny-403" \
--description "XSS attack filtering"
Note: we are just using the Cross-site scripting rule here, you may want to use other rules in addition to that too: Google Cloud Armor preconfigured WAF rules overview.
Assign this security policy to the GKE Gateway:
cat << EOF | kubectl apply -f -
apiVersion: networking.gke.io/v1
kind: GCPBackendPolicy
metadata:
name: asm-ingressgateway
namespace: ${INGRESS_NAMESPACE}
spec:
default:
securityPolicy: gke-dev-security-policy
targetRef:
group: ""
kind: Service
name: asm-ingressgateway
EOF
🎉 That’s it, congrats! 🎉
You just secured the Online Boutique sample apps behind the new Kubernetes Gateway API via an HTTPS endpoint and Cloud Armor with a WAF and a DDOS protection!
In addition to that, more security features could be added:
That’s a wrap!
In this blog post we saw how you can use the new Kubernetes Gateway
in the context of Istio, Anthos Service Mesh (ASM) and GKE. GKE Gateway
is now GA while the Istio and ASM Gateways
are still in beta/Preview. The latters allow to generate the Istio ingress Gateway (Deployment
and Service
) automatically, very convenient. We also how we can manually deploy our own Istio ingress gateway with the 2 Gateways
: Istio and GKE. Finally, we were able to secure the public endpoint exposed by our Gateway
via HTTPS and protect it behind Cloud Armor (WAF and DDOS protection).
Further resources
- Gateway API v1.0: GA Release | Kubernetes
- One API To Rule Them All? What the Gateway API Means For Service Meshes
- KubeCon + CloudNativeCon Europe 2023: Gateway API Project Update
- GatewayClass capabilities | Google Kubernetes Engine (GKE) | Google Cloud
- Multi-cluster Gateway controller for GKE is now GA | Google Cloud Blog
- Gateway API: Can I replace my Ingress Controller with Cilium? | Ogenki
- Introducing ingress2gateway; Simplifying Upgrades to Gateway API | Kubernetes
- From edge to mesh: Deploy service mesh applications through GKE Gateway | Cloud Architecture Center | Google Cloud
Hope you enjoyed this one! Stay safe out there and happy sailing! ⛵️