Running Istio on GKE Autopilot

GKE Autopilot is announced so tested Istio on it.

This info is outdated

• Autopilot has more capabilities now
• See: https://wdenniss.com/byo-service-mesh-on-gke-autopilot

TL;DR

• As of Feb. 25 2021, Istio does not work on GKE Autopilot
• MutatingWebhookConfiguration cannot be created from end user
• Linux capabilities “NET_ADMIN” is not allowed

GKE Autopilot

GKE is a fully managed Kubernetes service on Google Cloud. GKE Autopilot was announced today. With Autopilot you don’t have to manage nodes. So node-related operations like upgrading, scaling or changing OS are not necessary.

Standard and Autopilot

If you try to create a new GKE cluster on Cloud Console, you will choose Autopilot and Standard now.

Limitations

For providing this feature, there’re some limitations at this moment.

Istio

Istio is an open-source service mesh software. See the official website for more detail.

Creating Autopilot Clusters

Let’s create the cluster first. If Autopilot option does not appear on Cloud Console, you can create via CLI.

Like this,

gcloud container clusters create-auto test-autopilot --region asia-northeast1

This is only for testing, for production-level usage, consider specifying VPCs.

Installing Istio

Next step is installing Istio. Istio have three options for this: istioctl, operator and Helm. This time I used istioctl install as this is the easiest.

istioctl install

The command fails.

This will install the Istio 1.9.0  profile with ["Istio core" "Istiod" "Ingress gateways"] components into the cluster. Proceed? (y/N) y
✔ Istio core installed
- Processing resources for Istiod.                                                                                                                                                                                                                                                                                                           2021-02-25T03:44:11.256246Z     error   installer       failed to update resource with server-side apply for obj MutatingWebhookConfiguration//istio-sidecar-injector: mutatingwebhookconfigurations.admissionregistration.k8s.io "istio-sidecar-injector" is forbidden: User "[User Email]" cannot patch resource "mutatingwebhookconfigurations" in API group "admissionregistration.k8s.io" at the cluster scope: GKEAutopilot authz: cluster scoped resource "mutatingwebhookconfigurations/" is managed and access is denied
✘ Istiod encountered an error: failed to update resource with server-side apply for obj MutatingWebhookConfiguration//istio-sidecar-injector: mutatingwebhookconfigurations.admissionregistration.k8s.io "istio-sidecar-injector" is forbidden: User "[User Email]" cannot patch resource "mutatingwebhookconfigurations" in API group "admissionregistration.k8s.io" at the cluster scope: GKEAutopilot authz: cluster scoped resource "mutatingwebhookconfigurations/" is managed and access is denied
✔ Ingress gateways installed
- Pruning removed resources                                                                                                                                                                                                                                                                                                                  2021-02-25T03:46:55.287874Z     warn    installer       retrieving resources to prune type admissionregistration.k8s.io/v1beta1, Kind=MutatingWebhookConfiguration: mutatingwebhookconfigurations.admissionregistration.k8s.io is forbidden: User "[User Email]" cannot list resource "mutatingwebhookconfigurations" in API group "admissionregistration.k8s.io" at the cluster scope: GKEAutopilot authz: cluster scoped resource "mutatingwebhookconfigurations/" is managed and access is denied not found
Error: failed to install manifests: errors occurred during operation

For sidecar injection, installer tries to create a MutatingWebhookConfiguration called istio-sidecar-injector. This limitation is mentioned here.

FYI: run the following for sidecar injection mechanism.

kubectl get MutatingWebhookConfiguration istio-sidecar-injector -o yaml

This is okay anyway as it is possible to manually inject envoy proxy.

Istiod and Ingress Gateway looks fine

istiod, a control plane, and istio-ingressgateway is running normally.

GCLB is also provisioned.

But looking at the istiod’s log, istiod is failing creatign configmap in kube-system. Autopilot does not allow modifing resources inside kube-system.

2021-02-25T05:53:36.234474Z     info    Work item handle failed (error when creating configmap istio-ca-root-cert: configmaps is forbidden: User "system:serviceaccount:istio-system:istiod-service-account" cannot create resource "configmaps" in API group "" in the namespace "kube-system": GKEAutopilot authz: the namespace "kube-system" is managed and the request's verb "create" is denied), retry after delay 1s

Deploying Apps

Let’s deploy apps. I don’t think it is going to work though anyway. Here I’m going to deploy sample “Book Application”. As sidecar injection is not available, adding sidecar into the manifest is required before applying using istioctl kube-inject.

kubectl apply -f <(istioctl kube-inject -f samples/bookinfo/platform/kube/bookinfo.yaml)

istio-init doesn’t work

As a result it failed. NET_ADMIN, in the linux capabilities, is forbidden, which is documented here.

Error from server ([denied by autogke-default-linux-capabilities] linux capability <{"NET_ADMIN"}> on container <istio-init> not allowed; Autopilot only allows the capabilities: <["SETPCAP", "MKNOD", "AUDIT_WRITE", "CHOWN", "NET_RAW", "DAC_OVERRIDE", "FOWNER", "FSETID", "KILL", "SETGID", "SETUID", "NET_BIND_SERVICE", "SYS_CHROOT", "SETFCAP"]>. Requesting user: <[User Email]> and groups: <["system:authenticated"]>): error when creating "/dev/fd/63": admission webhook "validation.gatekeeper.sh" denied the request: [denied by autogke-default-linux-capabilities] linux capability <{"NET_ADMIN"}> on container <istio-init> not allowed; Autopilot only allow...

Here’s what istio-init is doing. In this initContainers, istio is trying to change traffic into/from application container to envoy. Due to the linux capabilities limitations this is failing.

      initContainers:
      - args:
        - istio-iptables
        - -p
        - "15001"
        - -z
        - "15006"
        - -u
        - "1337"
        - -m
        - REDIRECT
        - -i
        - '*'
        - -x
        - ""
        - -b
        - '*'
        - -d
        - 15090,15021,15020
        image: docker.io/istio/proxyv2:1.9.0
        imagePullPolicy: Always
        name: istio-init

Conclusions

• Istio cannot be used with GKE Autopilot at this moment.
• Still Autopilot is really nice in a way that user is free from node management.
• Looking forward to the further product improvements!