Cluster local issue with Knative Eventing v0.9.0

Mete Atamel
Oct 14 · 3 min read

In my previous post, I talked about Knative v0.9.0 and some of the eventing changes in the latest release. I’ve been playing with Knative v0.9.0 since then to read Google Cloud Pub/Sub messages using PullSubscription and I ran into a rather fundamental issue that baffled me for a while. I’d like to outline the problem and the solution here, just in case it’s useful to others.

Knative Services as eventing sinks

In my PullSubscription, I could define Kubernetes Services as event sinks as follows:

apiVersion: pubsub.cloud.run/v1alpha1
kind: PullSubscription
metadata:
name: testing-source-event-display
spec:
topic: testing
sink:
apiVersion: v1
kind: Service
name: event-display

This worked fine and allowed me to read Pub/Sub messages in a Kubernetes Service. As the next step, I wanted to use Knative Services as event sinks because Knative Services are much easier to deal with. In PullSubscription, I defined my sink as follows:

apiVersion: pubsub.cloud.run/v1alpha1
kind: PullSubscription
metadata:
name: testing-source-event-display
spec:
topic: testing
sink:
apiVersion: serving.knative.dev/v1alpha1
kind: Service
name: event-display

I could apply this definition and see that pods related to PullSubscription were created:

$ kubectl get podsNAME                 READY   STATUS
cre-pull-e7072664-ee7a-11e9-8ae9-42010a84006f-7db6dcf4b9-5ztgx 1/1 Running 2 7m12s
pubsub-s-testing-source-event-display-pullsubscription-cretw8mw 0/1 Completed 0 7m12s

However, when I checked the logs of the pods, I saw some errors:

$ kubectl logs cre-pull-e7072664-ee7a-11e9-8ae9-42010a84006f-7db6dcf4b9-5ztgx{"level":"info","ts":1571054849.2031848,"logger":"fallback","caller":"pubsub/transport.go:231","msg":"got an event!"}
{"level":"warn","ts":1571054849.2170587,"logger":"fallback","caller":"pubsub/transport.go:248","msg":"pubsub receiver return err","error":"Post http://event-display.default.svc.cluster.local: dial tcp: lookup event-display.default.svc.cluster.local on 10.0.16.10:53: no such host"}

PullSubscription gets the message but it cannot pass it to the Knative Service. The default.svc.cluster.local implied that Knative expected my service to be cluster local (internal). This wasn’t the case before. I tried to turn my Knative service a cluster local service as I explain in my tutorial and that didn’t work either.

Cluster local issue

At this point, I was stuck. After researching, I realized that I wasn’t alone. There is a bug (1973) where other users also report similar problems. Thanks to Knative Engineering team and comments in the bug, I figured out that Knative now requires an extra Istio cluster local gateway added to your cluster in order to have Knative Services as event sinks in PullSubscription.

Installing cluster local gateway

There’s an Updating your install to use cluster local gateway page that describes how to add local gateway to your Knative cluster. There’s also a third_party folder in Knative Service with istio-knative-extras.yaml that you can point to install the cluster local gateway.

In my case, I was using a GKE cluster with Istio add-on with the following versions:

  • GKE: 1.14.6-gke.1
  • Istio: 1.1.13-gke.0

The third_party folder in Knative had Istio versions 1.2.7 and 1.3.2 as of today. Not an exact match with my Istio version but I went with 1.2.7:

$ kubectl apply -f https://raw.githubusercontent.com/knative/serving/master/third_party/istio-1.2.7/istio-knative-extras.yamlserviceaccount/cluster-local-gateway-service-account created
serviceaccount/istio-multi configured
clusterrole.rbac.authorization.k8s.io/istio-reader configured
clusterrolebinding.rbac.authorization.k8s.io/istio-multi configured
service/cluster-local-gateway created
deployment.apps/cluster-local-gateway created

After this I was able to use Knative Services as events sinks.


This is still not ideal because I’m not sure if it’ll continue to work with different versions of Istio of as GKE add-on gets upgraded. However, it’ll work as a temporary solution until the we come up with a more complete installation instructions for Knative Eventing on GKE.

I outlined all the setup needed to receive Pub/Sub messages in Knative Eventing in my tutorial. Check it out here.

Google Cloud Platform - Community

A collection of technical articles published or curated by Google Cloud Platform Developer Advocates. The views expressed are those of the authors and don't necessarily reflect those of Google.

Mete Atamel

Written by

Developer Advocate at Google

Google Cloud Platform - Community

A collection of technical articles published or curated by Google Cloud Platform Developer Advocates. The views expressed are those of the authors and don't necessarily reflect those of Google.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade