Cluster local issue with Knative Eventing v0.9.0

Mete Atamel
Oct 14, 2019 · 3 min read

In my previous post, I talked about Knative v0.9.0 and some of the eventing changes in the latest release. I’ve been playing with Knative v0.9.0 since then to read Google Cloud Pub/Sub messages using PullSubscription and I ran into a rather fundamental issue that baffled me for a while. I’d like to outline the problem and the solution here, just in case it’s useful to others.

Knative Services as eventing sinks

In my PullSubscription, I could define Kubernetes Services as event sinks as follows:

apiVersion: pubsub.cloud.run/v1alpha1
kind: PullSubscription
metadata:
name: testing-source-event-display
spec:
topic: testing
sink:
apiVersion: v1
kind: Service
name: event-display

This worked fine and allowed me to read Pub/Sub messages in a Kubernetes Service. As the next step, I wanted to use Knative Services as event sinks because Knative Services are much easier to deal with. In PullSubscription, I defined my sink as follows:

apiVersion: pubsub.cloud.run/v1alpha1
kind: PullSubscription
metadata:
name: testing-source-event-display
spec:
topic: testing
sink:
apiVersion: serving.knative.dev/v1alpha1
kind: Service
name: event-display

I could apply this definition and see that pods related to PullSubscription were created:

$ kubectl get podsNAME                 READY   STATUS
cre-pull-e7072664-ee7a-11e9-8ae9-42010a84006f-7db6dcf4b9-5ztgx 1/1 Running 2 7m12s
pubsub-s-testing-source-event-display-pullsubscription-cretw8mw 0/1 Completed 0 7m12s

However, when I checked the logs of the pods, I saw some errors:

$ kubectl logs cre-pull-e7072664-ee7a-11e9-8ae9-42010a84006f-7db6dcf4b9-5ztgx{"level":"info","ts":1571054849.2031848,"logger":"fallback","caller":"pubsub/transport.go:231","msg":"got an event!"}
{"level":"warn","ts":1571054849.2170587,"logger":"fallback","caller":"pubsub/transport.go:248","msg":"pubsub receiver return err","error":"Post http://event-display.default.svc.cluster.local: dial tcp: lookup event-display.default.svc.cluster.local on 10.0.16.10:53: no such host"}

PullSubscription gets the message but it cannot pass it to the Knative Service. The default.svc.cluster.local implied that Knative expected my service to be cluster local (internal). This wasn’t the case before. I tried to turn my Knative service a cluster local service as I explain in my tutorial and that didn’t work either.

Cluster local issue

At this point, I was stuck. After researching, I realized that I wasn’t alone. There is a bug (1973) where other users also report similar problems. Thanks to Knative Engineering team and comments in the bug, I figured out that Knative now requires an extra Istio cluster local gateway added to your cluster in order to have Knative Services as event sinks in PullSubscription.

Installing cluster local gateway

There’s an Updating your install to use cluster local gateway page that describes how to add local gateway to your Knative cluster. There’s also a third_party folder in Knative Service with istio-knative-extras.yaml that you can point to install the cluster local gateway.

In my case, I was using a GKE cluster with Istio add-on with the following versions:

  • GKE: 1.14.6-gke.1
  • Istio: 1.1.13-gke.0

The third_party folder in Knative had Istio versions 1.2.7 and 1.3.2 as of today. Not an exact match with my Istio version but I went with 1.2.7:

$ kubectl apply -f https://raw.githubusercontent.com/knative/serving/master/third_party/istio-1.2.7/istio-knative-extras.yamlserviceaccount/cluster-local-gateway-service-account created
serviceaccount/istio-multi configured
clusterrole.rbac.authorization.k8s.io/istio-reader configured
clusterrolebinding.rbac.authorization.k8s.io/istio-multi configured
service/cluster-local-gateway created
deployment.apps/cluster-local-gateway created

After this I was able to use Knative Services as events sinks.

This is still not ideal because I’m not sure if it’ll continue to work with different versions of Istio of as GKE add-on gets upgraded. However, it’ll work as a temporary solution until the we come up with a more complete installation instructions for Knative Eventing on GKE.

I outlined all the setup needed to receive Pub/Sub messages in Knative Eventing in my tutorial. Check it out here.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store