CODEX

Break down Kubernetes Events Internally

Understand Events structure and walk through how they are generated

Stefanie Lai
Mar 25 · 6 min read
Image from Unsplash @bizarretribe

kubectl get events --namespace {name}

It is considered one of the essential CLI commands for debugging the objects in the cluster. We can usually solve the simple issues with kubectl get pod or kubectl logs pod xxx. But what if both of them are limited? Events objects are what we seek to first to race the problem.

Just like Pod, Deployment, and ConfigMap, events are also one type of resource objects of Kubernetes, not tied to Pod like logs. But why we fail to get a result when we try to find the corresponding Pod events through kubectl get events {eventName} --namespace? And is there a way to use kubectl get events to get the relevant information through the resource name in CLI? Let’s understand the necessity of events and analyze its design architecture first, then reveal the answer.

What is an Event?

Let’s run kubectl explain events to find out more.

Event is one of the cluster-type resource objects, which we can access through REST APIs provided by APIServer.

Create Event: POST /api/v1/namespaces/{namespace}/eventsUpdate Event: PATCH /api/v1/namespaces/{namespace}/events/{name}Replace Event: PUT /api/v1/namespaces/{namespace}/events/{name}Delete Event: DELETE /api/v1/namespaces/{namespace}/events/{name}Delete Event list: DELETE /api/v1/namespaces/{namespace}/eventsGet specified Event: GET /api/v1/namespaces/{namespace}/events/{name}List Events: GET /api/v1/namespaces/{namespace}/eventsList Events all namespaces: GET /api/v1/eventsWatch Event: GET /api/v1/watch/namespaces/{namespace}/events/{name}Watch Events List: GET /api/v1/watch/namespaces/{namespace}/eventsWatch Events all namespaces: GET /api/v1/watch/events

There’s a total of 11 APIs, while some common APIs, such as describe, are missing compared with the API list of some other resources. To trace the reason, we can see easily from Events objects’ attributes, lacking the commonly seen status and spec.

Our general understanding of Kubernetes objects is that they save their state in etcd, and schedule to reach the desired spec state via APIServer request, which is eventually consistent. But as the Event has neither status nor spec, how does it schedule? Is scheduling even needed? Is it still stored in etcd? And how does Event relate to other resources?

Let’s unveil them all step by step by dissecting Kubernetes' source code.

I don’t know Kubernetes' source code well, though reading it has been on my to-do list. The following is a record of my exploring footprints, starting from the Event.

Event objects are defined in core/v1/type.go, the same as the APIVersion path, and are easy to find. The definition fields of Event are identical to those seen in kubectl explain above. Besides the regular fields TypeMeta and ObjectMeta, there are several special fields.

  • InvolvedObject records the events that take place. And ObjectReference includes GVK, etc.
  • EventSeries records the number and time of events that occur.
  • Related binds to the secondary external objects in complex situations.

Now we have figured out the Event objects composition basically and known how to query the events of related objects via kubectl. That is, just use the field-selector option, such as

kubectl get events --namespace test --field-selector involvedObject.name=podtest

After understanding the composition of the Event, it’s time to see how Kubernetes operates it. However, the use place is not as easy to find as type definition.

I directly searched “Event” in full text and found interface.go when I was almost in a hopeless tangle.

In this Go file, we can find the definitions of interfaces related to Event record, sink, and broadcast. And there are three implementation struct of recorderImpl, eventBroadcasterImpl and EventSinkImpl. But it seems strange to me that the sink and broadcaster implementations are put together in one file, and why not have a separate event_sink.go?

Keep digging and discover where EventRecorder is used. Continuing with the full-text search, we can see it in controllermanager, kube-scheduler/app/server.go, and deployment_controller.go, etc., for example, EventRecorder is defined in DeploymentController.

  • Initialize EventRecorder

The eventBroadcaster here is the EventBroadcaster interface we saw before, and a EventSink implementation is injected during initialization.

  • Call Eventfmethod to generate an Event.

Proceed to look at the event generation. And it is event.go, generateEvent function that finally sends the event to the watcher queue.

Concerning Broadcaster, it is another story. And understanding it helps to grasp the watch mechanism of Kubernetes, which using Go channels in achieving.

The Event’s delivery and saving happen in event_broadcaster.go.

Kubernetes Events internal
  • First, when deployment_controller initializes NewBroadcaster, it creates the relevant channel, initializes the Broadcaster instance, and then monitors the incoming channel.
  • Call StartEventWatcher function to create a watcher, generating a blockQueue internally, running a goroutine to watch the result channel, and then logging the event.
  • At last, StartRecordingToSink is called to initialize a EventHandler to handle the events. The logic has two parts, using a Mutex lock to save the event to cache and taking care of event series calculation.

Then, it saves the event byrecordEvent that calls sink patch or creates function internally. Sink is an implementation of EventSinkImpl and invokes REST APIs in APIServer to complete the job.

One thing worth mentioning is EventCorrelator, which contains events’ LRU cache(default length 4096) and mitigates the possible backpressure or events flood. It is a typical code example of the Go LRU cache that would help if you are preparing for interviews that examine your Go algorithm coding.

How to get Events using client-go api?

Using obtained knowledge to build something that can facilitate our daily work is one of our ultimate goals. Therefore, I also want to figure out how to fetch events through client-go API after comprehending Event structure and how Kubernetes propagates events.

A perfect example came across my mind in no time, the kubectl describe pod command, which also contains pod-related events. And you can get the code in describe.go and the Describefunction of PodDescriber.

It is similar to the way of using field-selector in the kubectl command line.

Maybe you have heard of the Knative project, which gives a super nice declarative way to consume the events. Check it out here.

At the end

There are four takeaways in this article.

  • Know how to use kubectl get events, and get events based on its related resources.
  • Understand the essence and structure of the Event and its difference from other resource types.
  • Realize how Kubernetes propagates events internally.
  • Learn some Go coding skills while reading the source code. Functions are apparently the first-tier citizen in Go, and it is everywhere. You can not say that you are writing the correct Go code until you use enough functions in your code.

An open-source framework like Kubernetes is always an inspiration for me, encouraging me to learn more coding skills, especially designing the architecture and writing graceful code with flexibility.

Thanks for reading!

Everything connected with Tech & Code

Stefanie Lai

Written by

Live in Stockholm. Love writing, interested in cooking, drawing and reading. Want to travel all around Europe with my cat.

CodeX

Everything connected with Tech & Code

Stefanie Lai

Written by

Live in Stockholm. Love writing, interested in cooking, drawing and reading. Want to travel all around Europe with my cat.

CodeX

Everything connected with Tech & Code

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store