k8spacket - a fully based on eBPF right now

What is k8spacket? It is the tool to collect information about TCP traffic and TLS connection metadata in the Kubernetes cluster using eBPF and visualize it in Grafana. Try on Killercoda

k8spacket — node graph

k8spacket — TLS connections & certificates

Check this article for more details.

Before version 2.0.0, k8spacket used the gopacket Google library to collect information about TCP packets. The application could process replicated TCP raw data in user space with that library. It was a heavy and inefficient process regarding high network traffic inside the cluster.

eBPF is a Linux kernel technology that allows processing data in kernel space and sending events to user space with only the necessary details. There are a few methods to attach the eBPF program to the kernel calls, and k8spacket uses two of them:

• tracepoint inet_sock_set_state — to get information about TCP connections inside the cluster
• traffic control and queueing discipline filters on ingress and egress — to collect information about the TLS handshake process

Let’s go deeper and understand what’s happening with eBPF programs used in k8spacket.

Tracepoint inet_sock_set_state

k8spacket works as a DaemonSet resource on each Kubernetes cluster node. It has to sniff TCP connections on every network interface regarding all containers run on the node. An important fact is that the container could be client-side or server-side in the established TCP connection. k8spacket has to be vigilant at that point. The eBPF program listens to two types of TCP states to cover this.

• TCP_SYN_SENT — container as a TCP client
• TCP_SYN_RECV — container as a TCP server

TCP_SYN_SENT

When a container wants to connect to the TCP server, it sends a SYN packet to start setting up the connection. At the same time, the Linux TCP stack sets the state to TCP_SYN_SENT. k8spacket is attached to the tracepoint, which emits an event about changing the state. That’s how we know the connection to the server has been started and can store that information for further operations.

TCP_SYN_RECV

When someone from the outside (it can be another container in the cluster or even a client from the outside of the cluster network) wants to connect to the sniffed container, it must send a SYN packet to the receiver. When the TCP server gets this packet, after a few complicated operations (great article about it), changes the state to TCP_SYN_RECV. Now, k8spacket knows: ok, someone is knocking, let’s store it and wait for what will happen.

TCP_CLOSE

The last TCP state, interesting for k8spacket, is TCP_CLOSE. It’s a common state for both the client and the server side. When a TCP connection lands in that state, k8spacket receives a signal: conversation finished so we can collect data and send them to the user space.

To sum up, using only one tracepoint inet_sock_set_state, we get information about all TCP connections in the cluster node.

Traffic Control and queueing discipline filters

The second skill of k8spacket is collecting information about TLS connections. Before the eBPF approach, as mentioned earlier, it was a heavy and inefficient process concerning copying all packets to user space.

Regarding eBPF, there are two main approaches to sniffing packet data:

• TC (Traffic Control)
• XDP (eXpress Data Path)

k8spacket uses a traffic control approach because of two aspects:

• XDP works for incoming packets only
• XDP is supported by a long but still not complete list of network drivers (supported network drivers)

With the traffic control (TC) subsystem, we can attach ingress and egress filters to sniff incoming and outgoing TCP packets. k8packet uses the clsact queueing discipline to attach the eBPF program as a filter (see more info about clsact). The program waits for TCP packets. When a packet arrives, we check if the message belongs to a TLS handshake process. k8spacket is interested in two types of messages (clientHello and serverHello) and gets necessary data only from them. After all, it sends collected data to user space for further processing.

Collecting TLS domain certificates

eBPF program collects metadata about TLS handshake operation but without certificate raw data. It is pointless to do it in kernel space and send it to user space because of the amount of data and limitation of supported versions (less than TLS 1.3 because of encryption server handshake data from version 1.3). It could be done in user space by the eBPF program as well, but there are many libraries for TLS encryption, like OpenSSL, LibreTLS, BoringTLS, GnuTLS, GoTLS, etc. Development and maintenance would be a difficult task.

For now, k8spacket uses scrapping with cache for server certificates using the Golang tls package and tls.Dial() function. As a rarely fluctuating factor, a scraped certificate stays in the cache for 24 hours (could be configured).

Limitations

k8spacket uses ring buffer map, which forces using Linux kernel version at least 5.8 (BPF Features by Linux Kernel Version).

New features

• Dashboard k8spacket — TLS certificate expiration shows the remaining time until the certificates expire. By variable Past interval to analyze, you can narrow or expand the scope of data to analyze. Data sorted by expiring earliest first. It’s easy to build alerts based on the certificate expiration date. See more raw metrics in the dashboard k8spacket — TLS metrics

k8spacket — TLS certificate expiration

• Two principal k8spacket dashboards (k8spacket — node graph and k8spacket — TLS connections & certificates) are sensitive to Grafana time range control now. You can show cluster TCP traffic or TLS connections, f.e., for the last 5 minutes.

k8spacket — node graph

k8spacket — TLS connections & certificates

Summary

If you work with the Kubernetes cluster on Linux distribution and the kernel version is at least 5.8, upgrade or try the new k8spacket. It is faster, more efficient, and requires less memory than older versions.

The source code can be found at github.com/k8spacket