KubeShark: Wireshark for Kubernetes

A microscopic look into what’s happening in the cluster

Introduction

Wireshark is a well known tool for Packet Capture, analysis and troubleshooting. It can capture live network traffic and analyze it in real-time at a microscopic level, or read and process saved capture files. Wireshark can dissect and display many different protocols, and has a powerful filtering system for narrowing down traffic of interest. https://www.wireshark.org/

TCPDump/Wireshark gives us the power to visualize and understand whats happening in our network at a microscopic level. Imagine if something like this is possible in K8s, if you can see what exactly happens when you deploy a workload, get pods, create service account and how different workloads are interacting with each other etc..

Sample Gource visual of a cluster

Whats missing right now?

To see what exactly happens when you fire a kubectl command, you can simply use verbose flag with your kubectl command with some verbosity level like — kubectl get pods -v=6. Verbosity level starts from 0 and ends at 9, 0 being the min and 9 being max verbose. This is already present, but clear visualization of the same flow is not present.

Kubectl with -v flag allows you to see L7 API Calls, it won’t let you monitor any of the network traffic as such. Kubeshark does capture L3 & L7, in fact it will have access to the whole L2 frame. It will create a dashboard/view to visualize the same, just like Wireshark.

Kubeshark dashboard

What is Kubeshark solving?

1. Monitor everything — Be it monitoring Pod-Pod network traffic, API calls made, monitor a single node, a service, or a specific path like — /latest/meta-data or /health etc..
2. Visualize — Kubeshark Viewer is so simple, yet so powerful to view real time traffic, filter, analyze and troubleshoot any issue. Request Response details on the right pane are super clear.
3. PCAP store — Store PCAP for future analysis of an incident, can easily dump this into S3 or some other storage, load and visualize later.
4. Easier Debugging — Whether its debugging something or deepening the understanding of comms between various components in the cluster, this greatly enables to do so.
5. Real Time Alerting — Can build really powerful alerting for K8s with the Data Kubeshark is allowing me to use. Think of failed calls on get secrets, failed instance meta-data calls, many more anomalies like these.

Get your hands dirty

Lets install Kubeshark and see how does it work. You can install KubeShark using this command

sh <(curl -Ls https://kubeshark.co/install)

For specific release and arch, download binary from here

https://github.com/kubeshark/kubeshark/releases/

Start Kubeshark with —

kubeshark tap -A

tap -A taps all namespaces, you can specify -n to monitor a specific namespace.

kubeshark tap -n kube-system

Kubeshark will start monitoring/tapping the traffic & start a viewer on port 8899 on your localhost.

Kubeshark started, dashboard exposed on port 8899

I have my test cluster deployed using KOPS, note that Kubeshark binary looks at your current context from ~/.kube/config and uses the same credentials and cluster to create its resources & start monitoring.

You can create a local kind cluster to quickly get a feel of how Kubeshark works.

Playing around a bit

Lets look at what do we monitor and what capabilities kubeshark has got on its dashboard. These are some sample filters I was using to test out kubeshark.

Monitoring a single node —

Visualizing using Service Map —

Monitor AWS Instance Metadata Security credential path —

Final Thoughts

1. Helps deepen my understanding of how various components in the cluster communicate and work together at the lowest level possible.
2. Debugging/Troubleshooting becomes super simple by adding the appropriate filters to the traffic.
3. Service Maps provide a high level overview of how my cluster is looking from a visual perspective.
4. Can extend the data to do some real time alerting, both operational and security alerts.
5. It would be good to do a performance benchmark at decent scale and see how this tool scales, affects compute & storage, given the amount of traffic grows exponentially at massive scale.
6. Honestly, it is not a fair to compare this with Sysdig falco or Cilium Tetragon, but the way this captures and tunes the data makes me look at real time detection aspect and think about using it in some of the security aspects more related to K8s API anomalies.
7. May be this is a tool I should use extensively to better understand calls under the hood to write better run-time security policies more related to control plane. This will not help much to detect any container anomalies or the ones related to container runtime AFAIK.

Conclusion

Kubeshark is a powerful tool which was earlier known as Mizu, developed by UP9 as an API traffic viewer and later got open sourced, now the name is changed to Kubeshark to signify this is Wireshark re-born for K8s.

While there are sophisticated tools for K8s runtime security like Sysdig Falco, Cilium Tetragon which are built leveraging eBPF capabilities, it is mentioned in the documentation that Kubeshark also uses eBPF, but is more focused on providing a visualization layer at a microscopic level to understand the communications better. The data provided by Kubeshark can definitely be used in few aspects of Run Time Detection.

Github Reference — https://github.com/kubeshark/kubeshark
Site Reference — https://kubeshark.co

Please feel free to point out mistakes if there are any. Thank you for reading. You can connect with me on Linkedin / Twitter. Happy to answer your queries.