Debug Network Traffic in Kubernetes using a Sidecar

Photo by Universal Eye on Unsplash

Recently, I came across an issue where I wanted to debug the network traffic in an application in order to identify a database connectivity issue. This would have been a straightforward task if the application was running locally. I would only have to use either Wireshark to capture and analyze the traffic or use tcpdump in Linux based systems and analyze the dump. Alas, things are never that easy in life.

My first problem was that the application was deployed in Kubernetes. Sometimes you don’t have direct access to those deployments. Since our organization was following Kubernetes best practices and container design best practices, the containers only have a limited set of libraries and capabilities in order to comply with the said best practices. This means that tcpdump was not available in the container; which makes capturing and analyzing network traffic much more difficult. This is where sidecars come into play.

A sidecar container is exactly what it sounds like. Its a utility container running in the same pod as the application, providing additional capabilities to the application. LinkerD and Istio are some well known examples which provide sidecar injection. Here, my sole purpose of adding a sidecar is to capture and analyze the network traffic using tcpdump . Since containers in the same pod belong to the same network namespace (network layer) we can capture the network traffic in our application using the sidecar. You can use any image that contains tcpdump for the sidecar. One example would be https://hub.docker.com/r/nicolaka/netshoot. This image provides various types of network troubleshooting options.

To add the sidecar to our application’s pod, we need to add the netshoot container to the deployment. You could do this simply with kubectl edit deployment command or using other tools like kustomize. We need to add the following config under spec/templates/spec/containers.

- name: tcpdump-sidecar
image: nicolaka/netshoot
imagePullPolicy: Always

With this, we define that we need to create a new container with the name tcpdump-sidecar using the image nicolaka/netshoot. Once you save the configurations, a new pod will be created with the sidecar.

Now, all we have to do is to go into the sidecar container and get the network dump using tcpdump tool. We can use the following command to go into the container.

kubectl exec -it <pod-name> -n <namespace> -c tcpdump-sidecar -- /bin/bash

Once we are in there, we can verify whether tcpdump is configured by running tcpdump --version . Since I wanted to debug some connectivity issues with a MySQL database, I ran the following command capture the network traffic from my application.

tcpdump -i any -nn dst port 3306 -w network.pcap

Once you feel like you have captured enough data, we can finish the process by using Ctrl+C. Copy the .pcap file to your local working environment using the following command.

kubectl cp <namespace>:<pod-name>:/home/network.pcap -c tcpdump /User/rakhitha/network.pcap

Now all I have left todo is open the .pcap file with Wireshark and analyze the database network traffic.

Thats it! Here I have shown you how to obtain the network traffic data of an application running in Kubernetes pod with the help of a sidecar container. This may not be the most elegant solution, but it works. I hope this was of some use if you came across a similar problem.

Computer Science and Engineering Undergraduate at University of Moratuwa. Former Intern at IFS R&D. Software Engineer at WSO2.