Debugging Kubernetes Pods with Tcpdump from k8s worker node

“tcpdump” serves as a potent command-line packet analyzer tool utilized for network troubleshooting and monitoring. It facilitates the capturing and display of TCP/IP packets transmitted or received over a network interface connected to the system.

Notably, applications do not include the tcpdump utility by default within their images. This omission is intentional, driven by security considerations and adherence to industry best practices. Additionally, executing tcpdump necessitates root privilege or Kubernetes capability within the pod. Given that the images lack the tcpdump utility, any pods or containers derived from Kubernetes deployments or stateful sets utilizing these images will also be devoid of the tcpdump utility.

This page aims to elucidate the process of running the tcpdump utility on a running pod that does not include the tcpdump tool. Crucially, this process must ensure that the pod remains undisturbed in its running state throughout. Upon completion of the tcpdump operation, the pod should return to its original running state seamlessly. In essence, we will dynamically attach the tcpdump container to the application pod as needed from the k8s worker node using docker container runtime;

“We are not implementing the ephemeral container concept for running tcpdump. This approach requires root privileges within the pod, which is not always recommended for security reasons. Additionally, we are not directly running tcpdump on the Kubernetes worker node to collect all packets sent and received by the pods on that node. Instead, our focus is solely on running the tcpdump utility within the pod namespace.”

Prerequisite

1. Image: Firstly, the “tcpdump” utility package image must be created. I recommend building it using the following repository: https://github.com/nicolaka/netshoot Once the image is built, it needs to be uploaded into the Docker repository for on-demand access.
2. Access to Worker Node: The user needs to have access to the k8s cluster worker nodes where the Pod is currently running and should be a member of the docker group. This membership enables the user to execute Docker commands without requiring sudo access. This access is crucial for attaching a ‘tcpdump’ container within the existing Pod namespace, facilitating the execution of the tcpdump container.

Additional Information

All operations will be performed using the pod name rather than the individual container names within the pod. Utilizing the pod name is preferred as it simplifies the process compared to using container names. Additionally, it’s important to understand that all containers within the same pod share the same network namespace. Therefore, running commands with either the pod name or container name will result in accessing the same network namespace.

The ‘tcpdump’ utility captures all packets sent and received on the specified interface. These packets may belong to standard or non-standard protocols.

Executing ‘tcpdump’ within the Pod or Container namespace operates akin to performing ‘tcpdump’ within a VNF or on Bare-metal infrastructure.

Execution

It’s time to put our plan into action by ensuring that all the necessary prerequisites are configured and available in the environment. With these preparations in place, we can proceed to attach the tcpdump container to the running application pod using the ‘docker’ command and run the tcpdump utility on the fly. Let’s execute this plan promptly.

Firstly, we need to determine the name of the pod and the worker node where the pod is currently running. For example:

Figure-1: Determining the Pod to run tcpdump command.

We are currently focusing on the ‘nginx-585c95cfc7-l4ghl’ Pod running on the ‘demo-m02’ worker node to attach a tcpdump container as part of its network namespace.

This Pod does not include the ifconfig or tcpdump utility, and we are confirming this by attempting to run those commands upon logging into the pod as shown below.

Figure-2: Pod fails on running tcpdump utility.

Now, log in to the Worker node (in this case, ‘demo-m02’) and execute the following command: ‘docker ps | grep <PodName>’

Figure-3: Retrieving Container ID of the Pod

The above command will retrieve the container instance ID of the determined Pod.

We are not interested in the “/pause” container as shown in Figure-3 as it is automatically added by Kubernetes to the pod. We will focus solely on retrieving the container instance ID relevant to our target container.

Now let us run ‘docker run’ command` as below to attach the tcpdump image to the running container.

$ docker run -it — rm -v /tmp:/tmp/tcpdump/ — net container:b8aec9f96678 netshoot-tcpdump bash

Let’s break down the command:

- `docker run`: This command is used to create and run a Docker container.

- `-it`: This flag combines two flags: `-i` which stands for interactive, and `-t` which allocates a pseudo-TTY. This combination allows you to interact with the container using a shell.

- ` — rm`: This flag specifies that the container should be removed automatically when it exits. This is useful for temporary containers.

- `-v /tmp:/tcpdump/`: This flag mounts the host directory `/tmp` into the container directory `/tmp/tcpdump/` to copy files, if needed.

- ` — net container:<Container ID>`: This flag connects the container’s network stack to another container’s network stack. In this case, it connects to the container with ID `b8aec9f96678`.

- `netshoot-tcpdump`: This is the name of the Docker image that we want to bound to the specified pod namespace.

- `bash`: This is the command that will be executed inside the container. In this case, it will start an interactive `bash` shell.

So, when we run this command, it will start a new container from the specified Docker image (`netshoot-tcpdump`), mount the host’s `/tmp` directory to `/tmp/tcpdump/` inside the container, connect the container’s network to another pod with the ID `b8aec9f96678`, and then start an interactive `bash` shell inside the container.

You can exit the bash shell within the attached tcpdump container by running the ‘exit’ command. This action will bring you out of the attached tcpdump container and back to the command prompt of the k8s worker node.

The pcap files generated by tcpdump must be good to write in the `/tmp/tcpdump/` folder inside the tcpdump container. The `/tmp/tcpdump/` folder is mounted from the `/tmp` directory of the k8s worker node. Users can transfer the pcap files out of the k8s worker node using SCP or SFTP commands, which are available on the k8s worker node. The Wireshark can be used to open the pcap file later for packet analysis.

Output

$ docker run -it — rm -v /tmp:/tmp/tmp — net container:a6ae7ffb5d86 tcpdump:1.0.0 bash

Sample Output