How to TCPdump effectively in Kubernetes (part 2)

Philippe Bogaerts
Dec 5, 2019 · 5 min read

In previous blog posts, we focused on how to use TCPdump in a specific container (see https://medium.com/@xxradar/how-to-tcpdump-effectively-in-docker-2ed0a09b5406 ) as well as how we could add it to an existing K8S deployment using a patch (see https://medium.com/@xxradar/how-to-tcpdump-effectively-in-kubernetes-part-1-a1546b683d2f)

While researching some other things recently, I came across a comment suggesting a quick fix for another issue like this …

$ kubectl run -it --rm debug  --restart=Never --image=ubuntu --overrides='{"kind":"Pod", "apiVersion":"v1", "spec": {"hostNetwork":true}}'

This is pretty similar as described in a previous blog posts …

docker run -it --net=host ubuntu 

… but in the kubectl case, we do not need SSH access to a node or access to the docker client, nor do we need to re-deploy the deployment (aka restart of the pods)

So let’s try this !!
Create a small K8S cluster and deploy a simple nginx service (I used the managed K8S service from Digitalocean and tested as well on an Azure environment)

$ kubectl get no
NAME STATUS ROLES AGE VERSION
demo-pool1-lyg2 Ready <none> 2m3s v1.16.2
demo-pool1-lygl Ready <none> 2m12s v1.16.2
demo-pool1-lygt Ready <none> 2m14s v1.16.2
$ kubectl get svc -n radarhack
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-radarhack-clusterip ClusterIP 10.245.94.156 <none> 80/TCP 100s
$ kubectl get po -n radarhack -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
radarhack-deployment-655b776bd-6v6dl 1/1 Running 0 22m 10.244.0.232 demo-pool1-lygt <none> <none>
radarhack-deployment-655b776bd-wmqv9 1/1 Running 0 22m 10.244.1.87 demo-pool1-lygl <none> <none>
radarhack-deployment-655b776bd-zrxdd 1/1 Running 0 22m 10.244.2.223 demo-pool1-lyg2 <none> <none>

So lets deploy an ubuntu pod like described before …

$ kubectl run -it --rm debug  --restart=Never --image=ubuntu --overrides='{"kind":"Pod", "apiVersion":"v1", "spec": {"hostNetwork":true}}'
If you don't see a command prompt, try pressing enter.

… inside the pod, install following packages …

root@demo-pool1-lygt:/# apt-get update && apt-get install -y net-tools && apt-get install -y tcpdump
Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:2 http://archive.ubuntu.com/ubuntu bionic InRelease [242 kB]
Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
...
Setting up tcpdump (4.9.2-3) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...

… and now we can list all the interfaces of the host inside the pod !

root@demo-pool1-lygt:/# ifconfig
cilium_host: flags=4291<UP,BROADCAST,RUNNING,NOARP,MULTICAST> mtu 1500
inet 10.244.0.191 netmask 255.255.255.255 broadcast 0.0.0.0
inet6 fe80::44ca:4aff:fee7:ddba prefixlen 64 scopeid 0x20<link>
ether 46:ca:4a:e7:dd:ba txqueuelen 1000 (Ethernet)
RX packets 445 bytes 46829 (46.8 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 133 bytes 6790 (6.7 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

cilium_net: flags=4291<UP,BROADCAST,RUNNING,NOARP,MULTICAST> mtu 1500
inet6 fe80::144b:40ff:fee3:892d prefixlen 64 scopeid 0x20<link>
ether 16:4b:40:e3:89:2d txqueuelen 1000 (Ethernet)
RX packets 133 bytes 6790 (6.7 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 445 bytes 46829 (46.8 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

cilium_vxlan: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::4e9:f7ff:fe9e:8a5a prefixlen 64 scopeid 0x20<link>
ether 06:e9:f7:9e:8a:5a txqueuelen 1000 (Ethernet)
RX packets 248 bytes 23105 (23.1 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 250 bytes 29122 (29.1 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255
ether 02:42:b9:15:7e:44 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 188.166.35.94 netmask 255.255.192.0 broadcast 188.166.63.255
inet6 fe80::7421:6bff:fed4:7a97 prefixlen 64 scopeid 0x20<link>
ether 76:21:6b:d4:7a:97 txqueuelen 1000 (Ethernet)
RX packets 35107 bytes 184254131 (184.2 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 8237 bytes 637679 (637.6 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eth0:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.18.0.9 netmask 255.255.0.0 broadcast 10.18.255.255
ether 76:21:6b:d4:7a:97 txqueuelen 1000 (Ethernet)

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.133.91.245 netmask 255.255.0.0 broadcast 10.133.255.255
inet6 fe80::407:2eff:fee7:7135 prefixlen 64 scopeid 0x20<link>
ether 06:07:2e:e7:71:35 txqueuelen 1000 (Ethernet)
RX packets 3320 bytes 1297431 (1.2 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 3327 bytes 486607 (486.6 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 1933 bytes 123132 (123.1 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1933 bytes 123132 (123.1 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

...

We can now attach TCPdump to any of the interfaces and record the traffic.

root@demo-pool1-lygt:/# tcpdump -i eth0 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
12:29:40.285086 IP 178.62.231.175.443 > 188.166.35.94.56492: Flags [P.], seq 3344525609:3344526062, ack 3498904092, win 249, options [nop,nop,TS val 1350708887 ecr 1279555160], length 453
...
root@demo-pool1-lygt:/# tcpdump -i any -n port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
12:44:22.727590 IP 10.244.1.42.36658 > 10.244.0.232.80: Flags [S], seq 790184700, win 28200, options [mss 1410,sackOK,TS val 4002115421 ecr 0,nop,wscale 7], length 0
12:44:22.727695 IP 10.244.1.42.36658 > 10.244.0.232.80: Flags [S], seq 790184700, win 28200, options [mss 1410,sackOK,TS val 4002115421 ecr 0,nop,wscale 7], length 0
12:44:22.727738 IP 10.244.0.232.80 > 10.244.1.42.36658: Flags [S.], seq 2931529117, ack 790184701, win 27960, options [mss 1410,sackOK,TS val 3120146425 ecr 4002115421,nop,wscale 7], length 0
12:44:22.727751 IP 10.244.0.232.80 > 10.244.1.42.36658: Flags [S.], seq 2931529117, ack 790184701, win 27960, options [mss 1410,sackOK,TS val 3120146425 ecr 4002115421,nop,wscale 7], length 0
12:44:22.728424 IP 10.244.1.42.36658 > 10.244.0.232.80: Flags [.], ack 1, win 221, options [nop,nop,TS val 4002115422 ecr 3120146425], length 0
12:44:22.728460 IP 10.244.1.42.36658 > 10.244.0.232.80: Flags [.], ack 1, win 221, options [nop,nop,TS val 4002115422 ecr 3120146425], length 0

Remark: The traffic being displayed in the second example in address space 10.244.0.0/16 is inter-pod traffic. It is even possible to intercept the traffic in/out the kube-system pods ;-)

I hope this article provides an easy way to capture traffic and learn more on the inner workings of our K8S cluster. Thanks for reading, Philippe Bogaerts !!!

Philippe Bogaerts

Written by

BruCON co-founder, OWASP supporter, AviNetworks presales engineer and Docker enthusiast! Interested in webapp security and pentesting, music and food !!

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade