Rogue One: Network Connection Tracing on Kubernetes Pods using conntrack on CoreOS Container Linux
There are occasions as a K8s Admin that require detective work tracking down an offending pod(s) that is opening too many network connections to a downstream entity. An inordinate number of connections can cause issues such as resource starvation and grumpy operators.
In my case the downstream system was a CouchBase Query Node. The CouchBase admins reached out and asked to perform an audit of the upstream K8s cluster(s) to determine which pods were opening too many connections.
We were supplied with the CouchBase Query Node IP and Port (aka the destination) and the source IP and port (aka the K8s node).
The source IP and Port would point us to the Kubernetes Node (Worker/Minion). However, in most cases, there are a dozen or more pods that run on each node, and it’s not easy to pinpoint which one it could be. Especially if the Kube Admin is detached from the downstream specifics of the workloads that are running — and in our particular case where there are literally tens of thousands of pods running at any given time, its the case more than not.
One way to do this is use a handy utility called conntrack. Thus the purpose of this blog entry. This tool allows one to interact with the linux in-kernel Connection Tracking System.
conntrack is installed via toolbox on CoreOS Container Linux:
# /usr/bin/toolbox
# dnf -y install conntrack
# conntrack -h
Command line interface for the connection tracking system. Version 1.4.4
Usage: conntrack [commands] [options]
Commands:
-L [table] [options] List conntrack or expectation table
-G [table] parameters Get conntrack or expectation
-D [table] parameters Delete conntrack or expectation
-I [table] parameters Create a conntrack or expectation
-U [table] parameters Update a conntrack
-E [table] [options] Show events
-F [table] Flush table
-C [table] Show counter
-S Show statistics
…
Once conntrack is installed, I can grep its table information using the source IP and Port.
The typical command I like to use is:
# conntrack -L -f ipv4 -d xx.xxx.xxx.xxx -o extended
So for example, say the downstream IP is 10.11.12.13 and the port is 8093, then my query would be:
# conntrack -L -f ipv4 -d 10.11.12.13 -o extended
ipv4 2 tcp 6 86374 ESTABLISHED src=10.14.15.16 dst=10.11.12.13 sport=39802 dport=8093 src=10.11.12.13 dst=10.X.X.X sport=8093 dport=39802 [ASSURED] mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=1
conntrack v1.4.4 (conntrack-tools): 1 flow entries have been shown.
From this information, I can then use that first src address (10.14.15.16) and interpret that as the Pod IP of the offending rogue pod. Since that is associated with the connection to the downstream IP and Port (10.11.12.13:8093)
I can then simply use kubectl to lookup the name of the pod using that Pod IP address.
# kubectl get po — all-namespaces -o wide | grep 10.14.15.16
We hope this article helps!
- Jonathan T.
- George B.
