Kubernetes Tip — Run an Interactive Pod

TLDR

The other day on twitter Scott Lowe asked about connectivity betweeen pods in Kubernetes which opened a brief chat about testing connectivity between pods. I thought I’d share some tips that I’ve found helpful.

Sometimes debugging your kubernetes deployments can very opaque. Often the logs and debug information you can get from kubectl logs isn’t super informative, especially when running somebody elses app that you can’t easily throw some extra debug messages into the code. Often you also have minimal containers that don’t have useful things like ping/curl/etc in then so its hard to even kubectl exec into a container to check for connectivity.

Fortunately Kubernetes lets you run interactive pods so you can easily spin up a busybox (or insert preferred image here) pod and explore your deployment with it.

For example if you want to test that KubeDNS is working as expected and hosts are resolving and can connect:

$ kubectl run -i --tty --rm debug --image=busybox --restart=Never -- sh
If you don't see a command prompt, try pressing enter.
/ # host mysql
sh: host: not found
/ # ping mysql
PING mysql (10.0.0.33): 56 data bytes
^C
--- mysql ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss
/ # nc mysql 3306
J
5.6.35 ;`S][0{=��]<Uti2[Uy!6-mysql_native_password
^C
^D
/ # pod default/debug terminated (Error)
$

You can see here I run a busybox image and I try to run hostagainst my MySQL service which fails because host not installed by default in the busybox image. Next I run ping and it shows dns resolved, but ICMP fails. This is totally cool, the MySQL service just doesn’t respond to ICMP. My next step was to use netcat against the MySQL TCP port itself which connects!

If I need to go a step further and ensure the MySQL protocol is working correctly and I can connect I can use the mysql image instead of busybox:

$ kubectl run -i --tty --rm debug --image=mysql --restart=Never -- mysql -h mysql -proot123 registry -e 'show tables;'
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------------+
| Tables_in_registry   |
+----------------------+
| access               |
| access_log           |
| alembic_version      |
| clair_vuln_timestamp |
| img_scan_job         |
| img_scan_overview    |
| project              |
| project_member       |
| properties           |
| replication_job      |
| replication_policy   |
| replication_target   |
| repository           |
| role                 |
| user                 |
+----------------------+
pod "debug" deleted

Others chimed in with some tips:

Jérôme suggests using alpine.

Justin tells us to create useful aliases/functions to make this easier!

Following Justin’s great example I created a function to do this for me:

kcdebug() { kubectl run -i --rm --tty debug --image=busybox --restart=Never -- sh

So now I can run:

$ kcdebug
If you don't see a command prompt, try pressing enter.
/ # nc mysql 3306
J
5.6.358Sw}0qmO��#],b+M$jkZ_Rmysql_native_password
/ # pod "debug" deleted