Kubernetes — Kubectl debug

kubernetes — don’t you love it?

Introduction

In my last article, “Dockerfile — Kick it up a notch!” we have talked about docker layers and improved our final layer using scratch (void containing only our binary). But how can we debug it? That’s our topic for today.
I’ll introduce a relatively new (since 1.20, dated 2020) kubectl command, called debug .

Recap

If we did out job correctly, our kubernetes workloads, mainly pods are minimal and missing useful debugging & investigations tools.
If we’ve followed best practice, we can not even get inside the container using /bin/sh .

Quick Note

You should no debug containers by performing a “dive-in” into them. Moreover, if bugs are found in our applications which is running as part of a the workload running in our cluster, we missed crucial testing phase. But let’s imagine a scenario where we do need that ability, in our staging environment cluster.

Kubectl debug

The kubectl debug command was introduced as an alpha feature in Kubernetes 1.18 and became beta in Kubernetes 1.20. It's a part of the kubectl command-line interface (CLI), and it is commonly used to debug an existing workflow while preserving the ability to change certain values.

debug command

How to use?

Let’s demonstrate kubectl debug use and capabilities:

We’ll use KillerCoda’s (Recommended!) kubernetes arena. deploy an nginx workload and create a debug container named kubectl-debug-demo .

kubectl create deploy nginx --image nginx:latest --replicas 3

kubectl get pods

kubectl debug pod/nginx-56fcf95486-njdzj \
  --stdin \
  --tty \
  --image busybox \
  --share-processes \
  --copy-to my-debug-container

• --stdin : Bind host to conainter’s standard input.
• --tty : Allocate a teletypewriter, pseudo-teletypes slave (PTS).
• --image : Choose your desired image to debug with. alpine or busybox might be great for that purpose.
• --share-processes : By design debug containers share the same linux namespace as the application we’re debugging. This flags ensures our replica share the processes.
• --copy-to : Create a replica pod of our target, and insert the debug container to it. Also name it “my-debug-container” so I can find it easily. Discarding this flag will modify the pods configuration and insert a debug container alongside our application’s. Not recommended.

Now we can see the nginx worker processes running:

sharing the same processes

With some tweaks to the securityContext we can also share the file-system. I’ll demonstrate that without using the --copy-to flag, which again, is not recommended.

kubectl debug pod/nginx-56fcf95486-njdzj \
  --stdin \
  --tty \
  --image busybox \
  --share-processes \
  --target nginx

Sharing file system

Bonus — Colored output

Working with kubernetes in general and debugging particularly, involves a lot of --output yaml viewing. Here’s a cute little thing I use which makes my life easier. It’s called “bat”: A Cat clone with wings!

bat ( — > batcat on ubuntu) provides highlighting, git integration and much more!

bat — A Cat clone with wings!

Summary

In this article we have talked about kubernetes built-in debug command, it’s history, how to use it and how powerful it is. However, remember that if you follow the debugging methodology, you should start by checking the pod’s events and logs before intervening with it. Happy debugging!