Kubernetes Ephemeral Containers

Content disclosure: I have taken a lot of references for definitions and content (like examples) from kubernetes.io because they are clear and unambiguous. I do not intend to own this article as personally written material. I write articles to educate others and to keep as a personal document reference for future.

What are Ephemeral Containers?

‣ They lack guarantees for resources or execution.
‣ They cannot be automatically restarted.
‣ They cannot have ports or probes (liveness/readiness)
‣ Their resource allocation is immutable as such configuring “resource” is not allowed.
‣ For complete list of supported fields check Link

(Above definition taken from kubernetes.io)

Why do you need Ephemeral Containers?

⁃ They are useful for interactive troubleshooting when kubectl exec is insufficient because a container has crashed or a container image doesn't include debugging utilities.

⁃ They can also be useful when you want to login to a Kubernetes node to look at operating system logs for errors. Like in case you are debugging an OOMKilled pod.

Some Important things to know before jumping into Ephemeral Container. Keep your notebook ready, since the answer to these questions will decide important deployment field.

1) Do you want to connect to a URL, like curl to debug an issue? like checking connectivity issues.
2) Do you want to connect to a process to debug an issue? like taking a threaddump?
3) Do you want to check the main container’s file system for debugging an issue?

If (2) and (3) above are required in your case for debugging, then you need to read up on “shareProcessNamespace”. (Kubernetes documentation: Link)

To explain in brief, process and filesystem for any container (inside a pod) is sandboxed and cannot be accessed from any other container (in the same pod). Unless the parent container has this flag set to true. The file system for the process can be accessed under /proc/$pid/root

(Not to reinvent the wheel, here is a good article that demonstrates how this field works: Link)

Below examples are taken from the kubernetes.io website for clarity and brevity.

(1.0) Debugging an existing Container

[ Adding debugging container to existing container ]

I hope you have read through the “shareProcessNamespace” field and its hopefully set to true by this point if you want to debug an issue with a process. Please remember that if you change deployment for a running container to add this field, it will restart all pods for the deployment.

I am going to use the same example as mentioned on the link below, minor change to enable shareProcessNamespace:
Command used: (Reference Link)

kubectl run ephemeral-demo --image=registry.k8s.io/pause:3.1 --restart=Never --overrides='{"spec":{"shareProcessNamespace": true}}'
kubectl debug -it ephemeral-demo --image=busybox:1.28 --target=ephemeral-demo

To check if your target pod has shareProcessNamespace set to true, you can se the following command:
kubectl get pod <name> --output="jsonpath={.spec.shareProcessNamespace}" && echo

Note: The --target parameter must be supported by the Container Runtime. When not supported, the Ephemeral Container may not be started, or it may be started with an isolated process namespace so that ps does not reveal processes in other containers.
If you try this with docker-desktop on your local system it will not work. I tried !!

To check if the Ephemeral container is running inside the pod, you can run the following:

kubectl describe pod ephemeral-demo
...
Ephemeral Containers:
  debugger-8xzrl:
    Container ID:   docker://b888f9adfd15bd5739fefaa39e1df4dd3c617b9902082b1cfdc29c4028ffb2eb
    Image:          busybox
...

You should be able to view and debug the process running inside the original container if you have reached this point without errors.
Happy Debugging !!

(1.1) Debugging an existing Container

[ Making a copy of existing container and debugging]

In certain deployment configurations you cannot connect to an existing pod. Like if your application is crashing on startup. In these situations you can use kubectl debug to create a copy of the Pod with configuration values changed to aid debugging.

kubectl run myapp --image=busybox:1.28 --restart=Never -- sleep 1d 
kubectl debug myapp -it --image=ubuntu --share-processes --copy-to=myapp-debug

The second statement creates a copy of the original container along with the debug image of ubuntu as a separate container in the same pod. Additionally “- - share-process” flag enables the shareProcessNamespace so that you can debug the processes in the parent container.

(1.2) Debugging an existing Container

[ Making a copy of existing container, pass a new command to the container and debugging]

In certain conditions, you want to change the command passed to the container. Like passing a debug flag.
To simulate a crashing application, use kubectl run to create a container that immediately exits:

kubectl run --image=busybox:1.28 myapp -- false

You can see using kubectl describe pod myapp that this container is crashing:

Containers:
  myapp:
    Image:         busybox
    ...
    Args:
      false
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1

You can use kubectl debug to create a copy of this Pod with the command changed to an interactive shell:

kubectl debug myapp -it --copy-to=myapp-debug --container=myapp -- sh
If you don't see a command prompt, try pressing enter.
/ #

Now you have an interactive shell that you can use to perform tasks like checking filesystem paths or running the container command manually.

(1.3) Debugging an existing Container

[ Making a copy of existing container, change the image of container and debugging]

Debug a container while also changing its image

As an example, create a Pod using kubectl run:

kubectl run myapp --image=busybox:1.28 --restart=Never -- sleep 1d

Now use kubectl debug to make a copy and change its container image to ubuntu:

kubectl debug myapp --copy-to=myapp-debug --set-image=*=ubuntu

The syntax of --set-image uses the same container_name=image syntax as kubectl set image. *=ubuntu means change the image of all containers to ubuntu.

(2.0) Debugging on Kubernetes Node

This is an absolutely necessary thing to know about debugging with Kubernetes. I have personally seen occurrences of issues with Kubernetes Nodes having issues with restarts and instability where the PaaS Portal is not able to give information. Even some times the uptime information for the Nodes might not be correctly displayed on portal.

Other things you can debug with this could be:
▪ Connectivity issues, like TCP connection/firewall issues.
▪ Packet loss or network instability.
▪ Debug network latency issues.
▪ Confirming issues with pods when they get OOMKilled.

In the following example, you can connect to the Kubernetes Node and run command on the same:
First get the name of the node where you want to do the debugging. Like for example, if you want to check for a VM where pods are regularly crashing, you can short list the node(s) by passing “-o wide” like so:

kubectl get pods -o wide | grep <podname-that-is-restarting>

The output can have one or many node(s) depending on the number of pods running. Once you know the Node name, you can use the following command to start a debug pod there.

kubectl debug node/mynode -it --image=ubuntu

▪ The root filesystem of the Node will be mounted at /host
▪ The container runs in the host IPC, Network, and PID namespaces, although the pod isn’t privileged, so reading some process information may fail, and chroot /host will fail.
▪ If you need a privileged pod, create it manually.

I would like to add an example of Azure AKS, cluster where I have done some debugging at the Node level. (Azure Reference: Link)

Connecting to an AKS node:
kubectl debug node/aks-nodepool1-12345678-vmss000000 -it --image=mcr.microsoft.com/dotnet/runtime-deps:6.0

Once you are connected, you will get a prompt similar to this:

root@aks-nodepool1-12345678-vmss000000:/#

Connecting to an AKS node:
kubectl debug node/aks-nodepool1-12345678-vmss000000 -it --image=mcr.microsoft.com/dotnet/runtime-deps:6.0

Once you are connected, you will get a prompt similar to this:

root@aks-nodepool1-12345678-vmss000000:/#

Some things you can check while connected to the VM:
▪ uptime
▪ dmesg ( to check for any os level errors)
▪ To start examining the OOM log interactively, we may launch journalctl like:

$ journalctl --utc -b -X -ke

X should be an integer, (the minus sign is the part of the literal integer rather than an option syntax). journalctl interprets negative integers as counting from the last boot backward, while positive integers count from the first boot forward.

▪ Debugging CGroups OOM:
dmesg -d -f kern -L -T -w | grep -i memory

(Not going too indepth into the debugging of OOMKilled process)

What did we forget?

Aah, please dont forget to cleanup after completing your debugging. Delete any additional or unnecessary pods.

I hope this article helps or encourages you to read more about Ephemeral Containers and their importance in debugging issues with running or failing Kubernetes containers.