Ephemeral Containers — the future of Kubernetes workload debugging

Ephemeral Containers are a new kind of container that runs temporarily in an existing Pod to allow easy user integrations like troubleshooting a container workload.

Why do we need Ephemeral Containers?

The advantages of containers are that they run isolated processes by providing all needed dependencies using an immutable approach. By adding only the required dependencies into the image a container lowers attack vectors and provides faster startups and deployments. Building container images using the “distroless” approach (building them from scratch) is taking this to the next level by only containing the compiled application binary. Unlike ordinary container images, these are not based on any kind of Linux distribution and therefore do not contain any other binaries and tools which could be executed via kubectl exec for troubleshooting purposes. Besides this, Ephemeral Containers can also help to debug a crashed container process which wouldn’t be possible with kubectl exec either.

The above-described method helps to provide a secure and reliable runtime environment but also makes it very hard to debug issues when they occur. This is where Ephemeral Containers can help. They provide the foundation to attach a debugging container to your main process which then can be used to debug any kind of issues. The debugging container can be based on any image and therefore can be customized based on your needs. You can build your own debugging image which contains special debugging binaries or just tools like curl, OpenSSL and a MongoDB client. But you can also pick a Linux distribution like Ubuntu or just run the Busybox images which both already contain a bunch of helpful tools. It doesn’t matter.

How does it work?

Ephemeral Containers is a pretty new feature that was introduced in Kubernetes v1.16 as an alpha feature. It is not yet feature-complete and should therefore not be used in a production environment until its GA. The actual roadmap mentions that it’s planed to provide a feature-complete alpha with Kubernetes v1.18. You can find more details about the roadmap and features in this issue.

One of the missing features in v.16 is the client-side integration of Ephemeral Containers which would allow us to use kubectl to create and attach to them. In the future, you will be able to debug a container with a new kubectl debug command. As an example, the following command would attach to a shell in an Ephemeral Container called debugger based on alpine into the my-pod:kubectl debug -c debugger --image=alpine my-pod -- bash. Or course, this is still subject to change. More details are available here.

How to use them

As mentioned above, Ephemeral Containers is an alpha feature and is therefore disabled by default. You will need to activate the following feature gates to be able to use them:

• EphemeralContainers
• PodShareProcessNamespace (beta in v1.16, and therefore already enabled by default)

Because you shouldn’t use Ephemeral Containers in production yet it’s recommended to use a dedicated environment. In this example, I will create my cluster using kind (Kubernetes in Docker) which allows me to bootstrap a cluster based on my needs in just some seconds. You can of course also use tools like kubeadm, minikube or others.

First of all, we need a custom kind cluster definition (cluster.yaml) to enable the needed feature gates:

kind: Cluster
apiVersion: kind.sigs.k8s.io/v1alpha3
kubeadmConfigPatches:
- |
  apiVersion: kubeadm.k8s.io/v1beta2
  kind: ClusterConfiguration
  metadata:
    name: config
  apiServer:
    extraArgs:
      "feature-gates": "EphemeralContainers=true"
  scheduler:
    extraArgs:
      "feature-gates": "EphemeralContainers=true"
  controllerManager:
    extraArgs:
      "feature-gates": "EphemeralContainers=true"
- |
  apiVersion: kubeadm.k8s.io/v1beta2
  kind: InitConfiguration
  metadata:
    name: config
  nodeRegistration:
    kubeletExtraArgs:
      "feature-gates": "EphemeralContainers=true"
nodes:
- role: control-plane
- role: worker

We then can bootstrap the cluster with the following command:

kind create cluster --config cluster.yaml --image kindest/node:v1.16.3

After some seconds our cluster will be ready and we can go ahead and create a Pod which we will then use to attach our debugger container. In my case, I created a simple deployment based on Nginx:

kubectl create deployment nginx --image nginx:latest

As soon as our Pod is up and running we are ready to attach our debugging container. As mentioned above, the kubectl debug command isn’t available yet which means we will need to talk to the API directly. To do so, we first need to define our Ephemeral Container (debug.json):

{
    "apiVersion": "v1",
    "kind": "EphemeralContainers",
    "metadata": {
            "name": "nginx-6db489d4b7-tvgsm"
    },
    "ephemeralContainers": [{
        "command": [
            "sh"
        ],
        "image": "busybox:latest",
        "imagePullPolicy": "IfNotPresent",
        "name": "debugger",
        "stdin": true,
        "tty": true,
        "terminationMessagePolicy": "File"
    }]
}

In our example, we will attach to a shell based on the busybox image to troubleshoot the nginx-6db489d4b7-tvgsm Pod. To schedule the container we now need to update the Nginx Pod with the above Ephemeral Container definition (you will need to update the command as well as the debug.json to match your Pod name and namespace):

kubectl replace --raw /api/v1/namespaces/default/pods/nginx-6db489d4b7-tvgsm/ephemeralcontainers -f debug.json

You can now review the Pod with kubectl describe pod to verify that the debugger container is up and running. Finally, you can start debugging your main process by attaching to the debugger container:

kubectl attach -it nginx-6db489d4b7-tvgsm -c debugger

You are now able to access the main process using the Ephemeral Container. This is be done by the PodShareProcessNamespace feature gate I mentioned above. It allows the debugger container to share the same process namespace with the main process.

The debugger container will not be killed automatically unless the pod is destroyed. Ephemeral Containers will stop when their command exits, such as exiting a shell.

In the future, you will just usekubectl debug which will execute the needed steps in the background and directly attach you to the TTY of your Ephemeral Container.

Happy debugging!