Delaying application start until sidecar is ready

Taking advantage of a peculiar Kubernetes implementation detail to block containers from starting before another container starts

Some days ago, while I was writing another chapter of Kubernetes in Action, Second Edition, I noticed something unexpected in the way Kubernetes starts the containers in a pod. After inspecting the source code to confirm what I was seeing, I realized I had just found a solution to a long-standing problem in the Istio Service Mesh, which is what I’m currently working on at Red Hat.

I believe most Kubernetes users assume that after a pod’s init containers have finished, the pod’s regular containers are started in parallel. It turns out that’s not the case.

If you inspect the Kubelet code that starts the containers, you’ll notice that it does this sequentially. The code executes in a single thread and starts the containers in the order they are listed in the pod’s spec.containers array.

But who cares, right? Assuming that the container images are already stored locally, the time it takes for the Kubelet to start the second container after it starts the first one is negligible. Effectively, they both start at the same time.

This isn’t ideal when a container depends on another container and requires it to be fully started before it can run. One example of this is the Istio Proxy sidecar container. Because the application’s outgoing communication is routed through the proxy, the proxy must be up and running before the application itself is started.

You could add a shell script in the application container that waits for the proxy to be up, and then runs the application’s executable file. But this requires changing the application itself. Ideally, we want to inject the proxy into the Pod without any changes to the application or its container image.

It turns out that this can be done by taking advantage of the synchronous container start in Kubernetes.

First, we need to specify the proxy as the first container in spec.containers, but this is just part of the solution, as it only ensures that the proxy container is started first and doesn’t wait for it to become ready. The other containers are started immediately, leading to a race condition between the containers. We need to prevent the Kubelet from starting the other containers until the proxy is ready.

This is where the post-start lifecycle hook comes in. It turns out that the Kubelet code that starts the container blocks the start of the next container until the post-start handler terminates.

We can take advantage of this behavior. Not just in Istio, but in every pod where the sidecar container must start and become ready before the application container can start.

In a nut-shell, the solution to the sidecar startup problem is the following:

If the sidecar container provides an executable file that waits until the sidecar is ready, you can invoke it in the container’s post-start hook to block the start of the remaining containers in the pod.

The following figure should help you visualize what happens in the pod.

Here’s how the pod manifest might look like:

apiVersion: v1
kind: Pod
metadata:
  name: sidecar-starts-first
spec:
  containers:
  - name: sidecar
    image: my-sidecar
    lifecycle:
      postStart:
        exec:
          command:
          - /bin/wait-until-ready.sh
  - name: application
    image: my-application

This approach is not perfect, though. If either the command invoked in the post-start hook or the container’s main process fail, the other containers will be started right away. Nevertheless, this should be a good workaround until Kubernetes introduces proper support for sidecar containers. Or until someone decides to change the Kubelet’s behavior and make it start containers in separate goroutines.

While this technique fixes the container start up order, it doesn’t help with the order in which the pod’s containers are stopped when you delete the pod. The pod’s containers are truly terminated in parallel. When it shuts down a pod, the Kubelet executes the container termination code in goroutines — one for each container. Unlike post-start hooks, the pre-stop hooks therefore run in parallel.

If you need the sidecar to stop only after the main application container has terminated, you must handle this in a different way.

I’ll explain how in my next blog post…