Starting containers in order on Kubernetes with InitContainers

4 min readDec 11, 2017

Kubernetes provides a useful feature called InitContainers to perform some tasks during a pod’s initialization. One example use-case is ensuring that another service is available. Let’s take see how this applies to our sample url-shortener application.

The current situation

If we deploy our webapp before the database service/pod is available, the application fails miserably trying to start because of the lack of connection to the backend:

$ kubectl logs webapp-c7c8fd499-bl5sl
...
level=fatal msg="failed to start: failed to open connection to database: dial tcp: lookup postgres on 10.96.0.10:53: no such host"

With the application being written in Golang, logging a message with the fatal level also stops the application with an Exit(1)call. In a way, such a behavior makes sense: why would the application start if there is no database connection ?

In Kubernetes, deployments have a restartPolicy set to Alwaysby default, which means that the kubelet agent of the node attempts to restart the pods again and again, with an exponential back-off delay in case of failure.

$ kubectl get pods -w
NAME                     READY     STATUS             RESTARTS   AGE
webapp-c7c8fd499-bl5sl   0/1       Error              0           2s
webapp-c7c8fd499-bl5sl   0/1       Error              1           2s
webapp-c7c8fd499-bl5sl   0/1       CrashLoopBackOff   1           3s
webapp-c7c8fd499-bl5sl   0/1       Error              2          15s
webapp-c7c8fd499-bl5sl   0/1       CrashLoopBackOff   2          29s
webapp-c7c8fd499-bl5sl   0/1       Error              3          44s
webapp-c7c8fd499-bl5sl   0/1       CrashLoopBackOff   3          45s
webapp-c7c8fd499-bl5sl   0/1       Error              4           1m
webapp-c7c8fd499-bl5sl   0/1       CrashLoopBackOff   4           1m
webapp-c7c8fd499-bl5sl   0/1       Error              5           2m
webapp-c7c8fd499-bl5sl   0/1       CrashLoopBackOff   5           3m
webapp-c7c8fd499-bl5sl   0/1       Error              6           5m
webapp-c7c8fd499-bl5sl   0/1       CrashLoopBackOff   6           6m

Once the database is deploy and its service is created, the application is finally able to establish the connection and starts for good:

$ kubectl apply -f templates/database-deployment.yml
deployment "postgres" created$ kubectl apply -f templates/database-service.yml
service "postgres" created$ kubectl get pods -w
NAME                     READY     STATUS             RESTARTS   AGE
webapp-c7c8fd499-bl5sl      0/1    CrashLoopBackOff    9         25m
postgres-77b6bf95b7-wjlfg   0/1    Pending             0          0s
postgres-77b6bf95b7-wjlfg   0/1    Pending             0          0s
postgres-77b6bf95b7-wjlfg   0/1    ContainerCreating   0          1s
postgres-77b6bf95b7-wjlfg   1/1    Running             0         41s
webapp-c7c8fd499-bl5sl      1/1    Running            10         26m

This is ok-ish, but can we do better than that? (SPOILER ALERT: yes we can!)

InitContainers to the rescue

A Pod can have multiple application containers as well as several so-called InitContainers, which are executed in sequence and until termination, before the application containers are started. If one InitContainer fails, then the restartPolicy of the deployment applies to it.

The InitContainers feature also brings other benefits, such as running tools that may not be bundled in an application’s Docker image. In our case, the pg_isready command is not installed in our webapp Docker image, yet wouldn’t it be nice to be able to use it to verify that the database is ready to accept connections ?

We can verify this command by running it with kubectl run in a container based on the postgres:9.6.5 Docker image:

$ kubectl run check-db-ready \
  --image=postgres:9.6.5 \
  --restart=Never \
  --command -- /bin/bash -c \
  "pg_isready -h postgres -p 5432 && echo $?"
pod "check-db-ready" created$ kubectl logs check-db-ready
postgres:5432 - accepting connections
0$ kubectl delete pod check-db-ready # a bit of house cleaning...

(The pg_isready command exits with the return code 0 if the server is ready, 1 if the server is still starting, 2 if the server is not started yet and 3 is something else happened.)

Great! So now we can run the pg_isready -h postgres -p 5432 && echo $? command in an InitContainer of the webapp Deployment to verify that the database is accepting connections. See in particular the lines 15–20 of the deployment manifest below:

Let’s now see how the InitContainers affect the webapp pod startup when there’s no database pod/service available:

$ kubectl create -f templates/webapp-deployment.yml
deployment "webapp" created$ kubectl get pods
NAME                      READY     STATUS     RESTARTS   AGE
webapp-7cc6fffcd4-7drg2   0/1       Init:0/1   0          30s

Instead of juggling between CrashLoopBackOffand Error statuses as we’ve seen in the first part of this article, the webapp pod is now blocked in an Init:0/1 status, meaning that it’s waiting for the first InitContainer to complete successfully.

Checking the pod’s status and events

Checking a pod’s events is a good way to understand what’s happening at the containers level:

$ kubectl describe pod/webapp-7cc6fffcd4-7drg2
Name:           webapp-7cc6fffcd4-7drg2
...
Init Containers:
  check-db-ready:
    Container ID:  docker://aa63d4a27cd8
    Image:         postgres:9.6.5
    ...
    Command:
      sh
      -c
      until pg_isready -h postgres -p 5432 -U postgres; do echo waiting for database; sleep 2; done;
    State:          Running
      Started:      Sun, 10 Dec 2017 10:59:06 +0100
    Ready:          False
...

An InitContainer named check-db-ready is running but as expected, it is not ready yet. Checking the logs of this particular container shows what’s happening:

$ kubectl logs -f webapp-7cc6fffcd4-7drg2 -c check-db-ready
...
waiting for database
postgres:5432 - no response

(Note the -c argument to select the container on which the kubectl logs command applies.)

As one would expect, once the database is finally deployed, the InitContainer completes and the webapp starts with a successful connection to the backend.

$ kubectl create -f templates/database-deployment.yml
deployment "postgres" created$ kubectl create -f templates/database-service.yml
service "postgres" created$ kubectl logs -f webapp-7cc6fffcd4-7drg2 -c check-db-ready
...
waiting for database
postgres:5432 - accepting connections$ kubectl get pods
NAME                        READY     STATUS    RESTARTS   AGE
postgres-77b6bf95b7-mshzd   1/1       Running   0          1m
webapp-7cc6fffcd4-7drg2     1/1       Running   0          11m

We’ve covered a basic use-case of the InitContainers feature in this article, showing how it can be used to verify some pre-conditions before the application container starts.

Further documentation

You can check the official Kubernetes documentation if you want more information about it.

Starting containers in order on Kubernetes with InitContainers

The current situation

InitContainers to the rescue

Checking the pod’s status and events

Further documentation

Written by Xavier Coulon