Starting containers in order on Kubernetes with InitContainers

Xavier Coulon
4 min readDec 11, 2017


Kubernetes provides a useful feature called InitContainers to perform some tasks during a pod’s initialization. One example use-case is ensuring that another service is available. Let’s take see how this applies to our sample url-shortener application.

Photo by frank mckenna on Unsplash

The current situation

If we deploy our webapp before the database service/pod is available, the application fails miserably trying to start because of the lack of connection to the backend:

$ kubectl logs webapp-c7c8fd499-bl5sl
level=fatal msg="failed to start: failed to open connection to database: dial tcp: lookup postgres on no such host"

With the application being written in Golang, logging a message with the fatal level also stops the application with an Exit(1)call. In a way, such a behavior makes sense: why would the application start if there is no database connection ?

In Kubernetes, deployments have a restartPolicy set to Alwaysby default, which means that the kubelet agent of the node attempts to restart the pods again and again, with an exponential back-off delay in case of failure.

$ kubectl get pods -w
webapp-c7c8fd499-bl5sl 0/1 Error 0 2s
webapp-c7c8fd499-bl5sl 0/1 Error 1 2s
webapp-c7c8fd499-bl5sl 0/1 CrashLoopBackOff 1 3s
webapp-c7c8fd499-bl5sl 0/1 Error 2 15s
webapp-c7c8fd499-bl5sl 0/1 CrashLoopBackOff 2 29s
webapp-c7c8fd499-bl5sl 0/1 Error 3 44s
webapp-c7c8fd499-bl5sl 0/1 CrashLoopBackOff 3 45s
webapp-c7c8fd499-bl5sl 0/1 Error 4 1m
webapp-c7c8fd499-bl5sl 0/1 CrashLoopBackOff 4 1m
webapp-c7c8fd499-bl5sl 0/1 Error 5 2m
webapp-c7c8fd499-bl5sl 0/1 CrashLoopBackOff 5 3m
webapp-c7c8fd499-bl5sl 0/1 Error 6 5m
webapp-c7c8fd499-bl5sl 0/1 CrashLoopBackOff 6 6m

Once the database is deploy and its service is created, the application is finally able to establish the connection and starts for good:

$ kubectl apply -f templates/database-deployment.yml
deployment "postgres" created
$ kubectl apply -f templates/database-service.yml
service "postgres" created
$ kubectl get pods -w
webapp-c7c8fd499-bl5sl 0/1 CrashLoopBackOff 9 25m
postgres-77b6bf95b7-wjlfg 0/1 Pending 0 0s
postgres-77b6bf95b7-wjlfg 0/1 Pending 0 0s
postgres-77b6bf95b7-wjlfg 0/1 ContainerCreating 0 1s
postgres-77b6bf95b7-wjlfg 1/1 Running 0 41s
webapp-c7c8fd499-bl5sl 1/1 Running 10 26m

This is ok-ish, but can we do better than that? (SPOILER ALERT: yes we can!)

InitContainers to the rescue

A Pod can have multiple application containers as well as several so-called InitContainers, which are executed in sequence and until termination, before the application containers are started. If one InitContainer fails, then the restartPolicy of the deployment applies to it.

The InitContainers feature also brings other benefits, such as running tools that may not be bundled in an application’s Docker image. In our case, the pg_isready command is not installed in our webapp Docker image, yet wouldn’t it be nice to be able to use it to verify that the database is ready to accept connections ?

We can verify this command by running it with kubectl run in a container based on the postgres:9.6.5 Docker image:

$ kubectl run check-db-ready \
--image=postgres:9.6.5 \
--restart=Never \
--command -- /bin/bash -c \
"pg_isready -h postgres -p 5432 && echo $?"
pod "check-db-ready" created
$ kubectl logs check-db-ready
postgres:5432 - accepting connections
$ kubectl delete pod check-db-ready # a bit of house cleaning...

(The pg_isready command exits with the return code 0 if the server is ready, 1 if the server is still starting, 2 if the server is not started yet and 3 is something else happened.)

Great! So now we can run the pg_isready -h postgres -p 5432 && echo $? command in an InitContainer of the webapp Deployment to verify that the database is accepting connections. See in particular the lines 15–20 of the deployment manifest below:

Let’s now see how the InitContainers affect the webapp pod startup when there’s no database pod/service available:

$ kubectl create -f templates/webapp-deployment.yml
deployment "webapp" created
$ kubectl get pods
webapp-7cc6fffcd4-7drg2 0/1 Init:0/1 0 30s

Instead of juggling between CrashLoopBackOffand Error statuses as we’ve seen in the first part of this article, the webapp pod is now blocked in an Init:0/1 status, meaning that it’s waiting for the first InitContainer to complete successfully.

Checking the pod’s status and events

Checking a pod’s events is a good way to understand what’s happening at the containers level:

$ kubectl describe pod/webapp-7cc6fffcd4-7drg2
Name: webapp-7cc6fffcd4-7drg2
Init Containers:
Container ID: docker://aa63d4a27cd8
Image: postgres:9.6.5
until pg_isready -h postgres -p 5432 -U postgres; do echo waiting for database; sleep 2; done;
State: Running
Started: Sun, 10 Dec 2017 10:59:06 +0100
Ready: False

An InitContainer named check-db-ready is running but as expected, it is not ready yet. Checking the logs of this particular container shows what’s happening:

$ kubectl logs -f webapp-7cc6fffcd4-7drg2 -c check-db-ready
waiting for database
postgres:5432 - no response

(Note the -c argument to select the container on which the kubectl logs command applies.)

As one would expect, once the database is finally deployed, the InitContainer completes and the webapp starts with a successful connection to the backend.

$ kubectl create -f templates/database-deployment.yml
deployment "postgres" created
$ kubectl create -f templates/database-service.yml
service "postgres" created
$ kubectl logs -f webapp-7cc6fffcd4-7drg2 -c check-db-ready
waiting for database
postgres:5432 - accepting connections
$ kubectl get pods
postgres-77b6bf95b7-mshzd 1/1 Running 0 1m
webapp-7cc6fffcd4-7drg2 1/1 Running 0 11m

We’ve covered a basic use-case of the InitContainers feature in this article, showing how it can be used to verify some pre-conditions before the application container starts.

Further documentation

You can check the official Kubernetes documentation if you want more information about it.



Xavier Coulon

Halftime dad of two. Swimmer/cyclist/runner and occasionally triathlete. I develop tools for developers on OpenShift.