Combining Docker Multi-Stage builds and Health Checks

Photo by chiasheng tai

In release 1.12 Docker added the HEALTHCHECK instruction. While docker psallows you to see if your application has crashed, the HEALTHCHECKinstruction gives you visibility on the readiness of the application.

Note: If you are using kubernetes, kubernetes uses its own container readiness and liveness probes and subsequently ignores the docker HEALTHCHECK entry. For people using docker swarm, the rest of the post will still apply. It appears the health check instruction was added to docker in an attempt to attain feature parity with k8s.

It is quite easy to see how this would be a handy ops tool, so let’s take a look how we actually use this practice.

Let’s use a scenario I am sure we’re all overly familiar with: a health check with a webserver. The health check, in this case, could involve hitting an endpoint. Indeed, the docker documentation will have you calling curl like so:

HEALTHCHECK --interval=5m --timeout=3s CMD curl -f http://localhost/ || exit 1

It should be said that what exactly the health check endpoint actually does is up to you; one option could be checking your database connection (if you have one) with a SELECT 1.

This is fine and dandy and all, but we want to utilize the multi-stage buildcapability to get the smallest container size we can. A FROM scratch image with only our statically compiled go webserver will not have curl kicking around. After some googling, there seemed to be not on recommendation for how to best combine these two things. However, three solutions came to mind:

1. Use FROM alpine:latest and install curl
2. Build curl statically and add it to aFROM scratch container
3. Build a tiny client service in go as a replacement for curl and add that to a FROM scratch image instead

Option 1

This is our low bar, with this simple Dockerfile starting out at 5.39MB:

Probably overly paranoid with the rm -rf…

> docker images alpine-curl
REPOSITORY TAG IMAGE ID CREATED SIZE
alpine-curl latest 96c35a417d8a About an hour ago 5.39MB

Option 2

I could not find a good precompiled statically linked version of curl kicking around on their site, so I decided to attempt to build it myself. My C/C++ knowledge was a little rusty, so with some help from a colleague, we managed to get it working:

After many attempts this seemed to work, and it will inform you the output is statically compiled to boot!

> docker build -t static-curl .
./curl: ELF 64-bit LSB executable, x86–64, version 1 (GNU/Linux), statically linked, for GNU/Linux 2.6.32, BuildID[sha1]=e3dfdfad4c2344dac3084cca07dbb658973050ed, not stripped
> docker images static-curl
REPOSITORY TAG IMAGE ID CREATED SIZE
static-curl latest bd36e73250bb 38 seconds ago 1.52MB

This advantage of this solution over option 3 is that you get the full capabilities of curl without reinventing any wheels, and a smaller attack surface than Option 1. This is probably fine for 99% of cases, but I wanted to see if I could get it even smaller…

Option 3

For our purposes, all we really needed was the ability to hit an endpoint, and, if it returned 200, return 0, otherwise 1.

So I decided to whip up a small service that reads in the endpoint and return code to verify against.

Basic go client with appropriate return values and minimal configuration.

The resultant Dockerfile would look something like this:

Compiling and compressing the simple client and then adding the binary.

Notice I am using the upx program to compress the binary even further (before upx the binary is a fat 4MB!)

--> Running in 6d24f93c96b3
-rwxr-xr-x 1 root root 1.8M Apr 5 18:09 ./docker-health-check
./docker-health-check: ELF 64-bit LSB executable, x86–64, version 1 (GNU/Linux), statically linked, stripped
> docker images tiny
tiny latest bd928d821f3c 6 seconds ago 1.85MB

Unfortunately, this is still slightly larger than our static curl build. At this point I am not sure if it is possible to get it any smaller given that it is already stripped.

However, the Go version does build substantially faster; the static curl build takes quite a while (minutes) and has to download a bunch of stuff from remote urls which can sometimes fail. This can add a noticeable amount of time to your CI/CD process if you’re including this health check into all your multi-stage container builds.

I have placed the code here if you are interested in the Go version:

https://github.com/chrisaxiom/docker-health-check

One final note, with the FROM scratch images, since you don’t have a shell, be sure to invoke the health check via the exec form:

HEALTHCHECK — interval=8s — timeout=120s — retries=8 CMD [“/curl”, “ — fail”, “-s”, “http://localhost/api/status/"]

Otherwise, you will get a strange error that looks like the following:

docker inspect — format “{{json .State.Health }}” <your container>

Derp…

Happy health checking!

Written by Christopher Scott
Edited by Yasmine Nadery