Auto Healing Docker Containers Using Python & Docker API

Disclaimer: This blog post is part of Painless Docker ‘Unlock The Power Of Docker & Its Ecosystem’, a practical guide to master Docker and its ecosystem based on real world examples.

The tool detailed in this blog post is mainly created to help Painless Docker readers manipulate and understand Docker & Docker API, but I already started using it in production.

Another tool for monitoring Docker was created for the same reason and it is totally detailed in the book, if you’re curious, you can discover some of these details in this blog post: Monitoring Docker With Python & DoMonit

Restarting A Container When A Health Check Fails

This is the main goal of creating D-ealer, a tool that I started recently. D-ealer will continuously inspect every running container and see its status, if its health check returns an error, D-ealer will restart the container.

D-ealer — How To

Let’s use this Dockerfile as an example:

FROM alpine:3.3 
RUN apk update && apk add curl && rm -rf /var/cache/apk/* 
COPY helloworld.bin /  
HEALTHCHECK --interval=5s --timeout=3s --retries=3 CMD curl -f / http://localhost:80 || exit 1 
ENTRYPOINT ["/helloworld.bin"]

This Dockerfile will create a web server image using a the following healthcheck:

HEALTHCHECK --interval=5s --timeout=3s --retries=3 CMD curl -f / http://localhost:80 || exit 1

To build and run this container, make sure you have all of the files in the github repos, build it:

docker build -t helloworld:healthcheck .

Run it:

docker run -l -d -P helloworld:healthcheck

Make sure the .bin file is executable ( chmod +x otherwise).

CONTAINER ID        IMAGE                    COMMAND             CREATED             STATUS                    PORTS                   NAMES
a79307e481b5 helloworld:healthcheck "/helloworld.bin" 47 seconds ago Up 46 seconds (healthy)>80/tcp dazzling_brattain

If localhost is unreachable at port 80, the health check will return exit 1 and we can see the container state when using the inspection command:

docker inspect --format='{{json .State.Health}}' your-container-name

In order to monitor the health of this webserver, you can use D-ealer directly from Docker:

docker run -it --name d-ealer -v /var/run/docker.sock:/var/run/docker.sock  -d eon01/d-ealer

Notice that the container helloworld:healthchek is labeled by

D-ealer will continuously check the health of all of your containers, if one of them is unhealthy, it will restart it but only if this containers has the label set to 1.

Connect Deeper

If you resonated with this article, please submit your feedback/issues in the github project page.

You can also subscribe to DevOpsLinks : An Online Community Of Diverse & Passionate DevOps, SysAdmins & Developers From All Over The World.

You can find me on Twitter, Clarity or my blog and you can also check my books: SaltStack For DevOps,The Jumpstart Up & Painless Docker.

If you liked this post, please recommend and share it to your followers.