Ten tips for debugging Docker containers

1 — View stdout history with the logs command.

Lots of programs log stuff to stdout, especially when things go south. Anything that gets written to stdout for the process that is pid 1 inside the container will get captured to a history file on the host, where it can be viewed with the logs command.

$ docker run -d --name=logtest alpine /bin/sh -c “while true; do sleep 2; df -h; done”
35f6353d2e47ab1f6c34073475014f4ab5e0b131043dca4454f67be9d8ef1253
$ docker logs logtest
Filesystem Size Used Available Use% Mounted on
none 93.7G 2.0G 87.0G 2% /
tmpfs 3.8G 0 3.8G 0% /dev
tmpfs 3.8G 0 3.8G 0% /sys/fs/cgroup
... etc ...

2 — Stream stdout with the attach command.

If you want to see what is written to stdout in real time then the attach command is your friend.

$ docker run -d --name=logtest alpine /bin/sh -c “while true; do sleep 2; df -h; done”
26a329f1e7074f0c0f89caf266ad145ab427b1bcb35f82557e78bafe053faf44
$ docker attach logtest
Filesystem Size Used Available Use% Mounted on
none 93.7G 2.0G 87.0G 2% /
tmpfs 3.8G 0 3.8G 0% /dev
tmpfs 3.8G 0 3.8G 0% /sys/fs/cgroup
… etc …
Filesystem Size Used Available Use% Mounted on
none 93.7G 2.0G 87.0G 2% /
tmpfs 3.8G 0 3.8G 0% /dev
tmpfs 3.8G 0 3.8G 0% /sys/fs/cgroup
… etc …

3 — Execute arbitrary commands with exec.

Maybe the most powerful all-around tool in your kit, the exec command allows you to run arbitrary commands inside a running container.

$ docker run -d --name=exectest alpine watch “echo ‘This is a test.’ >> /var/log/test.log”
70ddfdf5169e755177a812959808973c92974dba2531c42a21ad9e50c8a4804c
$ docker exec exectest cat /var/log/test.log
This is a test.
This is a test.
This is a test.
This is a test.
$ docker run -d --name=exectest alpine watch “echo ‘This is a test.’ >> /var/log/test.log”
91e46bf5d19d4239a2f30af06669d4263580a01187d2290c33d7dee110f76356
$ docker exec -it exectest /bin/sh
/ # ls -al /var/log
total 12
drwxr-xr-x 2 root root 4096 Mar 23 05:37 .
drwxr-xr-x 10 root root 4096 Mar 23 05:37 ..
-rw-r — r — 1 root root 192 Mar 23 05:38 test.log
/ # exit
$

4 — Override the ENTRYPOINT.

Every docker image has an entrypoint and command, whether defined in the dockerfile at build time or as an option to the docker run command at run time. The relationship between entrypoint and command can be a little confusing, and there are different ways to use them, but here is one setup that follows best practices and gives you a lot of customization ability.

ENTRYPOINT [“python”, “manage.py”, “runserver”]
docker run -d -p 80:80 --entrypoint /bin/sh /myrepo/mydjangoapp

5 — Add options with the CMD.

Back to that confusing relationship between ENTRYPOINT and CMD. Here’s one way to look at it: the ENTRYPOINT defines the process that runs as PID 1 in the container, and the CMD adds options to it. If you don’t specify an ENTRYPOINT but you do specify a CMD then the implicit entrypoint is /bin/sh -c. CMD, like ENTRYPOINT, can be specified on the command line when docker run is called, which makes it another powerful tool for modifying container behavior. However, the usage is a little different.

docker run -d -p 80:8000 /myrepo/mydjangoapp 0.0.0.0:8000
ENTRYPOINT [“python”, “manage.py”, “runserver”]
CMD ["0.0.0.0:8000"]

6 — Pause and unpause a container.

I doubt you’ll have use for this one very often, but it’s cool so I’m throwing it in here anyway. Using the docker pause command you can pause all of the processes inside a container.

$ docker run -d --name=pausetest alpine /bin/sh -c “while true; do sleep 2; date; done”
e81e1bc519e4eb2e30a1c1e57198f0060147fa749ff8f93ba4f69bdf8a114311
$ docker attach pausetest
Wed Mar 23 06:21:40 UTC 2016
Wed Mar 23 06:21:42 UTC 2016
$ docker pause pausetest
pausetest
$ docker unpause pausetest
pausetest
mark@viking:~/workspace/cluster-iperf$ docker attach pausetest
Wed Mar 23 06:21:40 UTC 2016
Wed Mar 23 06:21:42 UTC 2016
Wed Mar 23 06:22:06 UTC 2016
Wed Mar 23 06:22:08 UTC 2016

7 — Get process stats with the top command.

The docker top command is exactly what it sounds like: top that runs in the container.

$ docker run -d — name=toptest alpine:3.1 watch “echo ‘Testing top’”
fc54369116fe993ae45620415fb5a6376a3069cdab7c206ac5ce3b57006d4241
$ docker top toptest
UID PID … TIME CMD
root 26339 … 00:00:00 watch “echo ‘Testing top’”
root 26370 … 00:00:00 sleep 2

8 — View container details with the inspect command.

The inspect command returns information about a container or an image. Here’s an example of running it on the toptest container from the last example above.

$ docker inspect toptest
[
{
“Id”: “fdb3008e70892e14d183f8 ... 020cc34fec9703c821”,
“Created”: “2016–03–23T17:45:01.876121835Z”,
“Path”: “/bin/sh”,
“Args”: [
“-c”,
“while true; do sleep 2; echo ‘Testing top’; done”
],
… and lots, lots more.
}
]
  • Current state of the container. (In the “State” property.)
  • Path to the log history file. (In the “LogPath” field.)
  • Values of set environment vars. (In the “Config.Env” field.)
  • Mapped ports. (In the “NetworkSettings.Ports” field.)

9 — View image layers with the history command.

This one is more of a build time diagnostic tool, but the questions it answers sometimes come up in debugging situations as well. The docker history command shows the individual layers that make up an image, along with the commands that created them, their size on disk, and hashes.

$ docker history alpine

10 — Run one process in each container.

This last one is less of a debugging tip, and more of a best practice that will definitely make debugging, and reasoning about the state of your container a lot easier. Docker containers are meant to run a single process that is PID 1 inside the sandbox. You can make that process a shell and spin up a bunch of stuff in the background, or you can make it supervisord and do the same, but it’s not really what containers are good at and it hobbles systems meant to manage and monitor them.

Conclusion

There are a few other advanced techniques that didn’t fit into my arbitrary limit of ten tips, but perhaps they’ll be a subject for another time. They include things like advanced logging scenarios, exporting and importing container file systems, and commands to diff image layers. In the meantime the tips in this article will hopefully get the top off that black box and make your transition to running containers a little easier.

--

--

Senior Devops Engineer at Olark, husband, father of three smart kids, two unruly dogs, and a resentful cat.

Love podcasts or audiobooks? Learn on the go with our new app.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Mark Betz

Mark Betz

Senior Devops Engineer at Olark, husband, father of three smart kids, two unruly dogs, and a resentful cat.