6 ways to debug an exploding Docker container

Tim Perry
2 min readApr 5, 2016

--

Everything crashes.

Sometimes things crash when they’re running inside a Docker container though, and then all of a sudden it can get much more difficult to work out why, or what the hell to do next. Docker’s great, but it’s an extra layer of complexity that means you can’t always easily poke your app up close any more, and that can really hinder debugging.

If you’re stuck in that situation, here’s my goto debugging commands to help you get a bit more information on what’s up:

  1. docker logs <container_id>
    Hopefully you’ve already tried this, but if not, start here. This’ll give you the full STDOUT and STDERR from the command that was run initially in your container.
  2. docker stats <container_id>
    If you just need to keep an eye on the metrics of your container to work out what’s gone wrong, docker stats can help: it’ll give you a live stream of resource usage, so you can see just how much memory you’ve leaked so far.
  3. docker cp <container_id>:/path/to/useful/file /local-path
    Often just getting hold of more log files is enough to sort you out. If you already know what you want, docker cp has your back: copy any file from any container back out onto your local machine, so you can examine it in depth (especially useful analysing heap dumps).
  4. docker exec -it <container_id> /bin/bash
    Next up, if you can run the container (if it’s crashed, you can restart it with docker start <container_id>), shell in directly and start digging around for further details by hand.
  5. docker commit <container_id> my-broken-container &&
    docker run -it my-broken-container /bin/bash
    Can’t start your container at all? If you’ve got a initial command or entrypoint that immediately crashes, Docker will immediately shut it back down for you. This can make your container unstartable, so you can’t shell in any more, which really gets in the way.
    Fortunately, there’s a workaround: save the current state of the shut-down container as a new image, and start that with a different command to avoid your existing failures.
    Have a failing entrypoint instead? There’s an entrypoint override command-line flag too.
  6. HTTP Toolkit for Docker. An open-source tool I’ve been building to help with Docker debugging. If you think there might be any kind of networking involved in your issue, you can run the same command in an intercepted container, and instantly capture all the HTTP(S) or WebSockets that gets sent before it crashes. You can even breakpoint & rewrite traffic, to see if you can modify the responses to stop your container crashing manually :-)

--

--