Dude, where’s my container?

After several years of using Docker, “where’s my container?” is still one of the most common questions I hear from new users. There are plenty of top quality blogs, courses and walkthroughs online to help get you started on your Docker journey but the purpose of this post is to share some of the most common commands I use to debug a container when it fails to start.

Hello World, Goodbye World

One of the most common places for beginners to start is the “hello-world” example available here. Let’s run this container and take a look at what happens.

Congratulations, you’ve run your first container! Let’s take a look at our new container running by using the “docker ps” command.

Wait … where’s my beautiful hello-world container? This is a good question and a natural question for new users to ask since we clearly started a container and it “did stuff”. When I powered on my laptop this morning it kept running. When I started my old VM named “ubuntu-2012-do-not-delete” it kept running. Why is the container that I started not running? This is the part which requires a bit of a shift in thinking. A container needs at least one process to be running inside the container for the container to remain running. Your hello-world container started, ran a process and then stopped/exited. The container still exists, but it is not running. To list all running and stopped containers on your system you need to use the “docker ps -a” command.

Here we can see some important information about our “hello-world” container. The Container ID is assigned by Docker, the COMMAND which ran inside the container, the STATUS includes the Exit Code and a generated container name “keen_blackwell”.

This is a nice simple example to start with but in the real world you may start a container and it may not “do stuff”. The following Docker commands are useful when trying to debug why a container failed to start; logs, diff, cp, inspect and stats.

Let’s start a container which we know will fail and then we can look at how to debug why it failed. I’ve put a script into a container that will run for 20 seconds before failing with exit code 1. Let’s start the container …

Check that the container is running :

When a container is running we can view what processes are running inside the container using the “docker top” command.

After a 20 second sleep the container will exit and we have a nice exit code available under the STATUS column in the “docker ps -a” output.

At this point we can start debugging what went wrong. Everyone has their own technique or approach to debugging and the following commands are commonly useful.

docker logs

I normally start with “docker logs”. Depending on the application there may be a huge amount of logs generated, a few lines, or in some cases nothing. In this example we have the incredibly helpful “Start app” message returned in the log. This isn’t much use. Let’s dig a bit more.

docker diff

Docker “diff” is one of my favourite commands since it will show you the files/directories created, updated or deleted when a container is started. In the example below we can see that when the container “ecstatic_pare” started the /tmp directory was created(C) and file app.log was added(A). The diff command will also show you any files have been deleted(D) when the container started.

docker cp

The app.log listed above may have some useful debugging information in it. Using “Docker cp” we can copy files and directories from the container to the host.

Again this log doesn’t really provide any useful information. Let’s keep digging.

docker inspect

Next up we will use the “docker inspect” command. This command will give us a lot of really useful information about the container configuration.

What was the container supposed to do? What is the “entrypoint” or “cmd” that the container is trying to run? 🤔

Step-by-step

One of the best ways to debug why a container failed to start is to walk through each step manually. Try starting the container and run the script which is defined as a “entrypoint” or “cmd”.

In the command below we are starting a container and using the -i (interactive) and -t (tty) options. This will allow us to interact with the container terminal. We are also overriding the “entrypoint” and this will take us into the shell of the container.

When inside the container we can run the “entrypoint.sh” script manually and debug why it is failing.

The script started successfully, printed “Start app”, slept for 20 seconds and then exited with exit code 1. This is the behaviour we expected from the “entrypoint.sh” script.

In the real world, the commands listed above can help you troubleshoot why a container is not behaving as expected and is a good starting point.

Error Code 137

Another common issue for new users is error code 137 and there is an explanation here for the underlying cause. Why is this common amongst new users? Docker is an enabling technology. With very little effort it can open the door to thousands of tools, services and applications with a one line command. It’s a little bit like Pandora’s box and when new users discover this power it becomes very tempting to run ALL the containers … at the same time.

New users may be more likely to start a container, play about with it, start another, play about that one and then repeat. This is where error code 137 may bite you. This error code is not Docker specific but is being returned by the process in a container when it is out of memory. I guess my point is that containers are cool but they don’t magically vanish when they are no longer needed. It’s good practice to “docker stop” a container or remove it if it is no longer needed. As you become a more experienced docker user, you may want to set resource limits on your containers, cleanup unused resources and ensure the Docker host is only running the required containers.

docker stats

You can check the number of resources being used by running containers with the “docker stats” command. You may inadvertently have containers running that you didn’t know were running and these are eating up system resources.

If folks find this post useful or want more details on any of the commands please feel free to reach out. Thanks for reading.

#Stay Home #Stay Safe #Stay Contained