Troubleshooting Strategies for Distroless Containers: Unveiling Solutions in Minimalist Environments

4 min readFeb 8, 2024

What are Distroless Containers ?

Distroless containers are a concept in the containerization space, particularly popularized by Google. The idea behind distroless containers is to create minimalistic container images that include only the necessary dependencies to run a specific application, without any unnecessary components or a traditional operating system distribution.

They do not contain package managers, shells or any other programs you would expect to find in a standard Linux distribution.

Why Distroless ?

When building Docker images, developers often start with a base image that includes a full OS. For example, one might use a base image that contains a lightweight Linux distribution. However, most applications don’t require the entirety of that distribution.

By only including the application and its direct dependencies, Distroless images effectively reduce the attack surface, decrease size, and optimises performance.

Key characteristics

Minimalistic base image
No Operating System packages
Focused on Application dependencies
Security benefits
Smaller image size

Even though we have a lot of advantages for distroless images, it also comes with certain disadvantages and the main disadvantage that we are going to address here is its’ Debugging Capabilities.

Why Debugging is important?

Debugging is crucial in a Docker image as it allows developers to identify and fix issues specific to containerized applications.

Debugging within a container ensures reproducibility of problems in an environment similar to production and helps manage dependencies within the isolated container environment.

It is essential for optimising image building, resolving application bugs, and ensuring security compliance.

You might wonder how we could debug a distroless image since we already know that there is no package managers or even a shell to begin with!

To demonstrate this, let me take a simple example for distroless image and check if I can perform a exec in the container.

docker run -d --rm \
  --name debug-distroless gcr.io/distroless/nodejs \
  -e 'setTimeout(() => console.log("Done"), 99999999)'

But when I run docker exec command, it fails.

$ docker exec -it debug-distroless sh

OCI runtime exec failed: exec failed: unable to start container process: exec: "sh": executable file not found in $PATH: unknown

But luckily, we have a few workarounds that we can perform in order to achieve debugging capabilities.

Debugging using images with debug tag

Google provides debugging capabilities with distroless images with debug tag.

So, let us create a new container with gcr.io/distroless/nodejs:debug image.

docker run -d --rm \
  --name debug-distroless gcr.io/distroless/nodejs:debug \
  -e 'setTimeout(() => console.log("Done"), 99999999)'

When we exec into this container, we now can reach it’s shell and perform linux commands.

However, if you need a slightly less common Linux utility, you will neither find it in the above container nor will be able to install it — the :debug base images still lack a proper package manager.

Using docker run with shared namespaces

With a simple docker run --pid container:<target> --network container:<target> you can start a debugger container that will share the pid and net namespaces of the target.

The debugger container can bring its own tools, and using ps or ip in it will show the exact same process tree and the network stack as the target container sees itself.

My investigation showed that there are three namespaces that can be shared this way: Process (pid), Network (net) and Inter-Process Communication (ipc).

Now, let us create a container and try to attach a debugger and see what happens.

$ docker run -d --rm \
  --name debug-distroless gcr.io/distroless/nodejs \
  -e 'setTimeout(() => console.log("Done"), 99999999)'

As the container gets started, now we can create a debugger container.

$ docker run --rm -it \
  --name debugger \
  --pid container:debug-distroless \
  --network container:debug-distroless \
  busybox \
  sh

The --pid container:debug-distroless and --network container:debug-distroless options connect the debugger container to the same process and network namespace as the debug-distroless container.
• It starts a shell (sh) inside the BusyBox container, allowing you to inspect and debug the processes running in the debug-distroless container.

Now when I try to exec into the container , it works fine!

Even though it is by far the best approach according to me to attach a debug container since it lacks most of the downside of other approaches, it does have a disadvantage.

The docker run (or docker create) command doesn't allow sharing the mnt namespace. Technically, it's possible for two containers to share the same filesystem

It means that the file system you see while in the debugger’s shell is not the file system of the target container.

But we can access the file system of the target container since we have the shared pid namespace by using the following command.

 ls -l /proc/1/root/

Conclusion

In summary, choose distroless containers for minimalistic, secure, and efficient deployments, prioritizing lightweight and secure production environments. When debugging distroless containers, perform strategic approaches such as attaching to a running container, using specialised debugging tools, and creating a separate debugging container to gain insights into the application’s runtime environment. Striking a balance between minimalism and effective debugging ensures the development of resilient containerized applications.