Container Wars: Across the Multiverse of Containers

In the realm of modern computing, containers have emerged as a transformative technology, redefining the way applications are developed, deployed, and managed. These lightweight, portable, and self-sufficient units encapsulate applications and their dependencies, ensuring consistency across various computing environments. The inception of containers dates back to the early 1970s, tracing its roots to UNIX-based technologies like chroot, and evolving significantly to usher in an era of standardized, scalable, and efficient software deployment methodologies.

Today, containerization stands as a pivotal force driving innovation in software development and deployment practices, offering a multitude of benefits spread across multiple vendors both commercial and free. This has created a multiverse of choices for container-based deployments and picking the right choice for your needs is of paramount importance.

In this article, we’ll be navigating these universes to identify the range of options that are at our disposal and can be used in managing containerized workloads. We’ll delve deep into common tools like Docker, exploring how they operate under the hood, and also examine emerging, free-to-use tools that are reshaping the industry as we know it.

Docker Universe — The realm of the reigning king

Containerized application deployment and orchestration have become the norm of application delivery and Docker is at the forefront of it, paving the way for convenient container app development, deployment, and orchestration. Debuted in 2013, Docker has laid the foundation for container-based application development to go mainstream and quickly it exploded in popularity.

The main products provided by Docker are

• Docker Engine: Docker’s core that is responsible for container creation and runtime management.
• Docker Desktop: All-in-one solution to build and run containers with an intuitive UI
• Docker Hub: World’s largest (as of today) library for container images, allowing for both public and private repositories

Apart from the above main products, Docker provides additional features like Docker Compose, a configuration-driven way of spinning up multi-container applications, and Docker Swamp Mode, the capability to manage a cluster of Docker engines.

Architectural components of Docker

Fig 1. Docker under the hood

Docker CLI

The client side of docker, which is a command line interface to key in instructions for the docker backend to execute

Docker Server

The backend of docker, is responsible for all the heavy lifting when it comes to container handling. It has the following core components.

• Docker daemon, also referred to as dockerd is the main backend daemon of docker that runs continuously as a service, listening in on any API requests and communicating with other required daemon services to manage, at a higher-level, the containers, images, storage, and other network components
• containerd, a backend daemon service developed by Docker that takes care of the low-level container and storage management tasks, image distribution, and network attachment via a set of gRPC endpoints exposed by the service.
• runc, is a lightweight binary that handles the actual running of containers as per the OCI specification, by interfacing with namespaces and cgroups required for a container. In the early stages of Docker, they were reliant on LXC (short for Linux containers) as their execution environment, which was Linux-specific. Later they made the use of LXC optional and started using libcontainer, an in-house solution for LXC with more flexibility in the selection of isolation technology. As of now, libcontainer is part of runc, maintained by the Open Container Initiative

Apart from the components, containerd-shim also plays a key role in the container management process (excluded from the diagram for brevity) by allowing for deamonless containers. A containerd-shim process is started per container, sitting between containerd and runc.

Registry

A Registry is an open-source implementation for storing and distributing container images that have been donated to the CNCF (Cloud Native Computing Foundation). Registry now goes under the name of Distribution, based on which Docker’s official cloud registry, Docker Hub, is implemented.

Docker Desktop

With the growing popularity of Docker, they came up with Docker Desktop to provide an all-in-one, UI-based solution for their users to better utilize Docker across platforms. Docker Desktop managed to hide some of the complexities the users would face if they were to run Docker on Windows or macOS with different virtualization techniques. For example, Docker Desktop for Windows gives you the option to run it with Hyper-V or on WSL whereas on macOS it depends on HyperKit, that is based on Apple’s Hypervisor Framework.

Docker Desktop provides a user-friendly interface for creating and managing containers directly on your local machine with important security features provided via Hardened Docker Desktop. Apart from the standard Docker features that are bundled with Docker Desktop, it provides the capability to bring in extensions to make the developer's life more convenient.

In August 2021, Docker changed their subscription plan to make Docker Desktop commercial, providing feature-rich subscription categories to better serve its users.

CNCF Universe — The realm of the open-source

Kubernetes, an open-source container orchestration tool developed by engineers at Google in 2014, later donated to the CNCF, has dominated the container orchestration tools market with a commanding 90+% market share (based on statistics by the CNCF). With the introduction of CRI (Container Runtime Interface) back in 2016, Kubernetes brought in a layer of abstraction for container runtimes and Docker didn’t work well with it given the high-level nature of Docker, as it was intended to be used by actual users. This gave birth to a bridge component which we come to know as the Dockershim. In late 2020, Kubernetes released version 1.20, deprecating the use of Dockershim, paving the way for the removal of Docker as an underlying container runtime in Kubernetes (full removal was after v1.24 in 2022).

While these events were taking shape in the container orchestration realm, in late 2017, Docker donated its core container runtime containerd to the Cloud Native Computing Foundation (CNCF). Making containerd open source assured various users in the container ecosystem that the core container runtime component would remain stable, and that the community would have a say in its advancement. This event alignment opened the floodgates for alternative runtimes to gain more traction and opened up a whole new universe as a result.

Around 2021, reports suggested that container usage showed a transition towards containerd with the new development around Kubernetes CRI, and doubled the growth the following year as well. In line with these statistics, Docker usage has shown a significant reduction as organizations moved to newer versions of Kubernetes that don’t support dockershim.

Containerd setup and usage

As explained in containerd official documentation (https://containerd.io/);

containerd is available as a daemon for Linux and Windows. It manages the complete container lifecycle of its host system, from image transfer and storage to container execution and supervision to low-level storage to network attachments and beyond.

To get started with containerd, you have access to a lightweight CLI called nerdctl (contaiNERD CTL). Simply put, nerdctl is to containerd what docker is to dockerd.

For Linux systems:

brew install nerdctl

and a few additional dependencies to be installed based on your use case

• CNI plugins: for using nerdctl run.
• BuildKit (OPTIONAL): for using nerdctl build. BuildKit daemon (buildkitd) needs to be running.
• RootlessKit and slirp4netns (OPTIONAL): For rootless mode

nerdctl release (full) bundle would provide the required dependencies

For Windows:

Linux containers can be managed with WSL2 and it’s known to provide experimental support for Windows containers.

For macOS:

Given the architectural differences in macOS, a Linux virtual machine is required to run nerdctl. The recommended approach to achieve this is to use the lima VM, which has built-in nerdctl support.

$ brew install lima
$ limactl start
$ lima nerdctl run <image of your choice>

Following nerdctl commands list will give you an idea as to how similar it is to Docker CLI.

Fig 2. nerdctl commands list, showing that it’s a drop-in solution for docker CLI

As of now, there are various features made available with nerdctl that are not yet available with Docker.

Containerd and Nerdctl leading the way

With containerd becoming open-source and the industry leaning more and more towards open-source tools, new container management tools that use containerd under the hood started to make headlines.

• Rancher Desktop

Rancher Desktop is an open-source project that brings Kubernetes and container management to the desktop for Windows, macOS, and Linux. It’s an Electron-based application that uses a virtual machine on macOS and Linux and Windows Subsystem for Linux v2 for Windows, to run containerd/Docker and Kubernetes

Fig 3. Rancher Desktop architecture (Ref: https://rancherdesktop.io/)

• Finch

Finch is an open source client for container development. Its simple installer provides a minimal native client along with an opinionated distribution of other open source components. Rather than creating even more options to reason about and choose from, Finch aims to help promote other projects by making it easy to install and use them, while offering a simple native client to tie it all together. (https://github.com/runfinch/finch)

Finch is a good all-in-one container management solution with containerd as its container runtime, for macOS-based development needs. It neatly puts together all the open-source tools required (containerd and nerdctl for container runtime, BuildKit for container building, and Lima VM for a Linux virtual environment to run them on) to get a container management solution up and running and use it as a drop-in replacement for Docker.

All these tools, including containerd, are reshaping the container development landscape by providing free-to-use (Apache 2.0 licensed), highly customizable solutions for managing enterprise-scale workloads, tasks that were historically entrusted to Docker.

Divergent Universe — The realm of the daemon-less

Red Hat, a pioneering figure in the open-source software realm, is acclaimed for its suite of enterprise solutions, and it’s been dabbling with container-related technologies for more than a decade. Developed in 2011, tools like OpenShift, an industry-leading enterprise Kubernetes platform that provides a comprehensive solution for deploying, managing, and scaling containerized applications, showcases the company’s maturity in delivering cloud-native containerization solutions.

The latest addition to their suite of products is Podman, a powerful and secure container engine that supports managing and orchestrating containers in Linux environments. Distinguishing itself from other container engines, Podman offers a unique approach by operating as a daemonless tool, which grants users the capability to manage containers and pods without a central daemon.

When using Podman, users are given the freedom to choose from a variety of different runtimes that contribute to the execution and management of containers. This ranges from the default and most widely used runtime runc to alternative runtimes like crun and Kata Containers, which offer distinct advantages such as improved performance or enhanced security. This diverse array of runtimes empowers users to select the most suitable runtime for their specific containerized workloads, whether prioritizing performance or security while complying with OCI standards.

Client-server vs Fork-exec

One of the most prominent container creation techniques that have been around for some time is using a daemon process that spawns a container from it (as a child) in response to CLI prompts, which is defined as a client-server architecture. Despite its widespread adoption, many have come to realize somewhat of a concerning security implication that stems from the fact that the daemon process runs as root.

In 2019, a new way of achieving containers in a much more secure way came out of an implementation done by engineers at Red Hat, which we have come to know as Podman. Its daemonless approach to container creation uses a common Unix feature known as Folk-exec.

As the name suggests, this model involves two distinct operations. Firstly, “fork” creates a copy of the existing process, and then “exec” replaces this copied process with another executable file. In Podman, this model is used to initiate a new container, where “fork” duplicates the initial Podman process and “exec” launches the container’s primary process, isolating it within its environment.

Fig 4. A simplified view of Fork-Exec flow in Podman

If we dig a bit deeper into the actual implementation of Podman, some may argue that it’s not exactly daemonless due to its use of a monitoring tool called conmon (short for container monitor). This monitoring process starts before the start of the container and provides the instructions required to start the container to the underlying container runtime and exits with the exit code of the container. Conmon also takes care of forwarding logs and cleanup actions once the container has exited

Fig 5. Podman together with conmon managing containers

With this daemonless approach, Podman has managed to gain an edge in areas such as security compared to its competitors. Having a daemonless runtime for containers would mean that containers can run as a non-root user, offering improved security by reducing the attack surface. Rootless containers provide a layer of isolation between the containerized applications and the host system, mitigating the potential impact of security vulnerabilities or breaches within the container.

By default, Podman leverages user namespaces and control groups (cgroups) to achieve this functionality. This enables users to work with containers in a more secure environment, ensuring that even if a container is compromised, it has limited access and impact on the underlying host system. This heightened security posture makes Podman an attractive choice for environments where security is a top priority.

In addition to the above, Podman introduces the concept of pods that mirrors the functionality found in Kubernetes, allowing users to group multiple containers and manage them as a single unit. Pods provide a shared context for containers within a network namespace, enabling them to communicate over localhost and share volumes seamlessly. This capability streamlines the deployment of interconnected services or applications by grouping related containers, fostering better coordination and resource utilization. Users can orchestrate and control various containers within a pod, facilitating efficient inter-container communication while maintaining isolation from other pods.

Podman Desktop

With the rising demands for efficient and seamless containerization solutions, Podman Desktop emerges as an optimal choice tailored for developers seeking a robust and user-friendly container development environment. Podman Desktop offers developers a familiar and accessible platform for creating, managing, and deploying containers without the overhead of a virtual machine, especially on platforms like Windows and macOS where it lacks Podman native support. This tool streamlines the container workflow by providing a lightweight, yet powerful, container engine that runs directly on the user’s desktop. Additionally, Podman Desktop can;

• Select containers to run as a Pod.
• Play Kubernetes YAML locally without Kubernetes.
• Generate Kubernetes YAML from Pods.

Conclusion

Although we perceive all these technologies existing in their realms as single possibilities, it is more like “Everything everywhere all at once”. As of now, this container multiverse has turned into a level-playing field with multiple vendors bringing in different solutions. It’s up to the users of containers to pick either a feature-rich, mature, commercial tool, a fully configurable open-source offering by the CFCN, or a revolutionary, security-focused, free-to-use tool.

Will you stick to one universe or will you choose to navigate these alternate universes to experience various possibilities?