buildkit-machine: A brand new project to enable building/pushing container images without requiring a Docker Daemon based on BuildKit

Special thanks to Erkan Zileli and Furkan Türkal for their effort while developing this project

In the previous blog post, we talked about a brand new toolkit (lima + nerdctl + rancher-desktop) that we can use to work with container images, and in that blog post, we mentioned the buildkit-machine project a couple of times. Therefore, I recommend you look at that blog first to understand where the idea of building such a project came from.

In a nutshell, we should think of BuildKit as a brand new builder engine for enabling performance, security, and a bunch of new features while building container images. More technically, BuildKit is a new project under the Moby umbrella for building and packaging software using containers. It’s a new codebase meant to replace the internals of the current build features in the Moby Engine. BuildKit was shipped with the Docker Engine on 18.06.

Key BuildKit Features according to the Moby BuildKit Project

• Automatic garbage collection
• Extendable frontend formats
• Concurrent dependency resolution
• Efficient instruction caching
• Build cache import/export
• Nested build job invocations
• Distributed workers
• Multiple output formats
• Pluggable architecture
• Execution without root privilege

You can enable BuildKit in two different ways. Either by setting the environment variable at the point of performing the build like the following:

export DOCKER_BUILDKIT=1

As an alternative, you can enable BuildKit by default for the Docker Daemon by adding the following steps to its configuration file located under /etc/docker/daemon.json and do not forget to restart the Docker Daemon to enable this:

{ "features": { "buildkit": true } }

🚨 Do not forget, your docker daemon is running in a VM in your macOS environment virtualized by using hyperkit, which means that you should get access to that shell to edit this configuration file. Don’t worry, and this is as easy as running the following command:

$ docker run -it — rm — privileged — pid=host justincormack/nsenter1

👉 How do I access the Docker CE virtual machine on macOS BigSur?

But, when you switch your build engine to BuildKit while using docker, it only helps you to get better caching and performance support which means that you can’t use all the BuildKit features it provides. To use all the features provided by the BuildKit, the buildx, Docker CLI plugin for extended build capabilities with BuildKit, the project has been developed. I highly recommend you to take a look at that project if you want to build multi-arch container images because this is where buildx really shines.

The BuildKit project itself consists of two key components. The first is the build daemon buildkitd. The second is a CLI tool called buildctl which facilitates command-line communication with the buildkitd instance. But, as you all know, the real power of creating containers comes from three Linux primitives: changing the root filesystem (chroot), namespaces (unshare), and finally, control groups (cgroups) which means that to work with those daemons such as docker daemon, buildkitd, containerd, we need to put them in a VM virtualized by some kind of hypervisors such as QEMU (in a higher level lima), moby/hyperkit, or a docker container.

QEMU (Quick EMUlator), emulates the machine’s processor through dynamic binary translation and provides a set of different hardware and device models for the machine, enabling it to run a variety of guest operating systems. It can interoperate with Kernel-based Virtual Machine (KVM) to run virtual machines at near-native speed. QEMU can also do emulation for user-level processes, allowing applications compiled for one architecture to run on another.

At the time of writing this, the only way of experimenting with buildkitd features in a macOS environment is that running buildkitd in a containerized way like the following:

$ docker run -d — name buildkitd — privileged moby/buildkit:latest
$ export BUILDKIT_HOST=docker-container://buildkitd
$ buildctl debug workers

Also, we are working on a issue to add an example of Buildkit to the examples folder. Once we do that, you will be able to run Buildkit with socket forwarding enabled:

$ export BUILDKIT_HOST=$(limactl list buildkit — format ‘unix://{{.Dir}}/sock/buildkitd.sock’)
$ buildctl debug workers

https://github.com/lima-vm/lima/pull/653

So, this is where buildkit-machine comes into play. Actually, we started as a proof-of-concept project to provide an alternative way of running the buildkitd daemon and making it accessible in the macOS environment in two different ways, either by connecting it over TCP connection, alternatively, over a socket. The first thing we need to do is find a way to virtualize buildkitd. There are various ways of doing that such as using moby/buildkit, QEMU, etc. But we want to do it with security in mind also which brings us to Rootless containers. Then, we delve deeper into Rootless containers to learn more about them which helps us to bring more security to the project. There is a special section in buildkitd to show people how they can use buildkitd in a rootless way which brings you to a project called rootlesskit.

RootlessKit is a Linux-native implementation of “fake root” using user_namespaces(7). When we bring all the pieces together, our way led to the already existing lima project. As I mentioned at the beginning of the blog post, we talked about the lima project in detail in our previous blog post, so, we won’t go into the details of the lima project but it does all the hard work for us such as facilitating QEMU under the hood to create a VM, running buildkitd in a rootless way by utilizing rootlesskit. So, we facilitate lima to do these tasks. Additionally, we use SSH to open a TCP connection to that buildkitd socket or forward it to the host VM from the guest VM which means that macOS users now be ab

le to connect that daemon in two different ways and start using buildctl to interact with the daemon to work with containers.

Many people do not realize that containers are Linux. As such, Linux containers cannot run natively on macOS. Therefore, the containers must run in a Linux virtual machine (VM), and a limactl interacts with that VM, which is Lima. Lima launches Linux virtual machines with automatic file sharing, port forwarding, and containerd. Lima uses QEMU under the hood. This is in line with all solutions for running containers on macOS.

To install and try this project is as easy as running the following command:

$ go install github.com/developer-guy/buildkit-machine@latest

To make it accessible Buildkitd Daemon over socket:

$ buildkit-machine start buildkitd --unix $(pwd)/buildkitd.sock

To make it accessible Buildkitd Daemon over TCP connection:

$ buildkit-machine start builtkitd --tcp 9999

Once you make buildkitd accessible to your host by whatever method you choose, you can be able to use client tooling such as buildctl to start building and pushing container images. There is an ongoing issue in the Docker Buildx side to let Buildx connect the remote Buildkit daemon. Once it is ready, we can use it for buildx too.

You can reach out to the project in GitHub, please do not forget to star the repository if you like it. Also, your feedbacks are very important for us to understand what we could do better, so, please don’t hesitate to share your thoughts 😇