A modern toolkit to start working with container images on macOS that meets your needs without requiring a Docker Daemon or even Docker Desktop
Co-authors: Erkan Zileli Furkan Türkal
Most of us stepped into the containerization world with Docker. So, we’ll always be grateful to Docker for that. But, to be honest, even we’re working with Docker, we know that it is not the only solution to work with container images but the most friendly one, no doubt. We were swept away by this feature, and suddenly it became a de-facto standard for us to work with container images daily. By doing so, our adaptation to Docker has increased, and whenever we try to find another tool to work with container images, we always seek the same ease. But, then in 2020, if you related with Kubernetes somehow, we were shocked by an item in the CHANGELOG of the Kubernetes v1.20 release and says, “Docker support in the kubelet is now deprecated and will be removed in a future release.” 😱 A little story behind it is that Docker is cool and useful because it has many UX enhancements that make it really easy to interact with people when doing development work, but for Kubernetes, these UX enhancements are not necessary, also don’ forget, Docker is an entire stack with many capabilities such as volume management, networking not just a container runtime.
Kubernetes is as extensible as possible by its design, which means that you can adapt any container runtime as long as it’s compliant with CRI (Container Runtime Interface). Docker, by default, does not fit into this category which means that it requires another component to be compliant with CRI called dockershim, which causes many troubles for the subsequent maintenance of Kubernetes. This is because Dockershim has become a heavy burden for Kubernetes maintainers and where this is where containerd comes into play also. containerd on the other hand is fully compliant with CRI and does not require any other component to manage. Also, there is no need to worry about its stability because containerd is a pretty mature technology. These benefits make containerd the most powerful one of the candidates as default container runtime for Kubernetes. But if you are curious like me, you had to search about how Docker works internally, which means that you’ve already had to know that Docker uses containerd as a higher-level runtime and containerd uses runc as a lower-level runtime under the hood. So, you shouldn’t have panicked. But, take it as a chance to search for another tool to do the same thing as Docker did. Also, suppose you’ve already worked on container images a quite an amount of time. In that case, you have to know that you don’t worry about building container images with another tool as long as it is compliant with the OCI Image format, which is the secret sauce behind Kubernetes’ ability to run container images with different container runtimes. So, let’s focus on the future.
We all know that Docker uses the client (Docker CLI)/server (Docker Daemon known as dockerd) architecture model. We also understand that the technology that makes containerization happen is Linux kernel technologies such as namespaces, cgroups, chroot, etc. So, What’s the magic behind docker working on macOS?. The answer is virtualization accomplished by the moby/hyperkit hypervisor (AFAIK), a toolkit for embedding hypervisor capabilities in your application, which means that dockerd works in a VM virtualized by the hyperkit. Why I’m telling you this is once you decide to work with containerd on your macOS environment to discover capabilities and more adapt to it because we assume you use containerd in your Kubernetes environment as a default container runtime, you had to have the same virtualization technology under the hood to let containerd working on macOS. Also, you need to have some client tooling to interact with containerd by keeping simplicity and usability in mind. This is when you should start thinking about using lima and nerdctl in your macOS environment.
Before we talk about lima, we should speak of nerdctl first. The nerdctl tool is designed as a drop-in replacement for the Docker client. You can think nerdctl is a Docker compatible CLI for containerd, which solves our first problem. Remember, we’ve always been sought the same ease in Docker CLI. By doing this replacement, we should be able to use cutting-edge features of containerd that are not present in Docker. Such features include, but are not limited to, lazy-pulling (stargz) and encryption of images (ocicrypt). So, all that’s left is to run containerd in a VM in a rootless way, which means that containerd started by a non-root user (lima in this case), enter: Lima.
To learn more about Rootless Containers, you can reach out to the official website here.
Lima is a hypervisor that launches Linux virtual machines with automatic file sharing, port forwarding, and containerd. The name of lima comes from an abbreviation of the first two capital letters of LInux MAchines. The design of Lima is similar to WSL2, but Lima focuses on macOS as the primary target host. Lima uses QEMU, which is a generic and open source machine emulator and virtualizer, as a hypervisor under the hood to achieve the virtualization thing. Lima currently does not support Windows hosts, but we can consider supporting Windows hosts, too, if there is a demand. Additionally, Lima is designed to let macOS users work with containerd by using nerdctl as a client tool. Lima can also work with other container engines such as Podman and even for non-container applications. By default, when lima launches a VM, it runs buildkitd and containerd in a rootless way and also downloads necessary client tooling around them such as buildctl, nerdctl. Everything will be set up for us. So, all that’s left is building, pulling, and running containers 🚀
We’ll be talking about buildkitd and buildctl in another post, also we’ll be introducing our brand new application called buildkit-machine in that post, please do not forget to subscribe to stay tuned with the latest updates.
Both lima and nerdctl can be installed through Brew but you don’t need to install nerdctl because nerdctl is already installed in the VM.
$ brew install lima
Once you install lima, there are a bunch of other tools that will be also installed that you might find useful such as “limactl”, “nerdctl.lima” because the actual CLI for interacting with the VM created by lima is “limactl”. “lima” and “nerdctl.lima” is just a wrapper around limactl to make some processes easier:
Once the installation has been completed, we need to start Lima:
$ limactl start demo
This will create a VM instance called demo and you will be asked about configurations but for now, let’s continue with the default one for the sake of simplicity:
In the official repository of lima, there is a folder called ./examples. In that folder, you can find additional configurations to launch a VM that is suitable for your needs.
Once the setup finishes, you should end up having something like the following:
The output above says we are ready to open a shell to the VM which is good because it means that everything went well and our VM is up and running. There are a bunch of methods to connect a VM:
# Use limactl shell command
$ limactl shell demo
lima@lima-demo:~$# Use lima and LIMA_INSTANCE environment variable
$ export LIMA_INSTANCE=demo
$ lima
lima@lima-demo:~$# Use ssh command
$ sh $(limactl show-ssh --format=args demo) lima@127.0.0.1
Last login: Thu Jan 13 19:04:38 2022 from 192.168.5.2
lima@lima-demo:~$
Once you’re done, you can use stop to stop the instance or delete to remove the instance.
$ limactl stop demo
$ limactl delete demo --forceOkay, now we’ve learned how to launch a VM with lima. Let’s continue with the fun part, building, pushing, and running container images with nerdctl 😋
As I said, you don’t need to set up nerdctl on your macOS, just use “nerdctl.lima” for that. (actually, we’ll be creating a symlink to a nerdctl binary installed by the rancher-desktop when we talk about rancher-desktop ✌️)
nerdctl.lima container run hello-world
docker.io/library/hello-world:latest: resolved |++++++++++++++++++++++++++++++++++++++|
index-sha256:975f4b14f326b05db86e16de00144f9c12257553bba9484fed41f9b6f2257800: done |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:f54a58bc1aac5ea1a25d796ae155dc228b3f0e11d046ae276b39c4bf2f13d8c4: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:2db29710123e3e53a794f2694094b9b4338aa9ee5c40b930cb8063a1be392c54: done |++++++++++++++++++++++++++++++++++++++|
elapsed: 2.8 s total: 4.4 Ki (1.6 KiB/s) Hello from Docker!
This message shows that your installation appears to be working correctly.To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
(amd64)
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bashShare images, automate workflows, and more with a free Docker ID:
<https://hub.docker.com/>For more examples and ideas, visit:
<https://docs.docker.com/get-started/>
Yay, we’ve just run our first container with nerdctl by interacting with containerd in a VM virtualized by lima 🥳
But what if we want to build a container image, how can we achieve that? Simple. By default, Lima mounts the host home directory into the guest filesystem, and also /tmp/lima is writable from both macOS and Linux.
$ mkdir -p work
$ cd work
$ cat <<EOF | xargs -I{} echo {} > Dockerfile
FROM nginx
RUN echo "hello lima" > /usr/share/nginx/html/index.html
EOF
$ nerdctl.lima build -t lima-test .
[+] Building 15.5s (6/6) FINISHED
=> [internal] load build definition from Dockerfile 0.2s
=> => transferring dockerfile: 103B 0.1s
=> [internal] load .dockerignore
..
$ nerdctl.lima image list
REPOSITORY TAG IMAGE ID CREATED PLATFORM SIZE
hello-world latest 975f4b14f326 10 minutes ago linux/amd64 20.0 KiB
lima-test latest afc24ee83b24 16 seconds ago linux/amd64 149.1 MiBI think you might be wondering what if I run the lima-test container would it be accessible from our host even if it is running on a guest VM, the answer is YES! Lima automatically maps 127.0.0.1:8080 of the guest VM (which is mapped to port 80 of the container) to 127.0.0.1:8080 of the host.
$ nerdctl.lima container run -d -p 127.0.0.1:8080:80 lima-test
4de89ef0a94f5dfdf3e59164df35d36250afd62d40eacedd069a52dea6eec078Then, please open up your default browser and visit: http://127.0.0.1:8080 ✨
To complete the whole story, we have to talk about rancher-desktop, Kubernetes, and container management to the desktop, too. On August 31, 2022, Docker announced a new subscription plan for Docker Desktop, so people started looking for an alternative solution. According to that, rancher-desktop gains more popularity because it seems a perfect solution for this because it is compatible with tooling lima and nerdctl, supports different versions of Kubernetes clusters by using k3s as a runtime which makes it very efficient and lightweight, has lovely UI, and builtin support for kim, the Kubernetes image manager. Yes, you heard me right. Rancher-desktop uses lima under the hood to set up Kubernetes cluster (k3s) and downloads many tools to make our lives easier while working with both container images and Kubernetes cluster.
As I mentioned before, we don’t need to install nerdctl because once we install rancher-desktop, it’ll be downloading a bunch of useful tools for us, and gives us a chance to create a symlink to the binary that it’s supported.
Now what? What will happen if I run nerdctl when rancher-desktop gets up and running? Let’s see:
$ nerdctl -n “k8s.io” image list
rancher/kim <none> 20704973bbdb 33 hours ago linux/amd64 46.2 MiB
rancher/klipper-helm v0.6.6-build20211022 4bf16ca677f9 33 hours ago linux/amd64 232.0 MiB
rancher/klipper-lb v0.3.4 7156afc9b247 33 hours ago linux/amd64 8.6 MiB
rancher/local-path-provisioner v0.0.20 004134f27c6e 33 hours ago linux/amd64 34.0 MiB
rancher/mirrored-coredns-coredns
... What did happen just a while ago? As mentioned before, rancher-desktop runs in a VM virtualized by lima. It runs a k3s cluster with the containerd as default runtime, which is perfect because now we can connect to that containerd and start working with it by using nerdctl downloaded by rancher-desktop.
BONUS: At the time of writing this, thanks to Bret Fisher, he made a YouTube Livestream, do not forget to watch it if you missed that, and discussed what are the free alternatives of Docker Desktop 👇
He did a really good job because he compared all of them in various categories and shared them with us through the spreadsheet.
Here is the link to the spreadsheet. 👇
