Early look at Docker containers on RISC-V
RISC-V seems to be a hot topic these days, at least according to my Twitter timeline. If you haven’t heard about it before, it is a new hardware instruction set architecture (ISA) or in plain terms a specification about how software can communicate with the hardware. What makes RISC-V cool is that it is fully open source and promises to address some of the pain points of the current de-facto standards.
RISC-V has been in the works for quite a while but what has changed lately is that as a result to user-space ISA reaching frozen status, more and more software starts to emerge with RISC-V support, as well as cool Linux-capable hardware. Docker obviously has a big influence on the way modern applications are built and deployed, so I decided to have a look at the current state of integrating it with RISC-V.
A bunch of new and modern build tooling has been built for Docker in the latest releases. We have created a completely new builder backend BuildKit to power docker build and its new features. At DockerCon2019 we showed a new Docker CLI plugin Buildx that enables a suite of new build capabilities powered by BuildKit. Both of these projects were created with multi-platform images in mind, eg. on Docker Desktop these tools provide a great experience for developing ARM images either locally on a laptop or with remote nodes forming a build farm. We already played around with leveraging those features for adding more platform types to the Docker ecosystem by showing how you can use them to build WebAssembly applications with the new WASI system interface, so let’s see if we can show something similar for building RISC-V containers and what needs to be done to run them on RISC-V.
At Docker, we obviously prefer building all of our software from Dockerfiles and the way to build multi-platform images with Dockerfiles is with the --platform flag on docker build.
docker build --platform=linux/arm64 .
docker buildx build --platform=linux/amd64,linux/riscv64 .Buildx allows specifying multiple platforms together and will generate a multi-platform image (manifest-list) after building the Dockerfile for each of the specified platforms.
Under the hood building for a specific platform is achieved with one of the three different methods: QEMU emulation, using (multiple) remote native nodes or cross-compilation in multi-stage builds. It is probably unlikely that anyone has access to enough actual RISC-V hardware for a build farm at the moment, so I’ll leave that one out for now. The other two are viable options and let’s start with QEMU.
QEMU
QEMU is probably the easiest method for building a Dockerfile for a custom platform or running a container on a non-native architecture. Leveraging QEMU emulation is based on a binfmt_misc feature in the Linux kernel that allows registering a program that kernel will use as a proxy when loading certain executables. For example, we can set up QEMU as an executor for RISC-V binaries and after that, we can now execute them directly on an x86 system. This setup can be a bit tricky to get right but we can bundle it into a Docker image to make it easy.
docker run --privileged --rm tonistiigi/binfmt:riscvExecuting the above command will load the RISC-V emulation support into your (modern) kernel. You can now start to run RISC-V containers!
docker run tonistiigi/hello:riscv # Hello world, I am linux/riscv64docker run tonistiigi/debian:riscv uname -mo # riscv64 GNU/Linux
If you have never run a RISC-V binary before I encourage you to do it now and unlock that achievement! These commands should work with almost any Docker version.
Once you loaded the binfmt helper you can also use it in your Dockerfiles. Just remember that you need to use a base image that has RISC-V support. RISC-V is in very early stage and not yet in official images. It is possible that the images introduced in this post are the very first RISC-V images.
FROM tonistiigi/debian:riscv
RUN apt-get update && apt-get install -y vim
COPY /src /srcbuilt with:
docker build -t myapp .docker inspect myapp | grep Architecture
| "Architecture": "riscv64",docker run -it myapp vim
Cross compilation
QEMU is a nice and easy way to check programs quickly and for building Dockerfiles without making any changes. As it is doing an emulation, the performance may not be optimal and you may find unsupported code-paths. You might also wonder how these base images were built in the first place and the answer is cross-compilation.
For cross-compilation in Dockerfiles you are using multi-stage builds so that some of the stages are based on your native build architecture (where you are running the compiler) and some are using the target architecture specified with docker build --platform . We don’t want you to hardcode specific platform names in the Dockerfiles, so instead there is support for automatic platform ARGs like BUILDPLATFORM and TARGETPLATFORM that the builder will fill up automatically.
There seems to be many existing toolchains targeting RISC-V already in Docker Hub, but I didn’t know how most of them were built or if they were outdated so I created my own or more precisely I let buildroot in Dockerfile generate it for me. I also have a repository of reusable base images at https://github.com/tonistiigi/xx that automatically integrate with these build arguments. This means that the user only needs to call the compiler in the Dockerfile and the base image automatically picks the correct toolchain for the specific build.
FROM --platform=$BUILDPLATFORM tonistiigi/xx:riscv-toolchain \
AS build
WORKDIR /src
COPY hello.c .
RUN riscv64-linux-gcc -o /hello --static hello.cFROM scratch
COPY --from=build /hello /
ENTRYPOINT ["/hello"]
When built with docker build --platform=linux/riscv64 . this will produce a valid RISC-V image by actually doing the compilation in your native build stage.
But there is one problem with our Dockerfile: it is RISC-V specific and we would rather have a Dockerfile that is reusable regardless of the architecture.
FROM tonistiigi/xx:gcc-sid AS build
WORKDIR /src
COPY hello.c .
ARG TARGETPLATFORM
RUN $CC -o /hello --static hello.cFROM scratch
COPY --from=build /hello /
ENTRYPOINT ["/hello"]
Thetonistiigi/xx:gcc-sid image already has knowledge of the TARGET* build arguments and can configure the $CC binary automatically. This means that you can build a multi-platform image with a command like docker buildx build --platform=linux/amd64,linux/arm64,linux/arm,linux/riscv64 . that can now run on any of these architectures. All this without mentioning any of these architectures in the Dockerfile.
Runtime support
So now that I’ve spent some time explaining how we can build RISC-V container images you might be wondering how can we actually run them. Previously I showed how these containers can be easily emulated through QEMU but with that method, you are still running the RISC-V binaries with your native (probably x86) kernel. This is far from the end goal. For the RISC-V containers to be useful, they should run on a RISC-V kernel, with a RISC-V Docker engine.
As you might expect, upstream Docker/Moby Engine does not build on RISC-V yet. I went through different components looking at what updates and patches would be required to make them work.
Most of the Docker tools are built using the Go programming language. Unfortunately, Go does not have builtin RISC-V support yet but the work on the port is ongoing and it seems that we can hope to find it in the next release. My understanding is that everything but cgo (Go code linking against embedded C code) should work in the working branch.
For some of the components not having cgo isn’t a problem. Docker CLI and containerd could be built without cgo and only required updating some of the vendored dependencies (x/sys, x/net, boltdb etc.) that have recently gained RISC-V support. Docker daemon is a little bit more tricky and doesn’t build without cgo out of the box. But by building with some of the non-critical features like devicemapper, adding some patches to cgo code paths and adding RISC-V support to netns dependency I had a working dockerd binary.
Now things got a bit more complicated. In order to actually start a container, containerd calls into runc that creates the container based on the OCI bundle Docker has prepared. runc uses cgo for forking a child process in order to prepare the actual container process. Removing the need for cgo in here seemed a bigger job than I was willing to take at the moment. Luckily Docker doesn’t only work with runc but with any runtime that follows the OCI runtime spec. One of them is crun from Giuseppe Scrivano that is built only in C and can be built with my regular RISC-V toolchain. To be honest, building crun didn’t go completely painlessly either. For example, I found out that libseccomp that is used to secure the containers doesn’t have a package for RISC-V yet because the PR with the support is still in review (with kernel patches). So I needed to compile that myself as well. I also needed to make a small patch to crun itself to make sure that it can be tested under QEMU.
Having done that I grouped all of these components into a single multi-stage Dockerfile that can build all of the tools required to run containers on RISC-V and also a test image based on Debian to try them out.
docker run -it --privileged tonistiigi/docker:riscv| # dockerd -s overlay2 -D &>/dockerd.log &
| # docker version
Client:
Version: unknown-version
API version: 1.40
Go version: devel +f980a63 Fri May 24 20:26:57 2019 +1000
Git commit: 668a9ff8
Built: Tue Jun 4 15:55:56 2019
OS/Arch: linux/riscv64
Experimental: falseServer:
Engine:
Version: library-import
API version: 1.41 (minimum version 1.12)
Go version: devel +f980a63 Fri May 24 20:26:57 2019 +1000
Git commit: library-import
Built: library-import
OS/Arch: linux/riscv64
Experimental: false
containerd:
Version: 1.2.0+unknown
GitCommit:| # docker run tonistiigi/hello
Hello world, I am linux/riscv64
There is also a Docker-in-Docker variant for testing with your local Docker CLI.
docker run -d --privileged -p 4000:4000 \
tonistiigi/docker:riscv-dind -H tcp://0.0.0.0:4000export DOCKER_HOST=tcp://0.0.0.0:4000
docker version
As mentioned before, if you are running the above examples with QEMU on your non-RISC-V machine, they will still use your x86 kernel. Using QEMU user emulation to run something as complex as Docker is quite extreme because Docker uses a lot of specific kernel features not well supported by the emulation layer. Some (non-RISC-V specific) QEMU patches were required to make it run containers in this mode. This was definitely the hardest part of all of the things in this post.
To test this setup with a full RISC-V kernel you can run the tonistiigi/docker:riscv-system-qemu image to emulate the full Fedora virtual machine with Docker tools already installed. Note that this image will be quite a lot slower and takes a while to boot.
docker run -it tonistiigi/docker:riscv-qemu-system# login: root/riscv
# systemctl status docker
# docker version
I do not have any actual RISC-V hardware to test this natively. If you have, you can let me know if it worked for you. A tarball of all of the Docker tools can be built with docker buildx build --platform=linux/riscv64 -o — --target=dist . > docker.riscv.tar .
Base images
The official images on Docker Hub all come with support for many architectures, but as this is all very new there isn’t RISC-V support in those images yet. I tried to create some as an example. Please note that these are just POC examples and not meant to have comparable quality to the official images.
Debian is one of the distros that seems to have somewhat good support for RISC-V already. tonistiigi/debian:riscv is the Debian sid image for RISC-V that I created with a simple debootstrap command and didn’t catch any issues during that process. Currently, RISC-V packages are not in the main Debian repositories so the base image sets up the package manager to use Debian ports as the source. You can also use tonistiigi/debian that is a multi-platform image, containing RISC-V support and using an official debian:sid image for the rest of the architectures.
As a second example, I tried to produce an alpine base image that is very popular in the Docker ecosystem. That proved to be a bit more tricky. Alpine Linux uses Musl libc for its components and RISC-V support isn’t upstream yet. From Musl roadmap you can see that support is planned in the next release with patches being in review. There is a great writeup about the current progress from the author, and he also has set up an Alpine repository with packages that currently have support. So I tried to use that repository to set up a base Docker image. Things didn’t go completely smooth and I had some trouble with the index on that repository. If you want to check it out, the image I built is at tonistiigi/alpine:riscv . Although local tools seem to work fine in that image, there seems to be some issue in the TLS library so you can’t install additional packages. If anyone knows what the issue/fix is, let me know. Work in progress.
Conclusions
RISC-V is still in the early stages, it usually requires a very specific set of tools and some patching to get complex things working. On the other hand, there seems to be a strong community effort to add RISC-V support to different tools so it is likely that you will already find PRs/patches for the tools you need. The patches I made to get the Docker tools working were relatively small compared to the work other people had already done with the individual lower level components. You can find the tracker for the current porting status of different components related to running containers at https://github.com/carlosedp/riscv-bringup .
The good thing is that if you are using Dockerfiles to build your software, having a very specific build flow isn’t really a problem because you just need to write it once in a Dockerfile, pinning your dependencies, and then everyone else can have known-to-work builds. Docker also helps with providing very easy ways to test your software through integration with emulation. While these are just the first steps you can be sure that once RISC-V becomes more popular and more hardware emerges, support will be added to the official images and the upstream components of Docker will build on RISC-V out-of-the-box. Docker will be a very easy way to develop for RISC-V or for any combination of architectures.
Links
Here is the list of images at Docker Hub that I built for this post:
tonistiigi/binfmt:riscv — registering RISC-V emulation support in the kernel. Dockerfile
tonistiigi/hello:riscv — RISC-V Hello world program
tonistiigi/hello — Multi-platform Hello world program, including support for wasi/wasm and RISC-V Dockerfile
tonistiigi/xx:riscv-toolchain — GNU toolchain for building RISC-V based on buildroot. Dockerfile
tonistiigi/xx:gcc-sid — GNU toolchain based on debian:sid that automatically picks up the correct cross-compilation target(including RISC-V). Dockerfile
tonistiigi/xx:golang-riscv — Golang image using the branch with GOARCH=riscv64 support. Dockerfile
tonistiigi/debian:riscv — debian:sid base image for RISC-V. Dockerfile
tonistiigi/debian — Multi-platform image for debian:sid that includes RISC-V support
tonistiigi/alpine:riscv — (Work in progress) Alpine Linux base image for RISC-V
tonistiigi/docker:riscv — Debian image with Docker tools installed. Dockerfile
tonistiigi/docker:riscv-dind — Docker-in-Docker image that runs a RISC-V Docker daemon
tonistiigi/docker:riscv-qemu-system — Full RISC-V emulated system with a RISC-V kernel and Docker installed
Patches to upstreams
https://github.com/vishvananda/netns/pull/34
https://github.com/docker/libnetwork/pull/2389
https://github.com/moby/moby/pull/39327
https://github.com/docker/cli/pull/1926
