Multiarch Support In LitmusChaos

In this blog, I will be discussing how to build a multi-arch image for any desired docker architecture to run litmus chore components. For those who are new to litmus and wanted to explore more about chaos engineering, I would recommend to check out litmus blog. Now coming back to the topic we will be using docker buildx for building the images in the following sections.

Pre-requisites:

• Docker(version 19.03)

Understanding Multiarch Builds

Docker introduced the principle of multi-arch builds to support the “Build once, deploy anywhere” concept which helps to use ARM targets and reduce your bills such as AWS A1 and Raspberry Pis instances. But how do we produce them? and How do they work?
A multi-arch Docker image supports a different architecture behind the same imagetag. Let’s compare the manifest of the multi-arch image with the image having a single arch.

Enable docker CLI if not already enabled using the following steps:

1. export DOCKER_CLI_EXPERIMENTAL=enabled
2. Add "experimental": "enabled", to ~/.docker/config.json (default location) at the beginning of the file and not at the end.

Image with single arch manifest

Multi arch image manifest

We can observe that the manifest for the multi-arch image is a simple list of manifests prepared for each platform while an image with a single arch will have only one platform in its manifest.

Build Multi-arch image for Litmus Components

For building a multi-arch image for litmus core components just follow the following simple steps:

• Setup buildx
• Build go binary for different architectures.
• Prepare Dockerfile for Multiarch.
• Build and push the multi-arch image.

Build using Buildx instance

Introduction:

Docker Buildx is a CLI plugin that extends the docker command with the full support of the features provided by Moby BuildKit builder toolkit. It has new features like creating scoped builder instances and building against multiple nodes concurrently.

Setup:

The Docker Buildx is included in Docker 19.03 to install it run the following commands:

Confirm the installation:

Output:

for any issue in installation refer installation guide.

Now the buildx is setup in your system. Buildx allows you to create new instances of isolated builders. You can use this to get a scoped environment for your CI builds that does not change the state of the shared daemon, or for isolating builds for different projects. You can create a new instance for a set of remote nodes, forming a build farm, and quickly switch between them.

Setup Builder instance

1. Using the QEMU emulation support in the kernel

2. Command to create new builder instances.

Output:

3. To show the builder instance created

Output:

4. Inspect the builder

Here the platforms will show the platform available to build images.

5. If the Status is not running run the following command

This will activate the Status of the builder instance.
Output:

6. Use the multibuilder instance

Build Go binary

We know that all the litmus core components (chaos experiment, operator, and runner) are written in Golang and we are building the go binaries and using them in the image.

Note:
You can build the go binaries and copy them in Dockerfile or we can build the binary in the Dockerfile itself and use them.

In this example, I’ll be following the first way that is building binary and copying them in the Dockerfile.
The main key point here is to build different binaries with Go ENVs set to desired values like GOARCH=amd64 &GOOS=linux for which we have to build the image.

The following script will help to do that:

Command

In the above script, we are creating the binary of github.com/litmuschaos/chaos-operator/cmd/manage with GOARCH & GOOS set in the platforms. This will create binaries as follow:

Prepare Dockerfile for Multiarch

Prepare a Dockerfile that should work for all the different architectures. Make sure that:
1. The base image should be multi-arch: Use the base image which is having multi-arch support.

Which supports both ARM64 and AMD64 arch.

2. Use TARGETARCH variable for using binary in Dockerfile: All binaries used should support multi-arch builds and are built accordingly. The Docker predefines a set of ARG variables according to the --platform we pass in buildx. This is only available when using the BuildKit backend. Here the TARGETARCH is the architecture component of TARGETPLATFORM. Check out more platform env here.

Example

This will copy the binaries created in the previous step while building the image.

Build and push the multi-arch image

BuildKit is designed to work well for building for multiple platforms and not only for the architecture and operating system that the user invoking the build happens to run. When you invoke a build, you can set the --platform flag to specify the target platform for the build output, (for example, linux/amd64, linux/arm64, darwin/amd64).

A sample command to build and push the multiarch image with the Dockerfile created in the previous stage.

The above command will build a multiarch image and push it to Dockerhub. The flags used are:

• --file: Path of the Dockerfile (Default is ‘PATH/Dockerfile’)
• --progress: Use to control verbosity. Set type of progress output (auto, plain, tty). Use plain to show container output.
• --platform: Set target platform for build.
• --tag: Set tag of the image in the ‘name:tag’ format.
• --push: Used to push the image to the registry.

for more flags checkout buildx man page.

Multiarch image in Dockerhub:

We have successfully built and pushed the multiarch images on Dockerhub. The images will look like:

Troubleshooting

1. standard_init_linux.go:211: exec user process caused "exec format error":

There could be multiple reasons for this. Try to check out the base image you're using is supporting all the platforms for which you're building the images.

Make sure while building and copying the binary in Dockerfile that the arch and os should be same. Like you're building the binary with linux/amd64 and you're trying to run it on linux/arm64.

2. rpc error: code = Unknown desc = failed to load LLB: runtime execution on platform linux/arm/v7 not supported:

This means the platform you're looking for is not available. You can verify the available platforms using docker buildx inspect <builder-name>.Now you can run the following commands to resolve the above error.

Conclusion

In this blog, we learn how to build and push an image that can be used at different docker architecture with the same image tag on it. This can be done using a powerful docker buildx feature which helps to build image with a single command. We can create images that can run on any desired platform using buildx and can be used to reduce your bills using ARM targets Raspberry Pis instance. So have ever tried building multiarch images? How useful it could be? Please leave your thoughts in the comment box.

🚀 Special Thanks 🙌

Thanks to Michael Fornaro for helping in suggesting, reviewing & adopting the multiarch support for Litmus Chore Components.

Are you an SRE or a Kubernetes enthusiast? Does Chaos Engineering excite you?
Join Our Community On Slack For Detailed Discussion, Feedback & Regular Updates On Chaos Engineering For Kubernetes: https://kubernetes.slack.com/messages/CNXNB0ZTN
(#litmus channel on the Kubernetes workspace)
Check out the Litmus Chaos GitHub repo and do share your feedback: https://github.com/litmuschaos/litmus
Submit a pull request if you identify any necessary changes.

litmuschaos/litmus