Wasm and Kubernetes: A New Era of Cloud-Native Application Deployment

13 min readNov 6, 2023

Wasm on K8s: everything you need to know 🐳

☸️ Introduction:

WebAssembly (Wasm) and Kubernetes (K8s) are two of the most transformative technologies in cloud computing today. Wasm is a portable, lightweight bytecode format that enables developers to run code written in any language on the web. Kubernetes is a container orchestration platform that automates many of the tasks involved in deploying and managing containerized applications.

Together, Wasm and K8s have the potential to revolutionize cloud-native application development. Wasm can be used to create portable, scalable, and secure applications that can run on any platform, including Kubernetes clusters. K8s can be used to manage and deploy Wasm applications at scale, making it easy to build and maintain complex cloud-native applications.

In this blog post, we will explore the connection between Wasm and K8s and discuss their potential to usher in a new era of cloud-native application development. We will also discuss some of the challenges and opportunities that lie ahead for this emerging technology stack.

WebAssembly basics:

WebAssembly is a revolutionary new technology that is changing the way web applications are developed. It is a low-level binary format that can compile a variety of programming languages, including C, C++, and Rust. This allows developers to create web applications that are much faster and more efficient than those written in JavaScript alone.

WebAssembly has three main advantages: speed, security, and support.

Speed: WebAssembly programs run at near-native speeds, which means that they can be just as fast as desktop applications.
Security: WebAssembly programs run in a memory-safe sandbox, which makes them more secure than JavaScript programs.
Support: WebAssembly is supported by all major browsers, so developers can be confident that their applications will work for all users.

WebAssembly is already being used to create a wide variety of web applications, including games, video editing software, and even machine learning applications. As the technology continues to mature, it is likely to have an even greater impact on the web development landscape.

🐋 Docker Vs WASM:

Docker has program and its dependencies packaged in a single image and then run as container. Docker container runs as full file system(utlities, binaries etc) appears to be complete operating system for application. Additionally you need to create image according to right system architecture e.g. intel, armd, arm etc. If someone has rasberry pi OS running docker image, then you need to create image for C/C++ application based on Linux image and compile for ARM process system architecture. If not done, then container won’t run as expected.

On other side, WASM module/binaries is precompiled C/C++ application and can be run swiftly on WASM runtime. It doesn’t rely/coupled with Host OS or System Architecture as it doesn’t contain precompiled file system, low level OS primitives similar to docker. Every directory, system resource is attached to WASM module during runtime facilitated by WASI.

Docker vs WASM Performance:

WASM provides near Native Speed Performance, quicker Startup time, and High Security, allow C/C++/Rust/Go code to run in the browser and even outside the browser. Docker on other hand provides Runtime isolation and better portability.

The key is that Wasm binaries don’t rely on host OS or processor architectures like Docker containers. Instead, all the resources the Wasm module needs (such as environment variables and system resources) are provisioned to the Wasm module by the runtime through the WASI standard. This means Wasm modules are not coupled to the OS or underlying computer. It’s an ideal mechanism for highly portable web-based application development.

WebAssembly on Kubernetes:

Kubernetes needs two things to be able to run WebAssembly workloads:

1. Worker nodes bootstrapped with a WebAssembly runtime.

2. RuntimeClass objects mapping to nodes with a WebAssembly runtime

We’ll explain all of the following in more detail:

1. Worker node configuration with contianerd and runwasi

2. Bootstrapping Kubernetes workers with Wasm runtimes

3. Using labels to target workloads

4. RuntimeClasses and Wasm

5. Wasm apps in Kubernetes Pods

containerd and runwasi:

Most Kubernetes clusters use containerd as the high-level runtime. It runs on every node and manages container lifecycle events such as create, start, stop, and delete. However, containerd only manages these events, it actually uses a low-level container runtime called runc to perform the actual work. A shim process sits between containerd and the low-level runtime and performs important tasks such as abstracting the low-level runtime.

The architecture is shown below:

runwasi is a containerd project that let’s you swap-out container runtimes for WebAssembly runtimes. It operates as a shim layer between containerd and low-level Wasm runtimes and enables WebAssembly workloads to seamlessly run on Kubernetes clusters. The architecture is shown below.

Everything from containerd and below is opaque to Kubernetes — Kubernetes schedules work tasks to nodes and doesn’t care if it’s a traditional OCI container or a WebAssembly workload.

Bootstrapping Kubernetes workers with Wasm runtimes:

For a Kubernetes worker to execute WebAssembly workloads it needs bootstrapping with a Wasm runtime. This is a two-step process:

1. Install the Wasm runtime
2. Register the Wasm runtime with containerd

In most cases, your Kubernetes distro will provide a CLI or UI that automates these steps. However, we’ll explain what’s happening behind the scenes.

Wasm runtimes are binary executables that should be installed on worker nodes in a path that’s visible to containerd. They should also be named according to the containerd binary runtime naming convention. The following list shows the wasmtime and spin runtime binaries named appropriately and installed into the /bin directory:

– wasmtime: /bin/containerd-shim-wasmtime-v1
– spin: /bin/containerd-shim-spin-v1

Once installed, runtimes need registering with containerd. This is done by adding them to the containerd config file which is usually located at /etc/containerd/config.toml.

The following extract shows how to register the wasmtime and spin runtimes in the containerd config.toml file.

[plugins.cri.containerd.runtimes.wasmtime]
  runtime_type = "io.containerd.wasmtime.v1"
[plugins.cri.containerd.runtimes.spin]
  runtime_type = "io.containerd.spin.v1"

Once the Wasm runtimes are installed and registered, the final node configuration step is to label the nodes.

Using labels to target workloads:

If all of your cluster nodes have the same runtimes you do not need to label them. However, if you have sub-sets of nodes with Wasm runtimes, you need to label them so that RuntimeClass objects can target them.

The following diagram shows a cluster with 6 nodes. Two have runc, 4 have wasmtime, and 2 have spin.

See how the labels make it obvious which nodes have which runtimes.

Be sure to use meaningful labels and avoid reserved namespaces such as kubernetes.io and k8s.io.

The following command applies the wasmtime=yes label to a node called wrkr3. RuntimeClasses can use this label to send Wasm workloads to the node.

$ kubectl label nodes wrkr3 wasmtime-enabled=yes

The output of the next command shows the label was correctly applied.

$ kubectl get nodes --show-labels

NAME     STATUS    ROLES    AGE     VERSION        LABELS
wrkr1    Ready     <none>   5d      v1.25.1
wrkr2    Ready     <none>   5d      v1.25.1
wrkr3    Ready     <none>   2m      v1.25.1        wasmtime-enabled=yes

With Wasm runtimes installed, registered with containerd, and labels applied, a node is ready to execute Wasm tasks.

The next step is to create a RuntimeClass that sends Wasm workloads to the node(s).

RuntimeClasses and Wasm:

RuntimeClasses allow Kubernetes to schedule Pods to specific nodes and target specific runtimes.

They have three important properties:

– metadata.name
– scheduling.nodeSelector
– handler

The name is how you tell other objects, such as Pods, to use it. The nodeSelector tells Kubernetes which nodes to schedule work to. The handler tells containerd which runtime to use.

The following RuntimeClass is called wasm1. The scheduling.nodeSelector property sends work to nodes with the wasmtime=yes label, and the handler property ensures the wasmtime runtime will execute the work.

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: "wasm1"
scheduling:
  nodeSelector:
    wasmtime-enabled: "yes"
handler: "wasmtime"

The following diagram shows three worker nodes. One of them is running the wasmtime and spin runtimes and is labelled appropriately. The Pod and RuntimeClass on the left target work against the wasmtime runtime. The Pod and RuntimeClass on the right target work against the spin runtime.

With worker nodes configured and RuntimeClasses in place, the last thing to do is bind Pods to the correct RuntimeClass.

Wasm apps in Kubernetes Pods

The following YAML snippet targets the Pod at the wasm1 RuntimeClass. This will ensure the Pod gets assigned to a node and runtime specified in the wasm1 RuntimeClass.

In the real world, the Pod template will be embedded inside a higher order object such as a Deployment.

apiVersion: v1
kind: Pod
metadata:
  name: wasm-test
spec:
  runtimeClassName: wasm1    <<<<==== Use this RuntimeClass
  container:
  - name: ctr-wasm
    image: <OCI image with Wasm module>
    ...

This Pod can be posted to the Kubernetes API server where it will use the wasm1 RuntimeClass to ensure it executes on the correct node with the correct runtime (handler).

Notice how the Pod template defines a container even though it’s deploying a Wasm app. This is because Kubernetes Pods were designed with containers in mind. For Wasm to work on Kubernetes, Wasm apps have to be packaged inside of OCI container images.

Deploying a Wasm app:

You’ll need Docker Desktop and K3d to follow along, and the remainder of the article assumes you have these installed.

You’ll complete the following steps:

Build a K3d cluster
Verify the runtime configuration
Configure node labels
Create a RuntimeClass
Deploy an app
Test the app
Scale the app
Inspect the containerd processes

Build a K3d Kubernetes cluster:

Run the following command to create a 3-node K3d Kubernetes cluster. You’ll need Docker Desktop installed and running, but you do not need the Docker Desktop Kubernetes cluster running.

$ k3d cluster create wasm-cluster \
--image ghcr.io/deislabs/containerd-wasm-shims/examples/k3d:v0.5.1 \
-p "8081:80@loadbalancer" --agents 2

Verify the cluster is up and running.

$ kubectl get nodes

NAME                        STATUS   ROLES                  AGE   VERSION
k3d-wasm-cluster-server-0   Ready    control-plane,master   2m    v1.24.6+k3s1
k3d-wasm-cluster-agent-0    Ready    <none>                 2m    v1.24.6+k3s1
k3d-wasm-cluster-agent-1    Ready    <none>                 2m    v1.24.6+k3s1

The command to build the cluster used an image with the spin and slight Wasm runtimes pre-installed. These are vital to running WebAssembly apps on Kubernetes.

The next step will verify the runtime configuration.

Verify the runtime configuration:

Use docker exec to log on to the k3d-wasm-cluster-agent-0 node.

$ docker exec -it k3d-wasm-cluster-agent-0 ash

Run all of the following commands from inside the exec session.

Check the /bin directory for containerd shims named according to the containerd shim naming convention.

$ ls /bin | grep containerd-

containerd-shim-runc-v2
containerd-shim-slight-v1
containerd-shim-spin-v1

You can see the runc, slight, and spin shims. runc is the default low-level runtime for running containers on Kubernetes and is present on all worker nodes running containerd. spin and slight are Wasm runtimes for running WebAssembly apps on Kubernetes.

With the shims installed correctly, run a ps command to verify containerd is running.

# ps
PID     USER     COMMAND
<Snip>
   58   0        containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml...

The output is trimmed, but it shows the containerd process is running. The -c flag is used to pass containerd a custom location for the config.toml file.

List the contents of this config.toml to see the registered runtimes.

$ cat /var/lib/rancher/k3s/agent/etc/containerd/config.toml

<Snip>
[plugins.cri.containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"[plugins.cri.containerd.runtimes.spin]
  runtime_type = "io.containerd.spin.v1"[plugins.cri.containerd.runtimes.slight]
  runtime_type = "io.containerd.slight.v1"

runc, spin, and slight are all installed and registered with containerd.

You can repeat these steps for the other two nodes and will get similar results as all three are running containerd and are configured with the spin and slight Wasm runtimes.

The next step is to label your nodes.

🏷 Configure node labels:

We’ll add a custom label to a single worker node and use it in a future step to force Wasm apps onto just that node.

Run the following command to add the spin=yes label to the k3d-wasm-cluster-agent-0 worker node.

$ kubectl label nodes k3d-wasm-cluster-agent-0 spin=yes

Verify the operation. Your output will be longer, but only the k3d-wasm-cluster-agent-0 should be displayed.

$ kubectl get nodes --show-labels | grep spin
NAME                        STATUS   ROLES     ...  LABELS
k3d-wasm-cluster-agent-0    Ready    <none>    ...  beta.kubernetes..., spin=yes

At this point, k3d-wasm-cluster-agent-0 is the only node with the spin=yes label. In the next step, you’ll create a RuntimeClass that targets this node.

🏃 Create a RuntimeClass:

The following YAML defines a RuntimeClass called spin-test. It selects on nodes with the spin=yes label and specifies the spin runtime as the handler.

Copy and paste the whole block into your terminal to deploy it.

kubectl apply -f - <<EOF
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: spin-test
handler: spin
scheduling:
  nodeSelector:
    spin: "yes"
EOF

The following command verifies the RuntimeClass was created and is available.

$ kubectl get runtimeclass
NAME         HANDLER   AGE
spin-test    spin      10s

At this point, you have a 3-node Kubernetes cluster and all three nodes have the spin runtime installed. You also have a RuntimeClass that can be used to schedule tasks against the k3d-wasm-cluster-agent-0 node. This means you’re ready to run WebAssembly apps on Kubernetes!

In the next step, you’ll deploy a Kubernetes app.

🏗 Deploy an app:

You can find the app code here: https://github.com/deislabs/containerd-wasm-shims/tree/main/images/spin

The following YAML snippet is from the app you’re about to deploy. The only bit we’re interested in is the spec.template.spec.runtimeClassName = spin-test field. This tells Kubernetes to use the spin-test RuntimeClass you created in the previous step. This will schedule the app to the correct node and ensure it executes with the appropriate handler (runtime).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: wasm-spin
spec:
  replicas: 1
  ...
  template:
    ...
    spec:
      runtimeClassName: wasmtime-spin     <<==== Targets the RuntimeClass
      containers:

Deploy it with the following command.

kubectl apply -f https://raw.githubusercontent.com/seifrajhi/wasm-k8s/main/wasp-app.yaml

Verify the app was deployed. It might take a few seconds for it to enter the ready state and it will only work if you followed all previous steps.

$ kubectl get deploy
NAME          READY   UP-TO-DATE   AVAILABLE   AGE
wasm-spin     1/1     1            1           14s

Verify it’s running on the correct node.

$ kubectl get pods -o wide
NAME                         READY   STATUS    AGE   NODE
wasm-spin-74cff79dcb-npwwj   1/1     Running   86s   k3d-wasm-cluster-agent-0

It’s running on the k3d-wasm-cluster-agent-0 worker node that has the label and handler specified in the RuntimeClass.

Test the app is working by pointing your browser to http://localhost:8081/spin/hello or by running the following curl command.

$ curl -v http://127.0.0.1:8081/spin/hello
<Snip>
Hello world from Spin!

Congratulations, the application is successfully deployed to the worker node specified in the RuntimeClass.

In the next step, you’ll scale the app to prove that all replicas get scheduled to the same node.

Scale the app:

Increase the number of replicas from 1 to 3.

$ kubectl scale --replicas 3 deploy/wasm-spin
deployment.apps/wasm-spin scaled

Verify all three Pods are running on the k3d-wasm-cluster-agent-0 node.

$ kubectl get pods -o wide
NAME                         READY   STATUS     AGE     NODE
wasm-spin-74cff79dcb-npwwj   1/1     Running    3m32s   k3d-wasm-cluster-agent-0
wasm-spin-74cff79dcb-vsz7t   1/1     Running    7s      k3d-wasm-cluster-agent-0
wasm-spin-74cff79dcb-q4vxr   1/1     Running    7s      k3d-wasm-cluster-agent-0

The RuntimeClass is doing its job of ensuring the Wasm workloads running on the correct node.

Next up, you’ll ensure they’re executing with the spin runtime and inspect the containerd processes.

Inspect the containerd processes:

Exec onto the k3d-wasm-cluster-agent-0 node.

$ docker exec -it k3d-wasm-cluster-agent-0 ash

Run the following commands from inside the exec session.

List running spin processes.

# ps | grep spin
PID    USER  COMMAND
<Snip>
1764  0     {containerd-shim}.../bin/containerd-shim-spin-v1 -namespace k8s.io -id ...
2015  0     {containerd-shim}.../bin/containerd-shim-spin-v1 -namespace k8s.io -id ...
2017  0     {containerd-shim}.../bin/containerd-shim-spin-v1 -namespace k8s.io -id ...

The output is trimmed, but you can see three containerd-shim-spin-v1 processes. This is one shim process for each of the three replicas.

The long hex ID attached to each of the three shim processes is the ID of the associated container task. This is because containerd runs each Wasm task inside its own container.

Run the following command to list containers on the host. Notice how some of the container task IDs match with the hex IDs associated with the spin shim processes from the previous command. The PID also matches the PID of the spin shim processes.

$ ctr task ls
TASK                                                                PID     STATUS
3f083847f6818c3f76ff0e9927b3a81f84f4bf1415a32e09f2a37ed2a528aed1    2015    RUNNING
f8166727d7d10220e55aa82d6185a0c7b9b7e66a4db77cc5ca4973f1c8909f85    2017    RUNNING
78c8b0b17213d895f4758288500dc4e1e88d7aa7181fe6b9d69268dffafbd95b    1764    RUNNING
</Snip>

The output is trimmed to only show the Wasm containers.

You can see more detailed info with the info command.

$ ctr containers info 3f083847f6...a37ed2a528aed1
{
    <Snip>
    "Runtime": {
        "Name": "io.containerd.spin.v1",
        <Snip>
        "annotations": {
            "io.kubernetes.cri.container-name": "spin-hello",
            "io.kubernetes.cri.container-type": "container",
            "io.kubernetes.cri.image-name": "ghcr.io/deislabs/containerd-wasm-shims/examples/spin-rust-hello:v0.5.1",
            "io.kubernetes.cri.sandbox-id": "3f083847f6818c3f76ff0e9927b3a81f84f4bf1415a32e09f2a37ed2a528aed1",
            "io.kubernetes.cri.sandbox-name": "wasm-spin-5bd4bd7b9-kqgjt",
            "io.kubernetes.cri.sandbox-namespace": "default"
            <Snip>

If you examine the output of the previous command, you’ll see typical container constructs such as namespaces and cgroups. The WebAssembly app is executing inside the normal WebAssembly sandbox, which, in turn, is executing inside a minimal container.

🍉Summary:

WebAssembly is a new technology that is changing the way cloud-native applications are developed and deployed. It is now possible to run WebAssembly applications on Kubernetes, which means that developers can take advantage of the performance, security, and portability of WebAssembly while also benefiting from the scalability and manageability of Kubernetes.

To run WebAssembly applications on Kubernetes, you need to bootstrap Kubernetes worker nodes with a WebAssembly runtime and use RuntimeClasses to map WebAssembly applications to the appropriate nodes. This is a relatively new process, but it is becoming increasingly popular as WebAssembly matures and becomes more widely adopted.