How to attach USB devices to Kubernetes pods using Akri

A while back, I decided to migrate my 3D printer’s OctoPrint server — a web interface for controlling 3D printers — from running directly in Docker to inside Kubernetes. Previously, it was set up in Docker by mounting the required USB device with the --device /dev/ttyUSB* argument. This setup worked well, but I wanted to leverage Kubernetes' orchestration capabilities for better resource management and scalability

At first glance, the Kubernetes documentation provided little guidance on translating Docker’s --device arguments into a Kubernetes pod specification. However, after some research, I found a potential “hacky” solution. Let’s take a quick look at this initial approach and its quirks before moving on to Akri, which I believe is the best method yet to manage USB devices in Kubernetes.

Hostpath volume mounts of devices

apiVersion: v1
kind: Pod
metadata:
  name: example
spec:
  containers:
  - name: example-container
    securityContext:
      privileged: true
    volumeMounts:
    - name: hostpath-volume
      mountPath: /dev/ttyUSB1
  volumes:
  - name: hostpath-volume
    hostPath:
      path: /dev/ttyUSB1

One somewhat direct translation of the --device argument in Docker is to use hostPath volumes in the pod specification. Typically, hostPath mounts are used to access the file system on the node where the pod is running, but it can also be employed to access devices. For instance, executing ls /dev | grep ttyUSB inside the pod would list the connected USB devices. However, simply making the device visible is not enough; it will not be functional unless the pod is also running as privileged.

Running a pod in privileged mode gives it elevated permissions, allowing it to access host devices and other sensitive system areas. This increases the risk of security vulnerabilities, as privileged pods can potentially interfere with the host system. Additionally, this approach is inflexible because the /dev/ttyUSB* path may change at any time, requiring manual updates to the configuration.

Enter the world of Akri

To overcome these security and flexibility issues, I turned to Akri, an open-source project that that simplifies the management of edge devices in Kubernetes. Akri allows USB devices to be discovered and exposed to the Kubernetes API, enabling the scheduling of pods that require these resources in a more secure and flexible manner.

Akri finds USB devices by utilizing udev queries, meaning that it queries each node for USB devices using the udev subsystem. This method allows Akri to quickly detect changes in hardware configuration. For example, if a USB device is unplugged from one node and plugged into another, Kubernetes will be promptly informed, enabling it to reschedule pods as needed.

Installation

Now that Akri has been introduced, let’s dive into the installation process of it. The most straightforward approach is to use the provided Helm chart. I personally use Argo CD to manage installations, which allows me to keep configurations in a Git repository. Trust me, having everything in a Git repository is invaluable — when something eventually goes wrong, you can easily spin up a new cluster and start over, much like quick saves in a game!

Here’s how you can set up Akri with Argo CD. Create the following two files for the Argo CD application utilizing Akri as a subchart:

.
├── Chart.yaml
└── values.yaml

Chart.yaml

apiVersion: v2
name: akri-example
version: 0.1.0
dependencies:
- name: akri
  version: 0.12.20
  repository: https://project-akri.github.io/akri/

Values.yaml

akri:
  kubernetesDistro: k3s # Specify Kubernetes distro here: k3s|microk8s|k8s
  udev:
    discovery:
      enabled: true

The udev discovery enabled part is crucial because it deploys a DaemonSet on every Kubernetes node. A DaemonSet ensures that a copy of a pod runs on all or some nodes, which in this case, allows each node to look for attached USB devices and expose them to the Kubernetes scheduler.

That is it! Now Akri should be installed and ready for use.

Determine udev rule for USB device detection

Akri uses udev to query for devices. Therefore, one must specify the exact udev rules for detecting desired devices. This process can be a bit tricky, but is manageable. Let’s go through an example; follow along on your terminal.

lsusb

hampus@laptopserver:~$ lsusb
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 003: ID 0bda:58c6 Realtek Semiconductor Corp. 
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 002: ID 174c:55aa ASMedia Technology Inc. Name: ASM1051E SATA 6Gb/s bridge, ASM1053E SATA 6Gb/s bridge, ASM1153 SATA 3Gb/s bridge, ASM1153E SATA 6Gb/s bridge
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 032: ID 054c:0155 Sony Corp. Eyetoy Video Device
Bus 003 Device 031: ID 1a86:7523 QinHeng Electronics HL-340 USB-Serial adapter
Bus 003 Device 030: ID 1a40:0101 Terminus Technology Inc. Hub
Bus 003 Device 029: ID 1a40:0201 Terminus Technology Inc. FE 2.1 7-port Hub
Bus 003 Device 028: ID 10c4:ea60 Silicon Labs CP210x UART Bridge
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

The lsusb command lists all USB devices connected to my computer. By comparing the output before and after plugging in the 3D printer, I identified the device as:

Bus 003 Device 031: ID 1a86:7523 QinHeng Electronics HL-340 USB-Serial adapter

Next, we need to find the specific fields to base our udev query on using the udevadm info command. This command provides detailed information about the device, which helps in creating precise udev rules. First, take note of the bus and device ID from the lsusb output.

Bus 003 Device 031: ID 1a86:7523 QinHeng Electronics HL-340 USB-Serial adapter

Combine them into /dev/bus/usb/<bus>/<device>, which in this example is: /dev/bus/usb/003/031.

udevadm info --attribute-walk --path=$(udevadm info --query=path /dev/bus/usb/003/031)

hampus@laptopserver:~$ udevadm info --attribute-walk --path=$(udevadm info --query=path /dev/bus/usb/003/031)Udevadm info starts with the device specified by the devpath and then
walks up the chain of parent devices. It prints for every device
found, all possible attributes in the udev rules key format.
A rule to match, can be composed by the attributes of the device
and the attributes from one single parent device.

looking at device '/devices/pci0000:00/0000:00:14.0/usb3/3-3/3-3.4/3-3.4.1':
    KERNEL=="3-3.4.1"
    SUBSYSTEM=="usb"
    DRIVER=="usb"
    ATTR{tx_lanes}=="1"
    ATTR{bDeviceClass}=="ff"
    ATTR{bmAttributes}=="80"
    ATTR{bMaxPacketSize0}=="8"
    ATTR{configuration}==""
    ATTR{authorized}=="1"
    ATTR{rx_lanes}=="1"
    ATTR{bDeviceProtocol}=="00"
    ATTR{quirks}=="0x0"
    ATTR{speed}=="12"
    ATTR{avoid_reset_quirk}=="0"
    ATTR{maxchild}=="0"
    ATTR{urbnum}=="454"
    ATTR{product}=="USB2.0-Serial"
    ATTR{devpath}=="3.4.1"
    ATTR{idProduct}=="7523"
    ATTR{bMaxPower}=="96mA"
    ATTR{version}==" 1.10"
    ATTR{bcdDevice}=="0254"
    ATTR{ltm_capable}=="no"
    ATTR{removable}=="unknown"
    ATTR{busnum}=="3"
    ATTR{bDeviceSubClass}=="00"
    ATTR{bNumConfigurations}=="1"
    ATTR{bConfigurationValue}=="1"
    ATTR{bNumInterfaces}==" 1"
    ATTR{devnum}=="31"
    ATTR{idVendor}=="1a86"

... lots of more text for parent devices, ignore these and scroll to the top

You can select many attributes depending on how precise you want your query to be. I often choose ATTR{idVendor} and ATTR{idProduct} since these are usually sufficient.

Important note: Transforming ATTR into ATTRS can be convenient as it broadens the selection, ensuring all required devices are detected. Meaning that the above would be turned into ATTRS{idVendor} and ATTRS{idProduct}

Add udev rules to Akri configuration

Now, let’s add our first Akri Configuration using the udev rules we determined. This configuration file specifies the discovery rules for Akri to identify and manage the USB devices. If following along using helm/ArgoCD, this is the folder structure I have been using:

.
├── Chart.yaml
├── templates
│   └── 3dprinter.yaml
└── values.yaml

Merge the found udev properties into a comma seperated list: ATTRS{idVendor}=="1a86", ATTRS{idProduct}=="7523" and put it into the Configuration object below. Feel free to change the name of the object to your liking.

3dprinter.yaml

apiVersion: akri.sh/v0
kind: Configuration
metadata:
  name: akri-3dprinter
spec:
  capacity: 1
  discoveryHandler:
    discoveryDetails: |
      groupRecursive: true # Recommended unless using very exact udev rules
      udevRules:
      - ATTRS{idVendor}=="1a86", ATTRS{idProduct}=="7523"
    name: udev

After applying this change to your Kubernetes cluster, one new Kubernetes resource of the type Instance should be automatically created if everything is properly configured.

apiVersion: akri.sh/v0
kind: Instance
metadata:
  name: akri-3dprinter-583e0a
spec:
  brokerProperties:
    UDEV_DEVNODE_0: /dev/bus/usb/003/009
    UDEV_DEVNODE_1: /dev/ttyUSB1
    UDEV_DEVPATH: /devices/pci0000:00/0000:00:14.0/usb3/3-3/3-3.4/3-3.4.1
  configurationName: akri-3dprinter
  deviceUsage:
    akri-3dprinter-583e0a-0: C:0:laptopserver
  nodes:
  - laptopserver
  shared: false

By looking at the brokerProperties fields, notice how Akri detects a few different paths/devices that are connected to the udev query. All of these will be visible inside the pods using this resource. How cool is that?

Using Akri USB device inside Kubernetes pod

Alright, now the usb device is being detected by Akri. Now let’s use it inside a pod. Akri resources are exposed to pods through the device plugin framework, meaning that they can be access using the resources field of the pod specification. Take the name of your Configuration object and put it into the limits and request fields such as this:

resources:
    limits:
      akri.sh/akri-3dprinter: "1"
    requests:
      akri.sh/akri-3dprinter: "1"

Full pod example

apiVersion: v1
kind: Pod
metadata:
  name: example
spec:
  containers:
  - name: example-container
    resources:
      limits:
        akri.sh/akri-3dprinter: "1"
      requests:
        akri.sh/akri-3dprinter: "1"

If using a deployment, then I do recommend using the recreate stragegy since only one pod at a time can make use of a single akri device.

Conclusion

In conclusion, migrating OctoPrint to run inside Kubernetes using Akri for USB device management offers significant advantages in terms of security, flexibility, and maintainability. Akri’s dynamic discovery and management of USB devices align well with Kubernetes’ resource management principles, making it a superior solution compared to using privileged pods with hostPath volumes.

Since starting to use Akri, I have used it for many more things than Octoprint. For example, I am using it for zigbee, cameras, and even microphones. Leave a comment and let me know what you will be using Akri for.