Auto-labeling GKE Nodes for XFS support

Allan Lei
Allan Lei
Sep 16, 2018 · 2 min read

To use XFS with Persistent Volumes, the host node needs to have the command xfs_mkfile available so disks can be created and formatted. The problem comes when needing to do this on GKE where there are 2 OSs available, ubuntu which has xfsprogs installed by default and cos which does not. Also, before ubuntu was available, container-vm was the used for XFS, but required a separate install.

The solution I took was to add a XFS support node label at node pool creation time using OS image ubuntu. While this sort of works, there are a couple problems.

  • The need to remember to add a XFS node label when creating a new node pool
  • The assumption that ubuntuhas xfsprogs installed
  • The assumption that cos does not have xfsprogs installed. (It doesn’t right now but you never know)

Detecting XFS host support

Since provisioning a PersistentVolume is done on the host before the Pod is running, this complicates things as the support would need to be for the host, but the detection done in a Pod.

For this, we will use nsenter. nsenter allows running a process in a different namespace, in our case, the host. For us to properly use nsenter, we will need to set hostPID:true and priviledged: true, allowing us to break out of the pod into the host.

The command command -v xfs_mkfile gets run on the host which detects if the command xfs_mkfile is available.

Labeling the node for support

The usage of the /tmp volume mount is to pass the results of detection into the labeling container. This is to drop the priviledged: true as soon as possible.

By using the Downward API, we can pass in the node name to the pod. This container will then label the node it is on with the label fs.type/xfs=true/false indicating support. With this, you can schedule pods with nodeAffinity.

Note: kubectl label node requires extra permissions which won’t be covered here. As a shortcut, setting spec.serviceAccountName: node-controller in the kube-system namespace provides these permissions.

Putting it together

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store