Mounting XFS on GKE

Update Sept 6 2017: As of Aug 8 2017, GKE has released the ubuntu image to replace container-vm. This image has xfsprogs pre-installed. Use this instead.

We had a mongo replicaset cluster setup in Google Container Engine that was using the default ext filesystem when creating PersistentVolumes. The recommendation by mongo is that the xfsshould be used instead.

1. The default image on GKE is gci, based on ChromiumOS, which does not have a straightforward method for installing packages for xfs.
2. container-vm does have xfsprogs, but is not installed by default
3. Custom images are not supported on GKE.
4. There’s no standard/recommended method for running node startup scripts.

Initially, we were headed in the direction of editing the Instance Templates the GKE was using and adding in apt-get install xfsprogs, but the problem was that it was not portable enough. If I added a new node-pool or upgraded an existing one, the changes to the Instance Template would not get carried over.

Our only option was container-vm for OS image so that we could just apt-get install xfsprogs on the node, preferably at init time, but how?

The solution turned out to be to use Kubernetes daemonsets. daemonsets run on every node, including newly created nodes. The recommendation even is briefly described on the Kubernetes docs and an example on kubernetes/contrib.

Putting it together

1. Create a node-pool using thecontainer-vmimage.
2. Build the startup script Docker image
3. Create a startup-script daemonset based on Kubernetes' startup-script example.
4. Add the daemonset to kubernetes
5. Create an empty disk on GCE.
6. Create a PersistentVolumewith fsType:xfs.
7. Apply the PersistentVolume and Kubernetes will format the disk to xfs on mount.

daemonset.yaml

Originally published at blog.helveticode.com on January 1, 2017.