We had a mongo replicaset cluster setup in Google Container Engine that was using the default
ext filesystem when creating
PersistentVolumes. The recommendation by mongo is that the
xfsshould be used instead.
- The default image on GKE is
gci, based on ChromiumOS, which does not have a straightforward method for installing packages for
xfsprogs, but is not installed by default
- Custom images are not supported on GKE.
- There’s no standard/recommended method for running node startup scripts.
Initially, we were headed in the direction of editing the Instance Templates the GKE was using and adding in
apt-get install xfsprogs, but the problem was that it was not portable enough. If I added a new node-pool or upgraded an existing one, the changes to the Instance Template would not get carried over.
Our only option was
container-vm for OS image so that we could just
apt-get install xfsprogs on the node, preferably at init time, but how?
The solution turned out to be to use Kubernetes
daemonsets run on every node, including newly created nodes. The recommendation even is briefly described on the Kubernetes docs and an example on kubernetes/contrib.
Putting it together
- Create a node-pool using the
- Build the startup script Docker image
- Create a startup-script
daemonsetbased on Kubernetes' startup-script example.
- Add the daemonset to kubernetes
- Create an
empty diskon GCE.
- Create a
- Apply the
PersistentVolumeand Kubernetes will format the disk to
Originally published at blog.helveticode.com on January 1, 2017.