Mounting a GCP bucket as NFS in kubernetes

Creating a fileshare of unlimited size as NFS mounted on a bucket inside a kubernetes cluster? Disregarding if this is a good idea or not, here is a little description of the problem we faced and how we solved it.

Background

Why did I want this as a NFS server in the first place? Why not simply mount it in the pod needing it using gcsfuse?

For my current client, the Kubernetes cluster was a managed airflow instance (cloud composer), and I had already setup a NFS server that was running smoothly inside this cluster (following this great guide). This NFS share was used by processes using kubernetes operator that spawns pods inside the composer cluster.

The bad thing about running a NFS server inside the Kubernetes cluster is that it is “hard” to get the volume outside the cluster. Both for developers and other processes.

One of the pain points that started to pop up, was something simple as listing a file for a developer. One had to kubectl into a pod and do an ls. It would actually be quite handy to just see the files in a bucket representation. And what about disk growing?

I could have used gcsfuse directly in the pod, right?

Not really — since the pod was created using the Airflow Kubernetes operator, I didn’t have full control over the creation environment, and I found no good way of running gcsfuse in a generic way for different images. Short of adding it, and a bunch of scripts to the dockerfile. But my spider sense kept telling me that someone would forget something in the future, and this would not work out well. I wanted something simple like volumes and mounts that could apply to all jobs without changing the docker image.

Could a bucket be mounted instead? This would certainly be nice. unlimited disk storage? Easily available outside the cluster to various jobs? Or just use the native bucket API.

There were actually no articles describing how to do this (maybe for good reasons? — a bucket is not really a filesystem, and doesn’t have locks like we are used to. In addition there is ingress and egress..)

Solution

My initial thought was to modify the NFS server and just use gcsfuse to mount a bucket instead of grabbing a volume claim.

Create a secret to hold service account

First you obviously create the service account and download it and gives it the access it needs. Test service account access locally first and ensure that you can use gsutil on the bucket, for example.

kubectl create secret generic sa-THE-SERVICE-ACCOUNT --from-file=key.json=sa.json

Create replicationcontroller

yaml/nfs-bucket-server-rc.yaml
apiVersion: v1
kind: ReplicationController
metadata:
  name: nfs-bucket-server
spec:
  replicas: 1
  selector:
    role: nfs-bucket-server
  template:
    metadata:
      labels:
        role: nfs-bucket-server
    spec:
      volumes:
        - name: sa-THE-SERVICE-ACCOUNT
          secret:
            secretName: sa-THE-SERVICE-ACCOUNT
      containers:
      - name: nfs-bucket-server
        image: anderselton/nfs-bucket-server:latest
        env:
        - name: BUCKET
          value: "CHANGE-ME"
        ports:
          - name: nfs
            containerPort: 2049
          - name: mountd
            containerPort: 20048
          - name: rpcbind
            containerPort: 111
        volumeMounts:
          - name: sa-THE-SERVICE-ACCOUNT
            mountPath: "/accounts"
            readOnly: true
        securityContext:
          privileged: true

Run

kubectl apply -f yaml/nfs-bucket-server-rc.yaml

Create service to expose server to other pods

yaml/nfs-bucket-server-service.yaml
kind: Service
apiVersion: v1
metadata:
  name: nfs-bucket-server
spec:
  ports:
    - name: nfs
      port: 2049
    - name: mountd
      port: 20048
    - name: rpcbind
      port: 111
  selector:
    role: nfs-bucket-server

Run

kubectl apply -f yaml/nfs-bucket-server-service.yaml

Create NFS volume and claim (for other pods to use)

yaml/nfs-bucket-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-bucket
spec:
  capacity:
    storage: 1Mi
  accessModes:
    - ReadWriteMany
  nfs:
    server: nfs-bucket-server.default.svc.cluster.local
    path: "/"

yaml/nfs-bucket-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nfs-bucket
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: ""
  resources:
    requests:
      storage: 1Mi

Run

kubectl apply -f yaml/nfs-bucket-pv.yaml
kubectl apply -f yaml/nfs-bucket-pvc.yaml

Since there was no image existing that did this particular thing, I had to write a custom dockerfile that contained both the NFS server, gcsfuse and a startup script to glue it all together. And of course a bunch of yaml config files.

Full GitHub repo, contains everything needed to do this in your own cluster:

https://github.com/ael-computas/gcp-kubernetes-bucket-nfs-server

What you could (and maybe should) do instead:

• Just use the bucket native API
• If you need to do local development, maybe abstract on your side, so you read from local FS when developing and bucket when in cloud/production.

It could turn out that this was a really bad idea, and that we should just use native API instead and do the rewrites we need to do.