Using EFS storage in Kubernetes

At Survata, we manage large volumes of survey response data. To manage the ingestion and analysis of this data, we have a data ingestion pipeline. As part of a larger set of improvements intended to improve stability and extensibility, and reduce deployment costs, we’re currently rebuilding this ingestion pipeline as a microservice, and deploying that microservice using Kubernetes.

This data ingestion pipeline needs access to a fast, extensible, sharable source of disk storage. AWS EFS is a good match for this requirement.

However, configuring our Kubernetes deployment to access EFS proved a lot harder than originally expected. It turns out this isn’t because it’s difficult — but because Amazon’s documentation buries the lede on a much simpler alternative.

Using EFS in Kubernetes

If you do a Google search for “mount EFS drive on EKS”, you’ll probably find this AWS Knowledge Center article, or the Github repository that contains the efs-provisioner utility mentioned in that article. The sample configuration provided with that repository includes:

1. A Storage Class for EFS storage
2. A ConfigurationMap that defines EFS-related values that will occur in multiple places
3. An EFS Provisioner Service that mounts the EFS storage
4. A ServiceAccount for the EFS Provisioner
5. Role-based access control for the EFS Provisioner Service Account
6. A PersistentVolume that defines the existence of storage
7. A PersistentVolumeClaim that claims the storage for use
8. A Pod that actually uses the EFS storage

We were eventually able to get a version of this configuration working — but not reliably. We initially had extensive problems configuring permissions (some of which are logged as known problems); once we resolved those problems, we saw intermittent problems with the EFS drive not being available after updating the services that were using the EFS storage.

After spending a lot of effort trying to diagnose why the EFS connection was unstable, we took a step back. Why does this seemingly simple task —mounting an EFS store in a deployment— require such a complex configuration? Is there a simpler way?

A simpler approach

It turns out there is — or, at least, there is for our purposes. You only need three configuration items to mount an EFS store— one of which you have probably already created.

First, you need define a configuration for a PersistentVolume:

persistent-volume.yaml

The mount options in this configuration come from the AWS recommendations for mounting EFS file systems. The capacity is a placeholder; it’s a value required by Kubernetes, but ignored at runtime because an EFS store will expand as needed (this is one of the major reasons we choose EFS over EBS for this use case). The access mode of ReadWriteMany is chosen because we want to allow multiple services to access this drive (which is the other reason we chose EFS; EBS only allows a single mount). If your service only requires read access, you could specify ReadOnlyMany here.

Next, add a second configuration for a PersistentVolumeClaim:

persistent-volume-claim.yaml

Again, the 1Mi size allocation is a required placeholder that is ignored by EFS.

Lastly, we define the Deployment that is going to use the PersistentVolumeClaim:

deployment.yaml

This will need to be modified to suit the service you’re deploying; you may already have a deployment configuration if you’re adding EFS to an existing deployed service. The only part of this configuration that is relevant to this discussion is the volume declaration connecting to the persistentVolumeClaim, and the volumeMount connecting to container the volume.

And that’s it. Apply these new Kubernetes configurations with kubernetes apply, and your deployed service will be able to use /data(or whatever mount point you nominate in your volumeMount definition) as EFS-mounted storage. The only complication beyond this is ensuring that your Kubernetes nodes can access the EFS—but as long as the EFS store shares a network interface and security AWS group with the cluster nodes, you should have no difficulties.

So — why does AWS recommend an extremely complicated configuration when a really simple one will suffice?

How Kubernetes handles storage

To understand why, you need to step back and look at what Kubernetes is actually trying to do.

Kubernetes isn’t a tool to manage the deployment of a single web app, or a single microservice — it’s a tool for managing a suite of applications on a shared cluster of hardware.

When you deploy a service that declares it needs 2GB of persistent storage, Kubernetes then needs to determine how it can satisfy that request. This is why there is a distinction between a PersistentVolume (an underlying source of storage), and a PersistentVolumeClaim (an allocation of storage):

• A cluster defines the existence of a block of storage (the PersistentVolume);
• An app deployed on that cluster declares it has a need to allocate a piece of that storage (the PersistentVolumeClaim); and
• A deployment within that app defines that it will use a particular block of allocated storage.

When, in the future, you decommission your app, the claim over the piece of storage is released, which allows some other app deployed in the cluster to lay claim to that piece of storage.

This model of storage comes from a world where the persistent volume is a physical disk connected to an actual computer — and in that context, it makes perfect sense. Storage is a limited resource, and usage of that resource must be carefully allocated.

The “provisioner”-based configuration published by AWS treats EFS as an underlying source of storage that Kubernetes must manage. Applications can request claims be made on that storage. When those applications are decommissioned, those claims can be released for others to use.

However, what does it mean to “claim” a block of storage when the underlying storage will expand as necessary? There’s no limit to how much can be allocated, so what does “claiming” storage mean? The Kubernetes storage model doesn’t really map to the capabilities of EFS.

The simplified configuration avoids this question entirely. We don’t need to allocate subblocks of the EFS storage, and share our single EFS store across multiple services. We know the only user of our EFS store will be our ingestion service, and so we can attach it directly. Kubernetes supports NFS as a way of connecting to persistent stores. EFS exposes an NFS interface. We define a single claim over the “entire” EFS/NFS mount, and attach that claim to our deployment. Simple!

Simpler isn’t necessarily better

The configuration suggested by AWS isn’t entirely misguided, though. Kubernetes is an infrastructure-as-code tool. It is designed to manage the full infrastructure needs of a deployed application. A cluster defines resources that are available, and applications make claims on those resources. The maintainer of a cluster should be able to add or remove resources to that cluster as needed. Applications should be able to deploy themselves without being aware of those changes.

However, the “simplified” configuration undermines this. It is not possible to deploy the “simplified” configuration without also configuring an EFS store — something that, in the Kubernetes purist sense, shouldn’t be a possible.

But does this matter? Benjamin Brewster once noted that in theory there is no difference between theory and practice, (…) in practice there is. In Survata’s case, we have a small infrastructure team, and a well defined set of services, so the overhead imposed by needing to managing creation of the EFS store outside Kubernetes isn’t a major concern, and can be managed by other tools (in our case, Terraform). Whether this is also true for your own usage is something you’ll need to determine on your own.