Running Stateful Applications in Kubernetes
A guide on how to run stateful applications in Kubernetes.
What are stateful apps?
Any app that does not make sense, if it doesn’t have access to previous data that was given to it can be defined as a stateful app. Let’s take an image processing app, each time you request the app to process the image it need not have to relate that to the previous request. It considers each request as new and processes the image for you. These are called stateless apps, which consider every request as an independent and new request.
Let’s take a database like MySQL as an example, when you query data from MySQL it has to have access to the data inserted from the previous requests or it doesn’t make sense. This ultimately translates to if the MySQL service goes down and comes up again it needs to have access to the disk where it used to store data, unlike the image processing service. Even if the image processing service goes down and comes up it need not have access to the same disk since it talks to MySQL to store the images. Most of the database, message queues, and caching services are stateful because they do not make sense without data on the disk.
Stateless apps are easier to scale and maintain than stateful apps. Why is that?
When you have multiple replicas of the image processing service running on different nodes behind a load balancer. The Load Balancer sends the traffic to any of the nodes since they are technically the same.
But let’s say you have multiple replicas of the mysql db and send requests to any of them, each of them will have a different response to it, because your write may happen on a node and read may happen on a different one. This creates an inconsistency. This is solved with a mechanism called sharding which is storing subsets of data on a particular node and keeping track of which data resides on which node, so the read can be redirected to that node. Now each replica of a stateful app is not identical, of course, they are the same code but have access to different subsets of data (can be on a disk), which makes each replica unique.
Kubernetes and stateful apps
Running stateful apps in virtual machines is comparatively easier than in containers.
If the VM goes down it can come up again and have access to the same disk it had before. But containers are designed to be ephemeral, meaning when the container gets deleted, the data it was storing on the disk also vanish since they are in the same temporary namespace. Still, you can mount a host directory path to the container and persist data but the container having access to host volumes directly is not recommendable for security reasons.
In the world of Kubernetes, which is used to manage containers, the concept of persistence volumes is used to persist data produced by the process in the containers using external disks.
What are Persistence volumes and how do they work?
Persistent volumes in general as the name suggests, it’s an external volume attached to the node and the data which gets stored will be there forever unless we intentionally delete it. Since containers are ephemeral and we do not want to store data on the host disk where the OS and the system files are stored, we need some external disk to be used by the containers.to store and persist data, that’s where PVs come into picture.
How to create PVs for containers in Kubernetes?
This is basically done by creating an Object called Persistent Volume Claims, as the name suggests, this is a claim for a persistent volume. This is what it looks like:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: my-pvc
spec:
storageClassName: aws-ebs
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10GiThe PVC is a requirement that you define for your PV and if any PV that’s available matches the requirement will be considered for mount or else it will be dynamically created.
Now that we have the volume created, we need to mount it to the pod. That’s done by volume mounts to the pod. For example,
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
spec:
replicas: 1
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
spec:
containers:
- name: my-container
image: your-image
volumeMounts:
- name: data-volume
mountPath: /data
volumes:
- name: data-volume
persistentVolumeClaim:
claimName: my-pvcHere we are creating a volume out of a PVC and mounting that volume in the “Volume mount” section of the manifest in a particular path where your app will store and fetch data. Now this is how you make your stateful apps have persistent storage in Kubernetes.
Troubleshooting Persistent volumes
Well, not everything is a fail proof solution, so does PVs. Common error scenarios include FailedAttachVolume and FailedMount.
FailedAttachVolume and FailedMount
FailedAttachVolume error occurs when a PV is unable to be detached from one another and attached to the other node when the pods restart and get scheduled on a different node from the previous one. Usually, this may occur due to a corrupted volume, or too many volumes attached to the node. To resolve this you may need to manually detach the volume from the node via console or API.
Sometimes the EBS volume in AWS may be in a different AZ than the pod, in that case, you may need to cordon off the current node and delete the pod so that it can schedule on a different node where the volume can be attached or simply add a node selector to get the pod scheduled on a specific node.
According to the documentation (https://v1-23.docs.kubernetes.io/docs/setup/best-practices/multiple-zones/)
When persistent volumes are created, the PersistentVolumeLabel admission controller automatically adds zone labels to any PersistentVolumes that are linked to a specific zone. The scheduler then ensures, through its NoVolumeZoneConflict predicate, that pods which claim a given PersistentVolume are only placed into the same zone as that volume.
Kubernetes tries to schedule the pod on the same AZ as the PV. But when it’s not able to (when there are not enough resources). It may schedule it in a different AZ.
FailedMount can happen when the volume is not able to mount on the given path in the node.
This may happen due to unavailable mount paths on the node.
Hope it was a useful read on persistent volumes on k8s :) Happy Clustering
