Deploying Stateful applications in Kubernetes

Jeganathan Swaminathan ( jegan@tektutor.org )

In Kubernetes, Deployment is used to create stateless applications while StatefulSet is used to create stateful applications.

Stateless application Example in real world

When you call a mobile customer care, we never know or need to know who on the other end picks up the call. In case, the call got dropped in between, when you call the customer care again, most likely a different call centre executive will pick up your call, our problems gets resolved one or other way. This is an example of stateless application, as the conversation we had with the first executive is not known to the second executive, hence we had to repeat the details of our problem all over again to the second executive, though it is an inconvenience but it still works out.

Kubernetes Deployment suits perfectly here. Let’s assume you deployed your stateless application as a Deployment with 10 Pod replicas running in multiple worker nodes. If one of those Pods running in a particular worker node got terminated due to some issue, the ReplicaSet Controller takes care of replacing the bad Pod with a new healthy Pod either on the same node or on a different node. The ReplicaSet controller ensures 10 Pod instances of your application are running at any point in time but doesn’t matter in which worker nodes they are running. This is possible because, the faulty stateless application Pod x hasn’t stored any data on worker node x, hence can be easily replaced by another Pod y running on a different worker node y.

Stateful application Example in real world

Let’s say you want to order some books via Amazon, you added some books to your shopping cart. After you added the book(s) to your shopping cart, let’s say you plumber arrived to fix an issue at your home, let’s say you just left the web browser as such, it is possible the session will timeout or you closed the web browser in a hurry. Well, you can anytime re-login to amazon portal and resume from where you left last time, as the amazon portal will ensure all items added to the Shopping cart are persisted in database.

Technically, if Amazon is using a ShoppingCart Microservice that runs in a Pod, the Pod that stored the book into the Shopping Cart might be different than the Pod that served you when you resumed your book purchase on a different day. The interesting thing to understand here is, the StatefulSet Controller managed to bind the exact same Persistent Volume to two Shopping Cart Pods associated to that customer at two different point of time. This is a practical example of Stateful application.

The StatefulSet Controller works differently than the ReplicaSet Controller. The StatefulSet Controller is aware of Persistent Volume and its Persistent Volume Claims. When a stateful application Pod becomes faulty, the StatefulSet Controller will ensure the same Persistent Volume that was used by the faulty Pod is mounted to the new Pod that replaces the faulty Pod. Pods get replaced in the process of scaling down and up as well.

The StatefulSet controller

• adds Pod references to PersistentVolumeClaims (PVC)
• makes the Pod sticky to a PVC
• retains the Pod name even when new Pod replaces a faulty Pod
• it has options to gracefully shutdown a Pod to ensure data is saved before the Pod is terminated

Let’s create a secret as shown below

mysql-secret.yml

apiVersion: v1
kind: Secret
metadata:
  name: mysql-pass
  namespace: default
type: Opaque
data:
  password: QWRtaW5AMTIzCg==

Let’s create the mysql password as a secret in Kubernetes cluster

kubectl apply -f mysql-secret.yml

Let’s create the manifest file for Persistent Volume

mysql-pv.yml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: mysql-pv
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/data/mysql"

Notice the storage class is manual, this Persistent volume can satisfy any PersistentVolumeclaim whose storage class is manual and its size is less than or equal to 1Gi and the access mode is ReadWriteOnce.

Let’s create the Persistent Volume in Kubernetes cluster

kubectl apply -f mysql-pv.yml

In a cloud environment, the cloud provider supports a default storage class. PersistentVolumeClaim that doesn’t have a storageClassName mentioned will be served by the Persistent Volumes that are marked as default. Typically, AWS uses EBS as a default storage volume, however this is configurable.

In case of bare-metal K8s setup, we need to create a Persistent Volume with storage class that matches the persistent volume claim without which the PVC will be never bound with any PV.

Let’s now create a Persistent volume claim manifest file.

mysql-pvc.yml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pv-claim
  labels:
    app: mysql
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

Let’s create the PersistentVolumeClaim in our Kubernetes cluster

kubectl apply -f mysql-pvc.yml

Let’s create a service manifest file for the mysql StatefulSet

mysql-service.yml

apiVersion: v1
kind: Service
metadata:
  name: mysql
  labels:
    app: mysql
spec:
  ports:
    - port: 3306
  selector:
    app: mysql
    tier: backend
  clusterIP: None

Let’s create the service in our Kubernetes cluster

kubectl apply -f mysql-service.yml

Let’s create the mysql StatefulSet manifest file.

mysql-statefulset.yml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
  labels:
    app: mysql
spec:
  selector:
    matchLabels:
      app: mysql
      tier: backend
  serviceName: mysql
  replicas: 2
  minReadySeconds: 10
  template:
    metadata:
      labels:
        app: mysql
        tier: backend
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - image: mysql:5.6
        name: mysql
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-pass
              key: password
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-pv-claim

Let’s create the StatefulSet into our Kubernetes cluster

kubectl apply -f mysql-statefulset.yml

Let’s now list out all the resources we created so far

kubectl get statefulset,service,po,pv,pvc

The expected output is

[jegan@master.tektutor.org ~]$ kubectl get statefulset,service,po,pv,pvc
NAME                     READY   AGE
statefulset.apps/mysql   2/2     148m
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP    3h36m
service/mysql        ClusterIP   None         <none>        3306/TCP   148m
NAME          READY   STATUS    RESTARTS       AGE
pod/mysql-0   1/1     Running   0              141m
pod/mysql-1   1/1     Running   1 (139m ago)   140m
NAME                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                    STORAGECLASS   REASON   AGE
persistentvolume/mysql-pv   1Gi        RWO            Retain           Bound    default/mysql-pv-claim   manual                  148m
NAME                                   STATUS   VOLUME     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/mysql-pv-claim   Bound    mysql-pv   1Gi        RWO            manual         148m

Let’s observe on which nodes these mysql-0 and mysql-1 pods are running.

kubectl get po -o wide

The expected out is

[jegan@master.tektutor.org ~]$ kubectl get po -o wide
NAME      READY   STATUS    RESTARTS       AGE    IP                NODE                   NOMINATED NODE   READINESS GATES
mysql-0   1/1     Running   0              146m   192.168.145.212   worker1.tektutor.org   <none>           <none>
mysql-1   1/1     Running   1 (144m ago)   145m   192.168.72.129    worker2.tektutor.org   <none>           <none>

Let’s now forcefully delete mysql pod and observe how the StatefulSet Controller repairs the damage. In my case, mysql-0 pod was running in worker1 node before deleting it.

kubectl delete pod/mysql-0

Let’s see where the new Pod is scheduled after we delete the old mysql-0 Pod.

kubectl get po -o wide

The expected output is

[jegan@master.tektutor.org ~]$ kubectl get po -o wide
NAME      READY   STATUS    RESTARTS       AGE    IP                NODE                   NOMINATED NODE   READINESS GATES
mysql-0   1/1     Running   0              146m   192.168.145.212   worker1.tektutor.org   <none>           <none>
mysql-1   1/1     Running   1 (144m ago)   145m   192.168.72.129    worker2.tektutor.org   <none>           <none>
[jegan@master.tektutor.org ~]$ kubectl delete pod/mysql-0
pod "mysql-0" deleted
[jegan@master ~]$ kubectl get po -o wide
NAME      READY   STATUS              RESTARTS       AGE    IP               NODE                   NOMINATED NODE   READINESS GATES
mysql-0   0/1     ContainerCreating   0              2s     <none>           worker1.tektutor.org   <none>           <none>
mysql-1   1/1     Running             1 (148m ago)   149m   192.168.72.129   worker2.tektutor.org   <none>           <none>
[jegan@master.tektutor.org ~]$ kubectl get po -o wide
NAME      READY   STATUS    RESTARTS       AGE    IP                NODE                   NOMINATED NODE   READINESS GATES
mysql-0   1/1     Running   0              10s    192.168.145.214   worker1.tektutor.org   <none>           <none>
mysql-1   1/1     Running   1 (148m ago)   149m   192.168.72.129    worker2.tektutor.org   <none>           <none>

As you can see, the new Pod name hasn’t changed. StatefulSet created a new Pod with the same Pod name mysql-0 and it is scheduled on the same node the old mysql-0 Pod was running.

You can follow the author to get notified when he publishes new articles.

If you found this post helpful, please click the clap 👏 button below a few times to show your support for the author 👇

Thank you !

My other articles

Setting up a 3 node Kubernetes on your local machine

https://medium.com/@jegan_50867/kubernetes-3-node-cluster-setup-50943378be41

Using Metal LB in a bare metal /onprem K8s setup

https://medium.com/@jegan_50867/using-metal-lb-on-a-bare-metal-onprem-kubernetes-setup-6d036af1d20c

Using Nginx Ingress Controller in bare-metal Kubernetes setup

https://medium.com/@jegan_50867/using-nginx-ingress-controller-in-kubernetes-bare-metal-setup-890eb4e7772