Persistent Storage for Kubernetes with Glusterfs
Introduction:
This blog discusses the pros and cons of various Filesystem storages for K8.Persistent volume’s can be broadly categorized into three major types:
- Filesystem Storage (for any filesystem)
- Block Storage (for DBs)
- Object Storage (S3 etc..)
The table below lists all the persistent storages supported by K8 and a quick summary of pros and cons for possible choices that we were able to identify for a Production grade K8 persistent storage implementation.
We narrowed down our evaluation to CephFS, Flocker and Glusterfs. We chose Glusterfs, as the technology has been around for a long time, its simpler than Ceph (although rook.io claims it makes it simpler, and needs to be explored). Flocker we didnt find as much good documentation but will need to explore in details.
Gluster: Nuts and Bolts
GlusterFS is a connector based storage system, i.e. by itself gluster doesnt provide storage, but it connects to a durable storage and extrapolates storage to make it seamless for K8 pods.
The high level topology is as described in the diagram below where one EBS volume is mounted per EC2 instance that is running a kubernetes node. We have 3 EC2, EBS, K8 node setup below. We form a glusterfs cluster using the 3 EBS nodes. We can then define and carveout several persistent volumes (pv) PV1, PV2 … PV5 out of the 3 mounted EBS volumes, making it homogenous and seamless for K8 pods to claim.
K8 schedules pods as per its algorithm on any K8 node and the pods can claim a persistent volume via a persistent volume claim. Persistent volume claim (pvc) is nothing but a label that identifies a connection between a POD and a persitent volume. Per the diagram below we have POD C claim PV1 while POD A claim PV4.
Cut the fluff: get hands dirty
- Login to the bastion host that can connect to K8 cluster
- install git and clone the repo
yum install gitgit clone https://github.com/kubernetes-incubator/external-storage && cd external-storage/gluster/glusterfs
3. find the K8 nodes and label them
kubectl get nodeskubectl label nodes ip-xxx–xx–xx–xxx.us-east-2.compute.internal storagenode=glusterfs
4. Deploy glusterfs and get pod IPs
Deploy: kubectl create -f deploy/glusterfs-daemonset.yamlCheck pod status: kubectl get pods -l glusterfs-node=pod — watchGet Pod IP: kubectl get pods -o wide | grep glusterfs | grep -v provisioner
5. peer probe glusterfs nodes
kubectl exec -it glusterfs-k9xxx -- gluster peer probe 172.xx.xx.xxxkubectl exec -it glusterfs-ppyyy -- gluster peer probe 172.yy.yy.yyy
7. Create EBS volumes and mount it to K8 nodes at /gfs
Get InstanceID of K8 nodes:aws ec2 describe-instances — query ‘Reservations[*].Instances[*].InstanceId’ --filters “Name=tag:Name,Values=nodes.kube-dev.ue2.yourcompany.com”
Create EBS volumes:aws ec2 create-volume --region us-east-2 --availability-zone us-east-2a --size 20Attach EBS volumes: aws ec2 attach-volume --volume-id vol-xxxxxxxxxxxxx --instance-id i-xxxxxxxxxxx --device /dev/xvdfLogin to the K8 node: mkfs -t ext4 /dev/xvdf
mkdir /gfs
mount /dev/xvdf /gfs
make entry into /etc/fstab, so it mounts on reboot/dev/xvdf /gfs ext4 defaults,nofail
6. Install glusterfs library on the K8 nodes
apt-get install glusterfs-client7. create storageclass.yml and deploy via kubectl (update brick root paths to IP of K8 nodes)
storageclass.yamlkind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: glusterfs-simple
provisioner: gluster.org/glusterfs-simple
parameters:
forceCreate: “true”
brickrootPaths: “172.xx.xx.xxx:/gfs/,172.yy.yy.yyy:/gfs/”Apply storageclass.yamlkubectl apply -f storageclass.ymlApply role based access controlcd external-storage/gluster/glusterfs/deploy/
kubectl create -f rbac.yamlDeploy Gluster provisionercd external-storage/gluster/glusterfs/deploy
kubectl apply -f create -f deployment.yamlCreate persistent volume claimcd external-storage/gluster/glusterfs/deploy
kubectl apply -f pvc.yaml
Validation:
Check if the PVC. is bound to PV
kubectl get pvc,pvList persistent volume list:
kubectl get pods
kubectl exec -it glusterfs-k9xxx -- gluster volume list