Increase GlusterFS volume size in Kubernetes

Published in

WeScale

6 min readMay 2, 2019

One of the common issues when using a persistent volume solution in Kubernetes, like GlusterFS, is to start running out of storage capacity in your cluster, and you have to find a smooth way to increase volume size without losing your data. In this short article I will show you two ways of doing it.

Before going deeper, let me introduce you the context and some terminologies that will be used throughout this article.

From official websites:

“Gluster is a scalable, distributed file system that aggregates disk storage resources from multiple servers into a single global namespace”.

“Heketi provides a RESTful management interface which can be used to manage the lifecycle of GlusterFS volumes”.

Topology.json, “heketi uses this configuration file to define cluster nodes topology for GlusterFS. It contains the nodes’ IP addresses and the storage devices to use in each node”.

It looks like:

{
 “clusters”: [
  {
   “nodes”: [
    {
     “node”: {
       “hostnames”: {
         “manage”: 
           [“node4”],
             “storage”: [“10.0.XX.XX”]
       },
         “zone”: 1
        },
          “devices”: [
          {
            “name”: “/dev/xvdb”,
            “destroydata”: false
          }
        ]
      }
…

I use GlusterFS and Heketi inside a Kubernetes cluster, running on six EC2 machines; three masters/etcd and three nodes (workers).

I use the Kubespray project to setup the cluster on an AWS environment. It has the advantage of using Ansible playbooks to setup the necessary bricks for Kubernetes, whether on AWS or elsewhere.

In the same project, there are also additional Ansible playbooks that I have used to setup GlusterFS and Heketi storage topology. The latter is running on k8s nodes and uses EBS volumes attached to EC2 instances. You need to customize this topology to fit your needs.

The picture below shows roughly how the setup is made.

I started with a volume of 30GiB mounted on /dev/xvdb device. The k8s cluster was almost empty, only some monitoring Daemonsets were running and other light applications with 8GiB of PersistentVolumes for each; everything was good until I had touched the limits. At this point I figured out that PersistentVolumeClaims were stuck in pending state forever. Here is an example of stucking pvc for Minio S3 storage.

minio persistentvolumeclaim/minio-pvc Pending 1d

Looking in logs, I found a clear message saying that there was not enough storage capacity to run the pvc.

Warning ProvisioningFailed … Failed to provision volume with StorageClass “gluster”: failed to create volume: heketi block volume creation failed: [heketi] failed to create volume: Failed to allocate new block volume: No space

Let’s get our hands dirty

There are two ways I have used to increase volume capacity:

Extend the volume attached to VM.
Add another storage device (for example: /dev/xvdc) to heketi topology.

1. Extend the volume:

This method assumes that you have already a volume mounted in heketi cluster. It can be used on cloud providers which allow you to expand volumes on-the-fly, like AWS, GCP etc.

The first step is to expand your volume size (from web GUI, command line, terraform …)
SSH into the said instance.
Expand the physical volume, in my case I added another 30GiB

pvresize /dev/xvdb
 Physical volume “/dev/xvdb” changed
 1 physical volume(s) resized / 0 physical volume(s) not resized

Before:

vgdisplay
 — — Volume group — -
 VG Name vg_a1751200878b2bb0a64d89a15e3bc801
 VG Size 29.87 GiB

After:

vgdisplay
 — — Volume group — -
 VG Name vg_a1751200878b2bb0a64d89a15e3bc801
 VG Size 59.87 GiB

4. Now tell Heketi to take into account the new volume size with Resync device. The heketi command lines are available on this website.

5. Connect to a master node or any other machine that have a remote and configured kubectl and do the following steps:

# Get heketi pod namekubectl get pods -n kube-system | grep heketi
# Get heketi secret for Admin access in order to use API as Adminkubectl get secret heketi-config-secret -n kube-system -o jsonpath — template ‘{.data.heketi\.json}’ | base64 -d
# List heketi cluster Namekubectl -n kube-system exec heketi-d55cbd78c-7tj8c -n kube-system — heketi-cli — user admin — secret XXXXXX cluster listClusterId: Id:65ddc905b8a32dc49db721a004a08ead
# Get detailed informations about the clusterkubectl -n kube-system exec heketi-d55cbd78c-7tj8c -n kube-system — heketi-cli — user admin — secret XXXXX cluster info 65ddc905b8a32dc49db721a004a08eadCluster id: 65ddc905b8a32dc49db721a004a08eadNodes:310744c2908338d7d229268a4445951b9ac5b33d03759ba7d9b161b879a4ea39e0fd9cbad89eac2989cff2b426a59288Volumes:29e2a3fd578bb98a7d44e9a538cc9e747c955fe8dff62136046fc5c794abb6a5b22ad96e06a4bff70c16e9fe61a4ed50Block: trueFile: true
# Get informations about one node, you can apply the command again with another node idkubectl -n kube-system exec heketi-d55cbd78c-7tj8c -n kube-system — heketi-cli — user admin — secret XXXXX node info 9ac5b33d03759ba7d9b161b879a4ea39Node Id: 310744c2908338d7d229268a4445951bState: onlineCluster Id: 65ddc905b8a32dc49db721a004a08eadZone: 1Management Hostname: node5Storage Hostname: 10.0.XX.XXDevices:Id:a1751200878b2bb0a64d89a15e3bc801 Name:/dev/xvdb State:online Size (GiB):34 Used (GiB):22 Free (GiB):12 Bricks:3
# Resync devicekubectl -n kube-system exec heketi-d55cbd78c-7tj8c -n kube-system — heketi-cli — user admin — secret XXXXX device resync a1751200878b2bb0a64d89a15e3bc801Device updated

6. Once the resync is done, you can check again the node to see what has changed.

# Get updated informations about node5kubectl -n kube-system exec heketi-d55cbd78c-7tj8c -n kube-system — heketi-cli — user admin — secret XXXXX node info 9ac5b33d03759ba7d9b161b879a4ea39Node Id: 310744c2908338d7d229268a4445951bState: onlineCluster Id: 65ddc905b8a32dc49db721a004a08eadZone: 1Management Hostname: node5Storage Hostname: 10.0.XX.XXDevices:Id:Id:a1751200878b2bb0a64d89a15e3bc801 Name:/dev/xvdb State:online Size (GiB):64 Used (GiB):22 Free (GiB):42 Bricks:3

7. At last, do not forget to run the same resync command for all glusterFS nodes (node4, node6 in my case).

And now you have just increased volume capacity in your cluster.

2. Add another storage device to heketi topology

In this case, I’m going to add a new storage device to heketi topology. This method is most likely used when you cannot increase volume sizes on-the-fly, and you have to add another volume in order to expand storage capacity in your cluster. These steps are inspired from IBM’s website.

Please note that the volume must be unmounted and empty, no partition or filesystem must be created on it, so it can be used by heketi.

Create and attach the volume to instance, for example: /dev/xvdc
You can make sure to clear up your volume by using:

wipefs — all — force /dev/xvdc

3. Connect to a master node or any machine that have a remote and configured kubectl.

4. This step from the previous procedure is optional, I am using it just to get more visibility about what is happening, and also a small reminder for whomever that uses the second procedure.

Get heketi pod name
Get heketi secret for admin access
List cluster Name
Get more informations about the cluster
Get informations about one node, you can apply the command again with another node id

5. Add this volume to heketi topology:

# To add a new device to heketi, you must use volume ID got by cluster info command or heketi-cli volume listkubectl -n kube-system exec heketi-d55cbd78c-7tj8c — heketi-cli — user admin — secret XXXXX device add — name=/dev/xvdc — node=e0fd9cbad89eac2989cff2b426a59288Device added successfully

Before:

kubectl -n kube-system exec heketi-d55cbd78c-7tj8c — heketi-cli — user admin — secret XXXXX node info 9ac5b33d03759ba7d9b161b879a4ea39Node Id: 310744c2908338d7d229268a4445951bState: onlineCluster Id: 65ddc905b8a32dc49db721a004a08eadZone: 1Management Hostname: node5Storage Hostname: 10.0.25.109Devices:Id:a1751200878b2bb0a64d89a15e3bc801 Name:/dev/xvdb State:online Size (GiB):34 Used (GiB):22 Free (GiB):12 Bricks:3

After:

kubectl -n kube-system exec heketi-d55cbd78c-7tj8c — heketi-cli — user admin — secret XXXXX node info 9ac5b33d03759ba7d9b161b879a4ea39Node Id: 310744c2908338d7d229268a4445951bState: onlineCluster Id: 65ddc905b8a32dc49db721a004a08eadZone: 1Management Hostname: node5Storage Hostname: 10.0.25.109Devices:Id:a1751200878b2bb0a64d89a15e3bc801 Name:/dev/xvdbState:online Size (GiB):34 Used (GiB):22 Free (GiB):12 Bricks:3Id:fcb295c112eb0a1bbfa9e6fca8a3f30a Name:/dev/xvdc State:online Size (GiB):59 Used (GiB):0 Free (GiB):59 Bricks:5

6. Do not forget to apply the previous step on all glusterFS nodes (node4 and node6 in my case).

Conclusion

GlusterFs is a serious project supported by RedHat, IBM and others. It is a good start to play with dynamic volumes in k8s. For now the project is still young and I didn’t test it in a production context.

I have used both methods shown in this article and they work fine. Now enjoy your storage :)

Special thanks to my friends from The WeTribu who have patiently reviewed this article.

Increase GlusterFS volume size in Kubernetes

Let’s get our hands dirty

1. Extend the volume:

2. Add another storage device to heketi topology

Conclusion

Written by Aymen Lamara