Configuring HA Kubernetes cluster on bare metal servers with GlusterFS & MetalLB . 2/3

Hello and welcome back, this is a second part of article about configuring Kubernetes cluster on bare metal servers. Previously we configured a HA Kubernetes cluster with external etcd, multi-masters schema and load balance. Well, so far now it’s time to setup additional environment and utilities for making your cluster more useful and closest to production ready.

In this part of article we’ll focusing on setting up the internal load balancer for our cluster services - it’ll be a MetalLB. Also we’ll install and configure a distributed file storage between our workers nodes. We’ll use a GlusterFS for the persistent volumes that will be available inside Kubernetes.

After all be done our cluster schema will looks like this:

1. Configuring MetalLB as internal load balancer.

Few words about MetalLB directly from the document page:

MetalLB is a load-balancer implementation for bare metal Kubernetes clusters, using standard routing protocols.
Kubernetes does not offer an implementation of network load-balancers (Services of type LoadBalancer) for bare metal clusters. The implementations of Network LB that Kubernetes does ship with are all glue code that calls out to various IaaS platforms (GCP, AWS, Azure…). If you’re not running on a supported IaaS platform (GCP, AWS, Azure…), LoadBalancers will remain in the “pending” state indefinitely when created.
Bare metal cluster operators are left with two lesser tools to bring user traffic into their clusters, “NodePort” and “externalIPs” services. Both of these options have significant downsides for production use, which makes bare metal clusters second class citizens in the Kubernetes ecosystem.
MetalLB aims to redress this imbalance by offering a Network LB implementation that integrates with standard network equipment, so that external services on bare metal clusters also “just work” as much as possible.

So, using this tool we can run our services under load balancer on our Kubernetes cluster, and many thanks to the MetalLB team for it. The configuration process are really easy and clear.

Previously in example we have choose a subnet for our cluster needs. Now we need to take some part of this subnet for our future load balancer.

Login to the machine that have configured kubectl utility and run:

control# kubectl apply -f

This will deploy a MetalLB to your cluster, under the metallb-system namespace. Make sure that all components of MetalLB working OK then:

control# kubectl get pod --namespace=metallb-system 
controller-7cc9c87cfb-ctg7p 1/1 Running 0 5d3h
speaker-82qb5 1/1 Running 0 5d3h
speaker-h5jw7 1/1 Running 0 5d3h
speaker-r2fcg 1/1 Running 0 5d3h

Now we need to configure MetalLB using configmap, we’ll use a Layer 2 configuration for this example. For more configurations variants please refer to the MetalLB documentation.

Create a metallb-config.yaml file in any directory with chosen IP range from our cluster subnet:

control# vi metallb-config.yaml
apiVersion: v1 
kind: ConfigMap
namespace: metallb-system
name: config
config: |
- name: default
protocol: layer2

And apply this configuration:

control# kubectl apply -f metallb-config.yaml

You can check and change the this configmap later if you need:

control# kubectl describe configmaps -n metallb-system
control# kubectl edit configmap config -n metallb-system

From now we got our own local load balancer configured, let’s check how it works using some example of Nginx service.

control# vi nginx-deployment.yaml
apiVersion: apps/v1 
kind: Deployment
name: nginx-deployment
app: nginx
replicas: 3
app: nginx
- name: nginx
image: nginx:latest
- containerPort: 80

vi nginx-service.yaml
apiVersion: v1 
kind: Service
name: nginx
type: LoadBalancer
app: nginx
- port: 80
name: http

Next create this test Nginx deployment and service:

control# kubectl apply -f nginx-deployment.yaml
control# kubectl apply -f nginx-service.yaml

And let’s check the result:

control# kubectl get po 
NAME                               READY STATUS    RESTARTS   AGE 
nginx-deployment-6574bd76c-fxgxr 1/1 Running 0 19s
nginx-deployment-6574bd76c-rp857 1/1 Running 0 19s
nginx-deployment-6574bd76c-wgt9n 1/1 Running 0 19s
control# kubectl get svc
nginx LoadBalancer 80:31604/TCP 107s

There was created a 3 Nginx pods as we specified in our deployment previously. Nginx service will send traffic to all of them with round robin balance schema. And also you can see the external IP that we got from our MetalLB LB.

Now try curl to the IP address and you will see the Nginx index.html page. Don’t forget to remove this test Nginx deployment and service also.

control# kubectl delete svc nginx 
service "nginx" deleted
control# kubectl delete deployment nginx-deployment 
deployment.extensions "nginx-deployment" deleted

OK, that all about MetalLB, now we can move on and configure the GlusterFS for the Kubernetes volumes.

2. Configuring GlusterFS with Heketi on workers nodes.

In fact using Kubernetes cluster are impossible without using volumes inside it. As you know Pod’s are ephemeral and it means that they can be created and deleted at any time. And all data they have inside will be lost. So in real cluster you will need some distributed storage, to have availability of sharing configs and data between nodes and applications inside it.

In Kubernetes you can use volumes, there is different variants of them so you can choose some for your need. In this example I wanna show a creating of a GlusterFS storage that we can use for any kind of internal applications as a persistent volumes. Previously I used for this a “system” installation of GlusterFS on all of the Kubernetes worker nodes and then I just created a hostPath type of volumes inside GlusterFS directories.

But now we have a new nice Heketi tool.

Few words from Heketi documentation:

RESTful based volume management framework for GlusterFS
Heketi provides a RESTful management interface which can be used to manage the life cycle of GlusterFS volumes. With Heketi, cloud services like OpenStack Manila, Kubernetes, and OpenShift can dynamically provision GlusterFS volumes with any of the supported durability types. Heketi will automatically determine the location for bricks across the cluster, making sure to place bricks and its replicas across different failure domains. Heketi also supports any number of GlusterFS clusters, allowing cloud services to provide network file storage without being limited to a single GlusterFS cluster.

That sounds good, and this tool also will bring more closer our “bare metal” cluster, to the big cloud based Kubernetes clusters. After all you will be able to create PersistentVolumeClaims, that will automatically provisioned and more else.

We can use some additional system hard disks for GlusterFS setup or just create some fake block devices. In this example I will use second way.

On all three workers nodes you need to create fake block devices:

worker1-3# dd if=/dev/zero of=/home/gluster/image bs=1M count=10000

This will create some file about 10 GB size, then we need to use losetup for add it as a loop device on this nodes:

worker1-3# losetup /dev/loop0 /home/gluster/image
Please mind that if you already have some loop0 device, then you need to choose any other number.

Also I have spend some time, before I figured out why Heketi didn’t want to work properly. So to prevent any issues in future configurations first we need to make sure that we have loaded dm_thin_pool kernel module and install glusterfs-client package on all workers nodes.

worker1-3# modprobe dm_thin_pool
worker1-3# apt-get update && apt-get -y install glusterfs-client

OK, now we must have a /home/gluster/image file and a /dev/loop0 device on our workers nodes as well. Don’t forget to create a systemd service that will run losetup and modprobe automatically on every boot of this servers.

worker1-3# vi /etc/systemd/system/loop_gluster.service
Description=Create the loopback device for GlusterFS
ExecStart=/bin/bash -c "modprobe dm_thin_pool && [ -b /dev/loop0 ] || losetup /dev/loop0 /home/gluster/image"

And enable it:

worker1-3# systemctl enable /etc/systemd/system/loop_gluster.service
Created symlink /etc/systemd/system/ → /etc/systemd/system/loop_gluster.service.

Now all preparation done and we ready to deploy GlusterFS and Heketi on our cluster. I’ll use this nice tutorial for it. Most commands we’ll run from our external control machine, and a very small of them we need to run from any master node inside our cluster.

First let’s clone the repository and create a GlusterFS daemonset:

control# git clone
control# cd heketi/extras/kubernetes
control# kubectl create -f glusterfs-daemonset.json

Then we need to label our three workers nodes for the GlusterFS, after labels will set, the GlusterFS pods will be created on them:

control# kubectl label node worker1 storagenode=glusterfs
control# kubectl label node worker2 storagenode=glusterfs
control# kubectl label node worker3 storagenode=glusterfs
control# kubectl get pod
NAME                     READY   STATUS    RESTARTS   AGE
glusterfs-5dtdj 1/1 Running 0 1m6s
glusterfs-hzdll 1/1 Running 0 1m9s
glusterfs-p8r59 1/1 Running 0 2m1s

Now we need to create a Heketi service account:

control# kubectl create -f heketi-service-account.json

We must now establish the ability for that service account to control the gluster pods. We do this by creating a cluster role binding for our newly created service account:

control# kubectl create clusterrolebinding heketi-gluster-admin --clusterrole=edit --serviceaccount=default:heketi-service-account

Now let’s create a Kubernetes secret that will hold the configuration of our Heketi instance:

control# kubectl create secret generic heketi-config-secret --from-file=./heketi.json

Then create a first initial heketi pod, that we’ll use for the first few configuration steps and remove it after that:

control# kubectl create -f heketi-bootstrap.json
service "deploy-heketi" created
deployment "deploy-heketi" created
control# kubectl get pod
NAME                            READY     STATUS    RESTARTS   AGE
deploy-heketi-1211581626-2jotm 1/1 Running 0 2m
glusterfs-5dtdj 1/1 Running 0 6m6s
glusterfs-hzdll 1/1 Running 0 6m9s
glusterfs-p8r59 1/1 Running 0 7m1s

After the Bootstrap Heketi Service was created and running we need to move to one of our master node, there we’ll run few command because our control external node are not inside our cluster, so we can’t get any access to the running pods with internal cluster networking.

First let’s download a heketi-client utility and copy it in our system bin dir:

master1# wget
master1# tar -xzvf ./heketi-client-v8.0.0.linux.amd64.tar.gz
master1# cp ./heketi-client/bin/heketi-cli /usr/local/bin/
master1# heketi-cli 
heketi-cli v8.0.0

Now we need to find out a heketi pod IP and export it as a system variable:

master1# kubectl --kubeconfig /etc/kubernetes/admin.conf describe pod deploy-heketi-1211581626-2jotm 
For me this pod have a ip
master1# curl
Handling connection for 57598
Hello from Heketi
master1# export HEKETI_CLI_SERVER=

Now let’s provide a Heketi with information about the GlusterFS cluster it is to manage. We provide this information via a topology file. Topology is a JSON manifest with the list of all nodes, disks, and clusters used by GlusterFS.

NOTE: Make sure that hostnames/manage points to the exact name as shown under kubectl get nodes, and hostnames/storage is the IP address of the storage nodes.
master1:~/heketi-client# vi topology.json
"clusters": [
"nodes": [
"node": {
"hostnames": {
"manage": [
"storage": [
"zone": 1
"devices": [
"node": {
"hostnames": {
"manage": [
"storage": [
"zone": 1
"devices": [
"node": {
"hostnames": {
"manage": [
"storage": [
"zone": 1
"devices": [

And load this file then:

master1:~/heketi-client# heketi-cli topology load --json=topology.json
Creating cluster ... ID: e83467d0074414e3f59d3350a93901ef
Allowing file volumes on cluster.
Allowing block volumes on cluster.
Creating node worker1 ... ID: eea131d392b579a688a1c7e5a85e139c
Adding device /dev/loop0 ... OK
Creating node worker2 ... ID: 300ad5ff2e9476c3ba4ff69260afb234
Adding device /dev/loop0 ... OK
Creating node worker3 ... ID: 94ca798385c1099c531c8ba3fcc9f061
Adding device /dev/loop0 ... OK

Next we are going to use Heketi to provision a volume for it to store its database, the name of the command is a bit surprised but it’s Ok. Also create a heketi storage then:

master1:~/heketi-client# heketi-cli setup-openshift-heketi-storage
master1:~/heketi-client#  kubectl --kubeconfig /etc/kubernetes/admin.conf create -f heketi-storage.json 
secret/heketi-storage-secret created
endpoints/heketi-storage-endpoints created
service/heketi-storage-endpoints created
job.batch/heketi-storage-copy-job created

This is all command that we need to run from master node, now we can move back to the control node and continue from there, first make sure that last runned command was successful:

control# kubectl get pod
glusterfs-5dtdj 1/1 Running 0 39h
glusterfs-hzdll 1/1 Running 0 39h
glusterfs-p8r59 1/1 Running 0 39h
heketi-storage-copy-job-txkql 0/1 Completed 0 69s

And heketi-storage-copy-job was completed.

You’ll get a error if at this time your worker nodes will not have installed glusterfs-client package on it.

Now it’s time to delete the bootstrap Heketi installation and clear a bit after:

control# kubectl delete all,service,jobs,deployment,secret --selector="deploy-heketi"

At last step we need to create the long-term Heketi instance:

control# cd ./heketi/extras/kubernetes
control:~/heketi/extras/kubernetes# kubectl create -f heketi-deployment.json
secret/heketi-db-backup created
service/heketi created
deployment.extensions/heketi created
control# kubectl get pod
NAME                          READY   STATUS    RESTARTS   AGE
glusterfs-5dtdj 1/1 Running 0 39h
glusterfs-hzdll 1/1 Running 0 39h
glusterfs-p8r59 1/1 Running 0 39h
heketi-b8c5f6554-knp7t 1/1 Running 0 22m

You’ll get a error if at this time your worker nodes will not have installed glusterfs-client package on it.And we almost done, now the Heketi db will persist in a GlusterFS volume and will not reset every time the Heketi pod is restarted.

To start using GlusterFS cluster with dynamic provisioning we need to create some StorageClass.

First let’s find the Gluster Storage Endpoint to be passed in as a parameter to the StorageClass (heketi-storage-endpoints):

control# kubectl get endpoints
NAME                 ENDPOINTS                            AGE
heketi 2d16h
....... ... ..

Then create a few files:

control# vi storage-class.yml
kind: StorageClass
name: slow
resturl: ""
control# vi test-pvc.yml
kind: PersistentVolumeClaim
apiVersion: v1
name: gluster1
annotations: "slow"
- ReadWriteOnce
storage: 1Gi

Use this files to create both class and pvc:

control# kubectl create -f storage-class.yaml
storageclass "slow" created
control# kubectl get storageclass
NAME   PROVISIONER               AGE
slow 2d8h
control# kubectl create -f test-pvc.yaml
persistentvolumeclaim "gluster1" created
control# kubectl get pvc
NAME   STATUS   VOLUME  CAPACITY   ACCESS MODES STORAGECLASS  AGE gluster1 Bound  pvc-27f733cd-1c77-11e9-bb07-7efe6b0e6fa5   1Gi        RWO            slow           2d8h

We can also view the volume PV:

control# kubectl get pv
pvc-27f733cd-1c77-11e9-bb07-7efe6b0e6fa5 1Gi RWO Delete Bound default/gluster1 slow 2d8h

Now we have a dynamically created GlusterFS volume that bound to a PersistentVolumeClaim, we can now use this claim in a any pod.

Will create some simple Nginx pod and test it:

control# vi nginx-test.yml

apiVersion: v1
kind: Pod
name: nginx-pod1
name: nginx-pod1
- name: nginx-pod1
- name: web
containerPort: 80
- name: gluster-vol1
mountPath: /usr/share/nginx/html
- name: gluster-vol1
claimName: gluster1
control# kubectl create -f nginx-test.yaml
pod "nginx-pod1" created

View the Pod (Wait a few minutes, it might need to download the image if it doesn’t already exist):

control# kubectl get pods 
NAME                     READY   STATUS    RESTARTS   AGE
glusterfs-5dtdj 1/1 Running 0 4d10h
glusterfs-hzdll 1/1 Running 0 4d10h
glusterfs-p8r59 1/1 Running 0 4d10h
heketi-b8c5f6554-knp7t 1/1 Running 0 2d18h
nginx-pod1 1/1 Running 0 47h

Now let’s exec into the container and create some index.html file:

control# kubectl exec -ti nginx-pod1 /bin/sh
# cd /usr/share/nginx/html
# echo 'Hello there from GlusterFS pod !!!' > index.html
# ls
# exit

Now you need to find the pod internal IP and curl to it from any master nodes:

master1# curl
Hello there from GlusterFS pod !!!

That it, we just test our new persistent volume.

Some useful commands to check your new GlusterFS cluster areheketi-cli cluster list and heketi-cli volume list.That you can run on your machine with heketi-cli installed, in this example it’s a master1 node.
master1# heketi-cli cluster list
Id:e83467d0074414e3f59d3350a93901ef [file][block]
master1# heketi-cli volume list
Id:6fdb7fef361c82154a94736c8f9aa53e    Cluster:e83467d0074414e3f59d3350a93901ef    Name:vol_6fdb7fef361c82154a94736c8f9aa53e
Id:c6b69bd991b960f314f679afa4ad9644    Cluster:e83467d0074414e3f59d3350a93901ef    Name:heketidbstorage

At this point we successfully configured an internal load balancer with file storage and our cluster now more close to the production.

At next part of the article we’ll focusing on creating cluster monitoring and also will run test project on it to utilise all resources that we have configured.

Stay in touch and good luck !