Configuring HA Kubernetes cluster on bare metal servers with GlusterFS & MetalLB . 2/3

Alexey Nizhegolenko
Jan 22, 2019 · 12 min read
Image for post
Image for post

Hello and welcome back, this is the second part of an article about configuring Kubernetes cluster on bare metal servers. Previously we configured an HA Kubernetes cluster with external ETCD, multi-masters schema, and load balance. Well, so far now it’s time to set up additional environment and utilities for making your cluster more useful and closest to production ready.

In this part of article, we’ll be focusing on setting up the internal load balancer for our cluster services — it’ll be a MetalLB. Also, we’ll install and configure a distributed file storage between our worker’s nodes. We’ll use a GlusterFS for the persistent volumes that will be available inside Kubernetes.

After all, be done our cluster schema will look like this:

Image for post
Image for post

1. Configuring MetalLB as internal load balancer.

Few words about MetalLB directly from the document page:

So, using this tool we can run our services under load balancer on our Kubernetes cluster, and many thanks to the MetalLB team for it. The configuration process is really easy and clear.

Previously in example, we have chosen a 192.168.0.0/24 subnet for our cluster needs. Now we need to take some part of this subnet for our future load balancer.

Login to the machine that has configured kubectl utility and run:

control# kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.7.3/manifests/metallb.yaml

This will deploy a MetalLB to your cluster, under the metallb-system namespace. Make sure that all components of MetalLB working OK then:

control# kubectl get pod --namespace=metallb-system 
NAME READY STATUS RESTARTS AGE
controller-7cc9c87cfb-ctg7p 1/1 Running 0 5d3h
speaker-82qb5 1/1 Running 0 5d3h
speaker-h5jw7 1/1 Running 0 5d3h
speaker-r2fcg 1/1 Running 0 5d3h

Now we need to configure MetalLB using configmap, we’ll use a Layer 2 configuration for this example. For more configurations variants please refer to the MetalLB documentation.

Create a metallb-config.yaml file in any directory with chosen IP range from our cluster subnet:

control# vi metallb-config.yamlapiVersion: v1 
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: default
protocol: layer2
addresses:
- 192.168.0.240-192.168.0.250

And apply this configuration:

control# kubectl apply -f metallb-config.yaml

You can check and change this configmap later if you need:

control# kubectl describe configmaps -n metallb-system
control# kubectl edit configmap config -n metallb-system

From now we got our own local load balancer configured, let’s check how it works using some example of Nginx service.

control# vi nginx-deployment.yamlapiVersion: apps/v1 
kind: Deployment
metadata:
name: nginx-deployment
spec:
selector:
matchLabels:
app: nginx
replicas: 3
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:latest
ports:
- containerPort: 80
control# vi nginx-service.yamlapiVersion: v1
kind: Service
metadata:
name: nginx
spec:
type: LoadBalancer
selector:
app: nginx
ports:
- port: 80
name: http

Next, create this test Nginx deployment and service:

control# kubectl apply -f nginx-deployment.yamlcontrol# kubectl apply -f nginx-service.yaml

And let’s check the result:

control# kubectl get poNAME                               READY STATUS    RESTARTS   AGE 
nginx-deployment-6574bd76c-fxgxr 1/1 Running 0 19s
nginx-deployment-6574bd76c-rp857 1/1 Running 0 19s
nginx-deployment-6574bd76c-wgt9n 1/1 Running 0 19s
control# kubectl get svcNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx LoadBalancer 10.100.226.110 192.168.0.240 80:31604/TCP 107s

There was created a 3 Nginx pods as we specified in our deployment previously. Nginx service will send traffic to all of them with round-robin balance schema. And also you can see the external IP that we got from our MetalLB LB.

Now try curl to the 192.168.0.240 IP address and you will see the Nginx index.html page. Don’t forget to remove this test Nginx deployment and service also.

control# kubectl delete svc nginx 
service "nginx" deleted
control# kubectl delete deployment nginx-deployment
deployment.extensions "nginx-deployment" deleted

OK, that all about MetalLB, now we can move on and configure the GlusterFS for the Kubernetes volumes.

2. Configuring GlusterFS with Heketi on workers nodes.

In fact, using the Kubernetes cluster is impossible without using volumes inside it. As you know Pod’s are ephemeral and it means that they can be created and deleted at any time. And all the data they have inside will be lost. So in a real cluster, you will need some distributed storage, to have the availability of sharing configs and data between nodes and applications inside it.

In Kubernetes you can use volumes, there are different variants of them so you can choose some for your needs. In this example, I wanna show a creating of a GlusterFS storage that we can use for any kind of internal applications as persistent volumes. Previously I used for this a “system” installation of GlusterFS on all of the Kubernetes worker nodes and then I just created a hostPath type of volumes inside GlusterFS directories.

But now we have a new nice Heketi tool.

Few words from Heketi documentation:

That sounds good, and this tool also will bring closer our “bare metal” cluster, to the big cloud-based Kubernetes clusters. After all, you will be able to create PersistentVolumeClaims, which will automatically be provisioned and more else.

We can use some additional system hard disks for the GlusterFS setup or just create some fake block devices. In this example, I will use the second way.

On all three workers nodes you need to create fake block devices:

worker1-3# dd if=/dev/zero of=/home/gluster/image bs=1M count=10000

This will create some file about 10 GB size, then we need to use losetup to add it as a loop device on these nodes:

worker1-3# losetup /dev/loop0 /home/gluster/image

Also, I have spent some time, before I figured out why Heketi didn’t want to work properly. So to prevent any issues in future configurations first we need to make sure that we have loaded dm_thin_pool kernel module and install glusterfs-client package on all workers nodes.

worker1-3# modprobe dm_thin_poolworker1-3# apt-get update && apt-get -y install glusterfs-client

OK, now we must have a /home/gluster/image file and a /dev/loop0 device on our worker’s nodes as well. Don’t forget to create a systemd service that will run losetup and modprobe automatically on every boot of these servers.

worker1-3# vi /etc/systemd/system/loop_gluster.service[Unit]
Description=Create the loopback device for GlusterFS
DefaultDependencies=false
Before=local-fs.target
After=systemd-udev-settle.service
Requires=systemd-udev-settle.service
[Service]
Type=oneshot
ExecStart=/bin/bash -c "modprobe dm_thin_pool && [ -b /dev/loop0 ] || losetup /dev/loop0 /home/gluster/image"
[Install]
WantedBy=local-fs.target

And enable it:

worker1-3# systemctl enable /etc/systemd/system/loop_gluster.serviceCreated symlink /etc/systemd/system/local-fs.target.wants/loop_gluster.service → /etc/systemd/system/loop_gluster.service.

Now all preparation is done and we ready to deploy GlusterFS and Heketi on our cluster. I’ll use this nice tutorial for it. Most commands we’ll run from our external control machine, and a very small of them we need to run from any master node inside our cluster.

First, let’s clone the repository and create a GlusterFS daemonset:

control# git clone https://github.com/heketi/heketicontrol# cd heketi/extras/kubernetescontrol# kubectl create -f glusterfs-daemonset.json

Then we need to label our three workers nodes for the GlusterFS after labels will set, the GlusterFS pods will be created on them:

control# kubectl label node worker1 storagenode=glusterfs
control# kubectl label node worker2 storagenode=glusterfs
control# kubectl label node worker3 storagenode=glusterfs
control# kubectl get podNAME READY STATUS RESTARTS AGE
glusterfs-5dtdj 1/1 Running 0 1m6s
glusterfs-hzdll 1/1 Running 0 1m9s
glusterfs-p8r59 1/1 Running 0 2m1s

Now we need to create a Heketi service account:

control# kubectl create -f heketi-service-account.json

We must now establish the ability for that service account to control the gluster pods. We do this by creating a cluster role binding for our newly created service account:

control# kubectl create clusterrolebinding heketi-gluster-admin --clusterrole=edit --serviceaccount=default:heketi-service-account

Now let’s create a Kubernetes secret that will hold the configuration of our Heketi instance:

control# kubectl create secret generic heketi-config-secret --from-file=./heketi.json

Then create a first initial heketi pod, that we’ll use for the first few configuration steps and remove it after that:

control# kubectl create -f heketi-bootstrap.jsonservice "deploy-heketi" created
deployment "deploy-heketi" created
control# kubectl get podNAME READY STATUS RESTARTS AGE
deploy-heketi-1211581626-2jotm 1/1 Running 0 2m
glusterfs-5dtdj 1/1 Running 0 6m6s
glusterfs-hzdll 1/1 Running 0 6m9s
glusterfs-p8r59 1/1 Running 0 7m1s

After the Bootstrap Heketi Service was created and running we need to move to one of our master nodes, there we’ll run a few commands because our control external node is not inside our cluster, so we can’t get any access to the running pods with internal cluster networking.

First, let’s download a heketi-client utility and copy it in our system bin dir:

master1# wget https://github.com/heketi/heketi/releases/download/v8.0.0/heketi-client-v8.0.0.linux.amd64.tar.gzmaster1# tar -xzvf ./heketi-client-v8.0.0.linux.amd64.tar.gz
master1# cp ./heketi-client/bin/heketi-cli /usr/local/bin/
master1# heketi-cli
heketi-cli v8.0.0

Now we need to find out a heketi pod IP and export it as a system variable:

master1# kubectl --kubeconfig /etc/kubernetes/admin.conf describe pod deploy-heketi-1211581626-2jotmFor me this pod have a 10.42.0.1 ipmaster1# curl http://10.42.0.1:57598/hello
Handling connection for 57598
Hello from Heketi
master1# export HEKETI_CLI_SERVER=http://10.42.0.1:57598

Now let’s provide a Heketi with information about the GlusterFS cluster it is to manage. We provide this information via a topology file. Topology is a JSON manifest with the list of all nodes, disks, and clusters used by GlusterFS.

master1:~/heketi-client# vi topology.json{
"clusters": [
{
"nodes": [
{
"node": {
"hostnames": {
"manage": [
"worker1"
],
"storage": [
"192.168.0.7"
]
},
"zone": 1
},
"devices": [
"/dev/loop0"
]
},
{
"node": {
"hostnames": {
"manage": [
"worker2"
],
"storage": [
"192.168.0.8"
]
},
"zone": 1
},
"devices": [
"/dev/loop0"
]
},
{
"node": {
"hostnames": {
"manage": [
"worker3"
],
"storage": [
"192.168.0.9"
]
},
"zone": 1
},
"devices": [
"/dev/loop0"
]
}
]
}
]
}

And load this file then:

master1:~/heketi-client# heketi-cli topology load --json=topology.jsonCreating cluster ... ID: e83467d0074414e3f59d3350a93901ef
Allowing file volumes on cluster.
Allowing block volumes on cluster.
Creating node worker1 ... ID: eea131d392b579a688a1c7e5a85e139c
Adding device /dev/loop0 ... OK
Creating node worker2 ... ID: 300ad5ff2e9476c3ba4ff69260afb234
Adding device /dev/loop0 ... OK
Creating node worker3 ... ID: 94ca798385c1099c531c8ba3fcc9f061
Adding device /dev/loop0 ... OK

Next, we are going to use Heketi to provision a volume for it to store its database, the name of the command is a bit surprised but it’s Ok. Also, create a heketi storage then:

master1:~/heketi-client# heketi-cli setup-openshift-heketi-storagemaster1:~/heketi-client#  kubectl --kubeconfig /etc/kubernetes/admin.conf create -f heketi-storage.jsonsecret/heketi-storage-secret created
endpoints/heketi-storage-endpoints created
service/heketi-storage-endpoints created
job.batch/heketi-storage-copy-job created

This is all command that we need to run from the master node, now we can move back to the control node and continue from there, first make sure that last run command was successful:

control# kubectl get pod
NAME READY STATUS RESTARTS AGE
glusterfs-5dtdj 1/1 Running 0 39h
glusterfs-hzdll 1/1 Running 0 39h
glusterfs-p8r59 1/1 Running 0 39h
heketi-storage-copy-job-txkql 0/1 Completed 0 69s

And heketi-storage-copy-job was completed.

Now it’s time to delete the bootstrap Heketi installation and clear a bit after:

control# kubectl delete all,service,jobs,deployment,secret --selector="deploy-heketi"

At the last step we need to create the long-term Heketi instance:

control# cd ./heketi/extras/kubernetescontrol:~/heketi/extras/kubernetes# kubectl create -f heketi-deployment.jsonsecret/heketi-db-backup created
service/heketi created
deployment.extensions/heketi created
control# kubectl get podNAME READY STATUS RESTARTS AGE
glusterfs-5dtdj 1/1 Running 0 39h
glusterfs-hzdll 1/1 Running 0 39h
glusterfs-p8r59 1/1 Running 0 39h
heketi-b8c5f6554-knp7t 1/1 Running 0 22m

You’ll get an error if at this time your worker nodes will not have installed glusterfs-client package on it. And we almost have done, now the Heketi DB will persist in a GlusterFS volume and will not reset every time the Heketi pod is restarted.

To start using the GlusterFS cluster with dynamic provisioning we need to create some StorageClass.

First, let’s find the Gluster Storage Endpoint to be passed in as a parameter to the StorageClass (heketi-storage-endpoints):

control# kubectl get endpointsNAME                 ENDPOINTS                            AGE
heketi 10.42.0.2:8080 2d16h
....... ... ..

Then create a few files:

control# vi storage-class.ymlapiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
name: slow
provisioner: kubernetes.io/glusterfs
parameters:
resturl: "http://10.42.0.2:8080"
control# vi test-pvc.yml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: gluster1
annotations:
volume.beta.kubernetes.io/storage-class: "slow"
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi

Use these files to create both class and PVC:

control# kubectl create -f storage-class.yaml
storageclass "slow" created
control# kubectl get storageclassNAME PROVISIONER AGE
slow kubernetes.io/glusterfs 2d8h
control# kubectl create -f test-pvc.yaml
persistentvolumeclaim "gluster1" created
control# kubectl get pvcNAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE gluster1 Bound pvc-27f733cd-1c77-11e9-bb07-7efe6b0e6fa5 1Gi RWO slow 2d8h

We can also view the volume PV:

control# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-27f733cd-1c77-11e9-bb07-7efe6b0e6fa5 1Gi RWO Delete Bound default/gluster1 slow 2d8h

Now we have a dynamically created GlusterFS volume that bound to a PersistentVolumeClaim, we can now use this claim in any pod.

Will create some simple Nginx pod and test it:

control# vi nginx-test.yml

apiVersion: v1
kind: Pod
metadata:
name: nginx-pod1
labels:
name: nginx-pod1
spec:
containers:
- name: nginx-pod1
image: gcr.io/google_containers/nginx-slim:0.8
ports:
- name: web
containerPort: 80
volumeMounts:
- name: gluster-vol1
mountPath: /usr/share/nginx/html
volumes:
- name: gluster-vol1
persistentVolumeClaim:
claimName: gluster1
control# kubectl create -f nginx-test.yaml
pod "nginx-pod1" created

View the Pod (Wait a few minutes, it might need to download the image if it doesn’t already exist):

control# kubectl get podsNAME                     READY   STATUS    RESTARTS   AGE
glusterfs-5dtdj 1/1 Running 0 4d10h
glusterfs-hzdll 1/1 Running 0 4d10h
glusterfs-p8r59 1/1 Running 0 4d10h
heketi-b8c5f6554-knp7t 1/1 Running 0 2d18h
nginx-pod1 1/1 Running 0 47h

Now let’s exec into the container and create some index.html file:

control# kubectl exec -ti nginx-pod1 /bin/sh# cd /usr/share/nginx/html
# echo 'Hello there from GlusterFS pod !!!' > index.html
# ls
index.html
# exit

Now you need to find the pod internal IP and curl to it from any master nodes:

master1# curl 10.40.0.1Hello there from GlusterFS pod !!!

That is, we just test our new persistent volume.

master1# heketi-cli cluster list
Clusters:
Id:e83467d0074414e3f59d3350a93901ef [file][block]
master1# heketi-cli volume listId:6fdb7fef361c82154a94736c8f9aa53e Cluster:e83467d0074414e3f59d3350a93901ef Name:vol_6fdb7fef361c82154a94736c8f9aa53eId:c6b69bd991b960f314f679afa4ad9644 Cluster:e83467d0074414e3f59d3350a93901ef Name:heketidbstorage

At this point, we successfully configured an internal load balancer with file storage and our cluster now more close to the production.

At the next part of the article, we’ll be focusing on creating cluster monitoring and also will run a test project on it to utilize all the resources that we have configured.

Stay in touch and good luck!

Image for post
Image for post

Join our community Slack and read our weekly Faun topics ⬇

Image for post
Image for post

If this post was helpful, please click the clap 👏 button below a few times to show your support for the author! ⬇

FAUN

The Must-Read Publication for Creative Developers & DevOps Enthusiasts

Sign up for FAUN

By FAUN

Medium’s largest and most followed independent DevOps publication. Join thousands of aspiring developers and DevOps enthusiasts Take a look

By signing up, you will create a Medium account if you don’t already have one. Review our Privacy Policy for more information about our privacy practices.

Check your inbox
Medium sent you an email at to complete your subscription.

Alexey Nizhegolenko

Written by

Sysadmin & DevOps Engineer in Internetvikings, Stockholm.

FAUN

FAUN

The Must-Read Publication for Creative Developers & DevOps Enthusiasts. Medium’s largest DevOps publication.

Alexey Nizhegolenko

Written by

Sysadmin & DevOps Engineer in Internetvikings, Stockholm.

FAUN

FAUN

The Must-Read Publication for Creative Developers & DevOps Enthusiasts. Medium’s largest DevOps publication.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store