Using Metal LB on a bare-metal(OnPrem) Kubernetes Setup

Jeganathan Swaminathan ( jegan@tektutor.org )

This article assumes, you already have a working Kubernetes Cluster setup locally either on your Laptop/Desktop/Workstation/Server.

If you want to setup a Kubernetes cluster locally, check my article here https://medium.com/@jegan_50867/kubernetes-3-node-cluster-setup-50943378be41.

I have my 3 node K8s cluster up and running

[jegan@master.tektutor.org ~]$ kubectl get nodes
NAME                   STATUS   ROLES                  AGE     VERSION
master.tektutor.org    Ready    control-plane,master   12m     v1.23.4
worker1.tektutor.org   Ready    <none>                 4m25s   v1.23.4
worker2.tektutor.org   Ready    <none>                 4m2s    v1.23.4

As you might be aware, Kubernetes supports 3 types of Services,

1. ClusterIP (Internal) Service
2. NodePort (External) Service and
3. LoadBalancer (External) Service

Out of the 3 services listed above, only the first two works out of the box in Local K8s setup. Minikube or MicroK8s are exceptions as they provide LoadBalancer out of the box just like Cloud environments. However, Minikube isn’t production grade setup hence, should be strictly avoided in production environment. Microk8s is a small-footprint K8s setup that also works well in production with small appliances like IOT.

In a Public cloud like AWS, Azure, GCP, etc., all the 3 types of Services works out of the box. For example, when you create a LoadBalancer service in AWS, it spins off a Network Load Balancer (NLB) and assigns the public IP of the NLB as external IP to the Load Balancer service. Interestingly, without MetalLB or equivalent deployment in Local K8s cluster, the Load Balancer external IP will remain in pending state forever.

Let’s create an nginx deployment.

kubectl create deploy nginx --image=nginx:1.20

Let’s now list the deploy, replicaset and pods as shown below

kubectl get deploy,rs,po

The expected output is

[jegan@master.tektutor.org ~]$ kubectl get deploy,rs,po
NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx   1/1     1            1           39s
NAME                               DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-6d777db949   1         1         1       38s
NAME                         READY   STATUS    RESTARTS   AGE
pod/nginx-6d777db949-sr8x6   1/1     Running   0          38s

Let’s scale up the nginx deployment

kubectl scale deploy/nginx --replicas=3

After scaling up nginx deployment to 3 replicas, the expected output is

[jegan@master.tektutor.org ~]$ kubectl get deploy,rs,po
NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx   3/3     3            3           2m8s
NAME                               DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-6d777db949   3         3         3       2m8s
NAME                         READY   STATUS    RESTARTS   AGE
pod/nginx-6d777db949-jttpw   1/1     Running   0          23s
pod/nginx-6d777db949-qmdk8   1/1     Running   0          23s
pod/nginx-6d777db949-sr8x6   1/1     Running   0          2m8s

Let’s create a LoadBalancer service for the above deployment

kubectl expose deploy/nginx --type=LoadBalancer --port=80

Let’s describe the nginx service we created

kubectl get svc
kubectl describe svc/nginx

The expected output is

[jegan@master.tektutor.org ~]$ kubectl get svc
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP      10.96.0.1       <none>        443/TCP        28m
nginx        LoadBalancer   10.103.19.189   <pending>     80:32019/TCP   5s
[jegan@master.tektutor.org ~]$ kubectl describe svc/nginx
Name:                     nginx
Namespace:                default
Labels:                   app=nginx
Annotations:              <none>
Selector:                 app=nginx
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.103.19.189
IPs:                      10.103.19.189
Port:                     <unset>  80/TCP
TargetPort:               80/TCP
NodePort:                 <unset>  32019/TCP
Endpoints:                192.168.145.193:80,192.168.145.194:80,192.168.72.129:80
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

Notice the External IP of the nginx LoadBalancer service is <pending> . In the absence of MetalLB or similar Load Balancer, the LoadBalancer service will never get an External IP in a bare metal K8s cluster, hence it will end up working exactly like a NodePort Service. This isn’t what you would have expected. MetalLB will solve this problem in our local K8s cluster.

Let’s scale down the nginx deployment to 0.

kubectl scale deploy/nginx --replicas=0

Metal LB supports two modes

1. Layer 2
2. BGP

Let us understand, lit bit of Kubernetes internal behaviour.

When we create an application deployment in K8s with multiple Pod replicas , the Pods will be scheduled on different workers nodes. When we expose those bunch of application Pods as a service, a single service routes the traffic to the application Pods, one Pod at a time. There is an instance of kube-proxy that runs on every node, which does basic load-balancing of the traffic within that node.

Metal LB in Layer 2 mode works as failover, rather than Load Balancer. In other words, all the traffic for a service will be forwarded to a single node and the kube-proxy in that node spreads the traffic to different Pods that belong to the service in that node. In case the leader node goes down, then the Metal LB will forward the traffic to kube-proxy on the other healthy node that was elected as new Leader node.

Metal LB in BGP mode works as a true Load Balancer.

This article demonstrates using Metal LB in Layer 2 mode.

The MetalLB installation procedure is simple.

Let’s create a namespace for MetaLB.

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.11.0/manifests/namespace.yaml

First you need to allocate some IP addresses for the internal use of Metal LB. You need to ensure the IP addresses aren’t already taken.

Each time we create a LoadBalancer service, Kubernetes will create an instance of MetalLB LoadBalancer. Kubernetes picks an available IP address from the reserved range of IP addresses specified in our config map and assigns it to the MetalLB LoadBalancer. The MetaLB LoadBalancer, will then Load Balance the bunch of Pod EndPoints associated with the LB Service we created for our application deployment.

Let’s create a file named metal-lb-cm.yml and append the below content.

metal-lb-cm.yml

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 192.168.254.240-192.168.254.250

Configure your firewall on master and all worker nodes

sudo firewall-cmd --permanent --add-port=7472/tcp --zone=trusted
sudo firewall-cmd --permanent --add-port=7472/udp --zone=trusted
sudo firewall-cmd --permanent --add-port=7946/tcp --zone=trusted
sudo firewall-cmd --permanent --add-port=7946/udp --zone=trusted
sudo firewall-cmd --reload
sudo firewall-cmd --list-all

The above ports are the default ports used by Metal LB, in case you have modified them, make sure you change the ports accordingly.

Let’s deploy MetalLB into our K8s cluster

kubectl apply -f metal-lb-cm.yml
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.11.0/manifests/metallb.yaml

Let’s list the pods running inside metallb-system namespace.

[jegan@master.tektutor.org ~]$ kubectl get pod  -n metallb-system
NAME                          READY   STATUS    RESTARTS   AGE
controller-7fbf768f66-m66ph   1/1     Running   0          30s
speaker-hj669                 1/1     Running   0          30s
speaker-l9sbp                 1/1     Running   0          30s
speaker-q9jjf                 1/1     Running   0          30s

Let’s now scale up the nginx deployment.

kubectl scale deploy nginx --replicas=3

Let’s now describe the LoadBalancer service to check if it is assigned with an external IP.

[jegan@master.tektutor.org ~]$ kubectl scale deploy nginx --replicas=3
deployment.apps/nginx scaled
[jegan@master.tektutor.org ~]$ kubectl expose deploy nginx --type=LoadBalancer --port=80
service/nginx exposed
[jegan@master.tektutor.org ~]$ kubectl get svc nginx
NAME    TYPE           CLUSTER-IP    EXTERNAL-IP       PORT(S)        AGE
nginx   LoadBalancer   10.102.5.84   192.168.254.240   80:31829/TCP   5s
[jegan@master.tektutor.org ~]$ kubectl describe svc/nginx
Name:                     nginx
Namespace:                default
Labels:                   app=nginx
Annotations:              <none>
Selector:                 app=nginx
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.102.5.84
IPs:                      10.102.5.84
LoadBalancer Ingress:     192.168.254.240
Port:                     <unset>  80/TCP
TargetPort:               80/TCP
NodePort:                 <unset>  31829/TCP
Endpoints:                192.168.145.195:80,192.168.145.196:80,192.168.72.132:80
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type    Reason        Age   From                Message
  ----    ------        ----  ----                -------
  Normal  IPAllocated   31s   metallb-controller  Assigned IP "192.168.254.240"
  Normal  nodeAssigned  16s   metallb-speaker     announcing from node "master.tektutor.org"

As you can see above, the nginx LoadBalancer service is assigned with an ExternalIP from the range we mentioned in the metallb config map.

You may now access the service as shown below.

[jegan@master.tektutor.org ~]$ curl http://192.168.254.240
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Troubleshooting Metal LB

Port Conflicts

By default, Metal LB uses the following ports 7472/tcp, 7472/udp, 7946/tcp and 7946/udp. In case those ports are already in use, you need to modify the ports to other non-conflicting ports in your VMs.

First delete all the metallb resources created under metallb-system namespace.

kubectl delete -f https://raw.githubusercontent.com/metallb/metallb/v0.11.0/manifests/metallb.yaml

You may download the manifest file to your system

wget https://raw.githubusercontent.com/metallb/metallb/v0.11.0/manifests/metallb.yaml

Search and replace the ports 7472 to 9472 ( or your preferred port ). Similarly replace all occurrences of port 7946 to 9946 ( or your preferred port ).

Make sure the line below with METALLB_ML_BIND_PORT and its values are uncommented and updated to use 9946 ( or your preferred port ).

# needed when another software is also using memberlist / port 7946
        # when changing this default you also need to update the container ports definition
        # and the PodSecurityPolicy hostPorts definition
        - name: METALLB_ML_BIND_PORT
          value: "9946"

Now try to deploy metal lb as shown below from your local file

kubectl apply -f metallb.yaml

This should fix the port conflict issue.

Did you configure your Firewall to open the Metal LB Ports?

In case your Firewall is active, it will prevent Metal LB from working correctly. Hence, make sure the ports used by Metal LB are opened in the firewall on master, worker1 and worker2 nodes.

sudo firewall-cmd --permanent --add-port=9472/tcp --zone=trusted
sudo firewall-cmd --permanent --add-port=9472/udp --zone=trusted
sudo firewall-cmd --permanent --add-port=9946/tcp --zone=trusted
sudo firewall-cmd --permanent --add-port=9946/udp --zone=trusted
sudo firewall-cmd --reload
sudo firewall-cmd --list-all

You may need to replace the above ports with the ports you configured in the metallb.yaml file as required.

You can follow the author to get notified when he publishes new articles.

If you found this post helpful, please click the clap 👏 button below a few times to show your support for the author 👇

My other articles

Setting up a 3 node Kubernetes Cluster locally

https://medium.com/@jegan_50867/kubernetes-3-node-cluster-setup-50943378be41

Using Nginx Ingress Controller in bare-metal Kubernetes setup

https://medium.com/@jegan_50867/using-nginx-ingress-controller-in-kubernetes-bare-metal-setup-890eb4e7772