Using Metal LB on a bare-metal(OnPrem) Kubernetes Setup
Jeganathan Swaminathan ( jegan@tektutor.org )
This article assumes, you already have a working Kubernetes Cluster setup locally either on your Laptop/Desktop/Workstation/Server.
If you want to setup a Kubernetes cluster locally, check my article here https://medium.com/@jegan_50867/kubernetes-3-node-cluster-setup-50943378be41.
I have my 3 node K8s cluster up and running
[jegan@master.tektutor.org ~]$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master.tektutor.org Ready control-plane,master 12m v1.23.4
worker1.tektutor.org Ready <none> 4m25s v1.23.4
worker2.tektutor.org Ready <none> 4m2s v1.23.4
As you might be aware, Kubernetes supports 3 types of Services,
- ClusterIP (Internal) Service
- NodePort (External) Service and
- LoadBalancer (External) Service
Out of the 3 services listed above, only the first two works out of the box in Local K8s setup. Minikube or MicroK8s are exceptions as they provide LoadBalancer out of the box just like Cloud environments. However, Minikube isn’t production grade setup hence, should be strictly avoided in production environment. Microk8s is a small-footprint K8s setup that also works well in production with small appliances like IOT.
In a Public cloud like AWS, Azure, GCP, etc., all the 3 types of Services works out of the box. For example, when you create a LoadBalancer service in AWS, it spins off a Network Load Balancer (NLB) and assigns the public IP of the NLB as external IP to the Load Balancer service. Interestingly, without MetalLB or equivalent deployment in Local K8s cluster, the Load Balancer external IP will remain in pending
state forever.
Let’s create an nginx deployment.
kubectl create deploy nginx --image=nginx:1.20
Let’s now list the deploy, replicaset and pods as shown below
kubectl get deploy,rs,po
The expected output is
[jegan@master.tektutor.org ~]$ kubectl get deploy,rs,po
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/nginx 1/1 1 1 39sNAME DESIRED CURRENT READY AGE
replicaset.apps/nginx-6d777db949 1 1 1 38sNAME READY STATUS RESTARTS AGE
pod/nginx-6d777db949-sr8x6 1/1 Running 0 38s
Let’s scale up the nginx deployment
kubectl scale deploy/nginx --replicas=3
After scaling up nginx deployment to 3 replicas, the expected output is
[jegan@master.tektutor.org ~]$ kubectl get deploy,rs,po
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/nginx 3/3 3 3 2m8sNAME DESIRED CURRENT READY AGE
replicaset.apps/nginx-6d777db949 3 3 3 2m8sNAME READY STATUS RESTARTS AGE
pod/nginx-6d777db949-jttpw 1/1 Running 0 23s
pod/nginx-6d777db949-qmdk8 1/1 Running 0 23s
pod/nginx-6d777db949-sr8x6 1/1 Running 0 2m8s
Let’s create a LoadBalancer service for the above deployment
kubectl expose deploy/nginx --type=LoadBalancer --port=80
Let’s describe the nginx service we created
kubectl get svc
kubectl describe svc/nginx
The expected output is
[jegan@master.tektutor.org ~]$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 28m
nginx LoadBalancer 10.103.19.189 <pending> 80:32019/TCP 5s
[jegan@master.tektutor.org ~]$ kubectl describe svc/nginx
Name: nginx
Namespace: default
Labels: app=nginx
Annotations: <none>
Selector: app=nginx
Type: LoadBalancer
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.103.19.189
IPs: 10.103.19.189
Port: <unset> 80/TCP
TargetPort: 80/TCP
NodePort: <unset> 32019/TCP
Endpoints: 192.168.145.193:80,192.168.145.194:80,192.168.72.129:80
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
Notice the External IP of the nginx LoadBalancer service is <pending>
. In the absence of MetalLB or similar Load Balancer, the LoadBalancer service will never get an External IP in a bare metal K8s cluster, hence it will end up working exactly like a NodePort Service. This isn’t what you would have expected. MetalLB will solve this problem in our local K8s cluster.
Let’s scale down the nginx deployment to 0.
kubectl scale deploy/nginx --replicas=0
Metal LB supports two modes
- Layer 2
- BGP
Let us understand, lit bit of Kubernetes internal behaviour.
When we create an application deployment in K8s with multiple Pod replicas , the Pods will be scheduled on different workers nodes. When we expose those bunch of application Pods as a service, a single service routes the traffic to the application Pods, one Pod at a time. There is an instance of kube-proxy that runs on every node, which does basic load-balancing of the traffic within that node.
Metal LB in Layer 2 mode works as failover, rather than Load Balancer. In other words, all the traffic for a service will be forwarded to a single node and the kube-proxy in that node spreads the traffic to different Pods that belong to the service in that node. In case the leader node goes down, then the Metal LB will forward the traffic to kube-proxy on the other healthy node that was elected as new Leader node.
Metal LB in BGP mode works as a true Load Balancer.
This article demonstrates using Metal LB in Layer 2 mode.
The MetalLB installation procedure is simple.
Let’s create a namespace for MetaLB.
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.11.0/manifests/namespace.yaml
First you need to allocate some IP addresses for the internal use of Metal LB. You need to ensure the IP addresses aren’t already taken.
Each time we create a LoadBalancer service, Kubernetes will create an instance of MetalLB LoadBalancer. Kubernetes picks an available IP address from the reserved range of IP addresses specified in our config map and assigns it to the MetalLB LoadBalancer. The MetaLB LoadBalancer, will then Load Balance the bunch of Pod EndPoints associated with the LB Service we created for our application deployment.
Let’s create a file named metal-lb-cm.yml
and append the below content.
metal-lb-cm.yml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: default
protocol: layer2
addresses:
- 192.168.254.240-192.168.254.250
Configure your firewall on master and all worker nodes
sudo firewall-cmd --permanent --add-port=7472/tcp --zone=trusted
sudo firewall-cmd --permanent --add-port=7472/udp --zone=trustedsudo firewall-cmd --permanent --add-port=7946/tcp --zone=trusted
sudo firewall-cmd --permanent --add-port=7946/udp --zone=trustedsudo firewall-cmd --reload
sudo firewall-cmd --list-all
The above ports are the default ports used by Metal LB, in case you have modified them, make sure you change the ports accordingly.
Let’s deploy MetalLB into our K8s cluster
kubectl apply -f metal-lb-cm.ymlkubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.11.0/manifests/metallb.yaml
Let’s list the pods running inside metallb-system namespace.
[jegan@master.tektutor.org ~]$ kubectl get pod -n metallb-system
NAME READY STATUS RESTARTS AGE
controller-7fbf768f66-m66ph 1/1 Running 0 30s
speaker-hj669 1/1 Running 0 30s
speaker-l9sbp 1/1 Running 0 30s
speaker-q9jjf 1/1 Running 0 30s
Let’s now scale up the nginx deployment.
kubectl scale deploy nginx --replicas=3
Let’s now describe the LoadBalancer service to check if it is assigned with an external IP.
[jegan@master.tektutor.org ~]$ kubectl scale deploy nginx --replicas=3
deployment.apps/nginx scaled
[jegan@master.tektutor.org ~]$ kubectl expose deploy nginx --type=LoadBalancer --port=80
service/nginx exposed
[jegan@master.tektutor.org ~]$ kubectl get svc nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx LoadBalancer 10.102.5.84 192.168.254.240 80:31829/TCP 5s
[jegan@master.tektutor.org ~]$ kubectl describe svc/nginx
Name: nginx
Namespace: default
Labels: app=nginx
Annotations: <none>
Selector: app=nginx
Type: LoadBalancer
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.102.5.84
IPs: 10.102.5.84
LoadBalancer Ingress: 192.168.254.240
Port: <unset> 80/TCP
TargetPort: 80/TCP
NodePort: <unset> 31829/TCP
Endpoints: 192.168.145.195:80,192.168.145.196:80,192.168.72.132:80
Session Affinity: None
External Traffic Policy: Cluster
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal IPAllocated 31s metallb-controller Assigned IP "192.168.254.240"
Normal nodeAssigned 16s metallb-speaker announcing from node "master.tektutor.org"
As you can see above, the nginx LoadBalancer service is assigned with an ExternalIP from the range we mentioned in the metallb config map.
You may now access the service as shown below.
[jegan@master.tektutor.org ~]$ curl http://192.168.254.240
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p><p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p><p><em>Thank you for using nginx.</em></p>
</body>
</html>
Troubleshooting Metal LB
Port Conflicts
By default, Metal LB uses the following ports 7472/tcp, 7472/udp, 7946/tcp and 7946/udp. In case those ports are already in use, you need to modify the ports to other non-conflicting ports in your VMs.
First delete all the metallb resources created under metallb-system
namespace.
kubectl delete -f https://raw.githubusercontent.com/metallb/metallb/v0.11.0/manifests/metallb.yaml
You may download the manifest file to your system
wget https://raw.githubusercontent.com/metallb/metallb/v0.11.0/manifests/metallb.yaml
Search and replace the ports 7472 to 9472 ( or your preferred port ). Similarly replace all occurrences of port 7946 to 9946 ( or your preferred port ).
Make sure the line below with METALLB_ML_BIND_PORT
and its values are uncommented and updated to use 9946 ( or your preferred port ).
# needed when another software is also using memberlist / port 7946
# when changing this default you also need to update the container ports definition
# and the PodSecurityPolicy hostPorts definition
- name: METALLB_ML_BIND_PORT
value: "9946"
Now try to deploy metal lb as shown below from your local file
kubectl apply -f metallb.yaml
This should fix the port conflict issue.
Did you configure your Firewall to open the Metal LB Ports?
In case your Firewall is active, it will prevent Metal LB from working correctly. Hence, make sure the ports used by Metal LB are opened in the firewall on master
, worker1
and worker2
nodes.
sudo firewall-cmd --permanent --add-port=9472/tcp --zone=trusted
sudo firewall-cmd --permanent --add-port=9472/udp --zone=trustedsudo firewall-cmd --permanent --add-port=9946/tcp --zone=trusted
sudo firewall-cmd --permanent --add-port=9946/udp --zone=trustedsudo firewall-cmd --reload
sudo firewall-cmd --list-all
You may need to replace the above ports with the ports you configured in the metallb.yaml
file as required.
You can follow the author to get notified when he publishes new articles.
If you found this post helpful, please click the clap 👏 button below a few times to show your support for the author 👇
My other articles
Setting up a 3 node Kubernetes Cluster locally
https://medium.com/@jegan_50867/kubernetes-3-node-cluster-setup-50943378be41
Using Nginx Ingress Controller in bare-metal Kubernetes setup