Deploying a Personal Kubernetes Cluster — An Updated Guide

Ubuntu and Kubernetes logos — Canonical

Introduction

About a year ago, I wrote an article — The Ultimate Guide to On-Prem Kubernetes, that focused on deploying a Kubernetes cluster from scratch in a VMware private cloud with Terraform and Ansible automation.

While the guide is still up to date, and results in a healthy Kubernetes cluster, it is rather complicated and requires VMware Enterprise products to work correctly.

From extensive testing, I have found some issues with typical on-prem Kubernetes deployments:

• Kubernetes deployments are complicated. The previous guide relied on a working knowledge of Linux administration, complex configuration files, and had several usability issues.
• Etcd is I/O intensive and has very specific hardware / software requirements. During testing, I noticed that the cluster was using an abnormal amount of disk I/O — to the point where the datastore was complaining about drive contention, and the etcd instances on the master hosts ended up crashing due to a lack of storage performance. Note, this was tested on a local test server, without the multi-gigabit SANs, SSD accelerated drives, and the datacenter environment that the production cluster was deployed in.
• This item is most likely the result of the etcd issues outlined above. When building the cluster and joining nodes, I noticed that the master control plane would sometimes crash or freeze, resulting in join timeouts. This was annoying, because at the time, there was no way to reset the nodes or control plane if the joining failed.

The previous cluster was useable, and worked fine for personal projects and even some small-scale production workloads. But because of the items above and general usability issues, I decided to re-work the setup process and eliminate some of the unneeded steps.

Fixes / Improvements

Here are some of the fixes / improvements on this version vs the one I wrote about a year go:

• Kubeadm is now fully integrated and now runs the entire control plane, including certificate management.
• CFSSL is no longer used to pre-generate certificates. They are now auto-generated by Kubeadm.
• Swap configuration is improved. I noticed some redundant steps in the previous setup.
• More flexibility over the network configuration in the cluster. No IPs are assumed and the user gets to pick the subnet they want to use, as well as the LAN and WAN IPs for the cluster and ingress.
• MetalLB and Ngnix are used for load balancer and ingress services. This addresses an issue with internal load balancing and gives the user full load balancing and ingress control without having to rely on cloud providers.
• Terraform and Ansible have been replaced with VM templates. You can still use both platforms for VM creation, however the process relied on using out of date playbooks that do not work with Ubuntu 20.
• Added support for Ubuntu 20.
• Simplified HAProxy config.
• Support for Docker bleeding edge versions.
• Kubeadm initialization is now a single step, as is the join process for both the masters and workers.
• Simplified kubectl configs and support for remote management.
• Updated Kubernetes dashboard, configured as a service, with no proxying.
• Reduced hardware requirements and storage requirements. More lightweight.
• Updated packages.

So now that we covered the changes, lets start the install.

Hardware Requirements

For a fully redundant and highly available setup, we will be using 6 nodes. 3 masters and 3 workers. Note: the cluster is scalable. At minimum, I would recommend 2 masters and 1 worker, however you can adjust the setup at any time by adding or removing nodes via a single command.

We will also be creating one HAProxy machine. If you have an existing load balancer or HAProxy instance with internal IPs on the same subnet, that will also work.

First, we need to create a baseline template that we will then clone to create the additional nodes. This template will be the baseline for both the master and the worker nodes.

I would recommend the following specs:

• 120 GB HDD
• At least 2GB RAM
• x1 Gigabit network adapter
• 2 or more vCPUs

Note: You can also use physical machines or even RaspberryPis.

The recommended operating systems are:

• Ubuntu 20
• Debian 10
• CentOS 8

For this guide, I will be using Ubuntu 20.

Prepare the VMs

First, go into your VMware console or hypervisor of choice and create a new, blank machine with only the SSH package selected at install. Do not install anything else.

Note: You can use any private IPs for the VMs. The configuration no longer depends on a preconfigured set of IPs. The only requirement is that the machines should be able to communicate with each other. I highly recommend that none of the nodes are exposed to the internet. Instead, we will be using an external proxy server.

Once you have the first machine built, we can start installing packages, and the initial configuration. Make sure the machine’s network is configured (I recommend leaving DHCP off and using private static IPs).

First, upgrade the packages, update apt, and reboot:

sudo apt update 
sudo apt -y upgrade && sudo reboot now

Note: Make sure that all nodes can communicate with each other, not only via IP, but via hostname or FQDN. The FQDN or hostname of every machine in the cluster must be reachable from every other node.

If you use an internal or external DNS server, configure your A records now. If not, you can edit /etc/hosts and add the hostnames / FQDNs / IPs of every machine to the file on each node. Make sure to complete this step for every new node.

We will be using kubectl for sending commands to the cluster, kubelet for the actual Kubernetes service, and kubeadm for the control plane manager.

Next, add the repositories:

sudo apt update
sudo apt -y install curl apt-transport-https
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

Now we can install the packages:

sudo apt update
sudo apt -y install git curl kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

Next, check if kubeadm and kubectl were properly installed. Both commands should give you an output with package version information.

kubectl --version
kubeadm version 

Container hosting platforms break if swap is enabled, so lets turn it off:

sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
sudo swapoff -a

Configure sysctl:

These steps are required to enable transparent masquerading and to facilitate VxLAN traffic for communication between Kubernetes pods across the cluster, as well as some other critical configuration settings required by Kubernetes.

sudo modprobe overlay
sudo modprobe br_netfilter

sudo tee /etc/sysctl.d/kubernetes.conf<<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF

sudo sysctl --system

Next, we install docker and containerd.

sudo apt update
sudo apt install -y curl gnupg2 software-properties-common apt-transport-https ca-certificates
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt update
sudo apt install -y containerd.io docker-ce docker-ce-cli

sudo mkdir -p /etc/systemd/system/docker.service.d

sudo tee /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

Then, configure docker and kubelet to start on boot:

sudo systemctl daemon-reload 
sudo systemctl restart docker
sudo systemctl enable docker
sudo systemctl enable kubelet

Setup HAProxy

Build a new, clean machine that can run HAProxy. Note, you can use the template, however this is not recommended.

Once the machine is built, run:

sudo apt update && apt upgrade
sudo apt install haproxy

Edit the HAProxy config and add the control plane load balancing config:

frontend kubernetes
bind haproxy-ip:6443
option tcplog
mode tcp
default_backend kubernetes-master-nodes
backend kubernetes-master-nodes
mode tcp
balance roundrobin
option tcp-check
server master-1 192.168.2.52:6443 check fall 3 rise 2
server master-2 192.168.2.53:6443 check fall 3 rise 2
server master-3 192.168.2.54:6443 check fall 3 rise 2

Make sure that you change the IPs to match your own. In this setup, there are 3 master nodes, add or remove nodes as you need.

Note: The bind address in the frontend config is the control plane address that will be used later in the cluster initialization commands.

Once the config is updated, restart HAProxy and enable start at boot with:

service haproxy restart
service haproxy enable

Initialize the Cluster

Before initializing the cluster, we have to set up the other nodes.

You can now clone the virtual machine you have configured as many times as needed to create the cluster. Make sure that each node has a unique IP and hostname.

Also, make sure your hosts file or DNS server records are updated and every machine in the cluster can communicate with every other machine in the cluster via hostname or FQDNs (not just via the node IPs).

We can now start the initialization.

Control Plane Setup

On the first master node:

Pull the containers required by kubeadm:

sudo kubeadm config images pull

We can now run the initialization command.

Here is the breakdown of the commands:

• pod-network-cidr — define the network that will be used by the cluster. You can set any unused IP range in here.
• control-plane-endpoint-proxy — this is the address of the HAProxy load balancer configured previously. The FQDN must be reachable and mapped either in a DNS server or in the machine’s host file on every node.
• upload-certs — Save the certificates to a secret in the cluster. This enables sharing of the secret with other nodes upon joining without having to generate them or to manually copy them to each node.

Run the initialization command to create the control plane:

sudo kubeadm init \
  --pod-network-cidr=192.168.2.0/16 \
  --control-plane-endpoint=haproxy-loadbalancer.domain.com \
  --upload-certs

Once the command finishes running, it will show several things:

• Next steps on how to configure kubectl
• A master node join command
• A worker node join command

Save these commands as they will be used to join the other nodes.

Note: the control plane hostname / FQDN should map to the ip you set in the frontend config in HAProxy.

Configure Kubectl and Check Control Plane Health

Now that we have completed the initialization steps, we can start joining the other nodes.

First, lets configure kubectl:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

You can also scp the admin.conf file to a local machine and configure kubectl locally if needed.

Once kubectl is configured, we can check the status of the cluster and make sure everything initialized properly:

kubectl get nodes
kubectl cluster-info

The get nodes command should show a single node (your first master) in a NotReady status. The not ready status indicates that we have not yet configured networking on the cluster, so this is expected.

The cluster info command should show the address of the Kubernetes control plane along with some other information.

Join Master Nodes

Now we can join the remaining master nodes to the cluster.

Make sure your other VMs are configured properly and the hostname / FQDN of the cluster is reachable, then run the command that you saved from the init command.

Note: There are sperate join commands for the worker nodes and the master nodes. Also, the pull images step is not required as it will be completed when you run the join command.

The command will look something like:

kubeadm join cluster-name.domain.com:6443 --token some token \
    --discovery-token-ca-cert-hash sha256:some random string \
    --control-plane

Check the status of the control plane nodes:

kubectl get nodes

The master nodes should all appear and show a status of “Not Ready”.

Setting Up the Cluster

You should now have a healthy, unconfigured cluster. Before we do anything else, we need to configure the networking plugin. For this guide, I will be using Calico, which has advanced enterprise features along with many integrations. You can use any networking plugin that is compatible with Kubernetes and meets your requirements.

Deploy Calico:

kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

Once the command finishes running, all of the resources should show a status of “created”.

Confirm that Calico deployed successfully:

kubectl get pods --all-namespaces

All pods should show as “Running”.

Confirm that the cluster is ready:

kubectl get nodes -o wide

All of the master nodes should show a status of “Ready”.

Join Worker Nodes

Make sure your other VMs are configured properly and the hostname / FQDN of the cluster is reachable, then run the other (worker join) command that you saved from the init command.

Make sure you create the master cluster before joining your worker nodes.

Reminder, the command will look something like:

kubeadm join cluster-name.domain.com:6443 --token some token \
    --discovery-token-ca-cert-hash sha256:some random string \
    --control-plane

Once the process is complete, run:

kubectl get nodes

You should see 6 nodes (or however many you joined), 3 with the master / control plane roles, and 3 with no roles or worker roles. All of the nodes will show “Ready”.

If any of your node fail to join the process, or show as not ready (wait up to 5 minutes before taking any action for things to settle), you can reset kubeadm on the node:

kubeadm reset

Otherwise, proceed to the next step.

Setting Up the Dashboard (Optional)

The Kubernetes Dashboard gives you all kinds of information in a nice and clean GUI. The dashboard can be deployed to your Kubernetes cluster using the following steps:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.1.0/aio/deploy/recommended.yaml
kubectl delete service kubernetes-dashboard -n kubernetes-dashboard
kubectl expose deployment kubernetes-dashboard --type=LoadBalancer --name=kubernetes-dashboard --external-ip=your node ip here -n kubernetes-dashboard

These commands:

• Deploy the dashboard from the deployment YAML.
• Delete the included node port service
• Expose it as a load balancer service instead

You can access the dashboard at https://node-ip:8001

To configure RBAC / authentication:

kubectl delete  clusterrolebinding kubernetes-dashboard -n kubernetes-dashboard
kubectl create clusterrolebinding kubernetes-dashboard --clusterrole=cluster-admin --serviceaccount=kubernetes-dashboard:kubernetes-dashboard
kubectl get secrets -n kubernetes-dashboard

Find the secret that looks like kubernetes-dashboard-token-12345

Then run:

kubectl describe secret kubernetes-dashboard-1235 -n kubernetes-dashboard

Copy and paste the secret string to the Kubernetes dashboard “token” feild. Note: Save this string in a safe place! It is a master admin account that can do anything in the cluster.

External Management

If you want to add your cluster to an app like Kuber for iOS, use https://cluster-ip:6443 and the cluster admin account you created for the dashboard.

Other Notes

Note: There are articles out there that say “Kubernetes out of the box deployments cannot use load balancers unless you are in a cloud environments”.

This is not true. You can expose deployments via load balancer right out of the box using this cluster deployment method. Also, I’m not a fan of kubectl proxy. There is not risk or cluster overhead if you expose the service via load balancer instead (assuming you don’t use ingress or have your cluster exposed to the open internet).

Deploy Ingress Controller (Optional)

This is an optional step to deploy the Nginx Ingress Controller.

(More information at https://kubernetes.github.io/ingress-nginx/deploy/)

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.44.0/deploy/static/provider/baremetal/deploy.yaml

Deploy MetalLB (Optional)

Metal LB is a load balancing service for bare metal Kubernetes clusters.

(More information at https://metallb.universe.tf/)

Note: Metal LB is beta software. It also requires you to edit the cluster’s ConfigMaps. Be careful, you can break stuff there.

Thank you for reading!