Kubernetes the hard way on bare metal/VMs — V1.23
A new all encompassing series designed for beginners. From VM configuration to K8S cluster building.
Introduction
3 years after the first article I wrote on Kubernetes the hard way on Bare Metal and 11 Kubernetes versions later, figured it was time to bring this guide up to date. We’ll be going through the latest version of Kubernetes, V1.23, which was released on 7th December 2021.
As before, I strongly recommend tutorials such as Kelsey Hightower’s “Kubernetes The Hard Way”. It’s an excellent tutorial that takes you through every step of setting up a Kubernetes Cluster however presumes you’re using GCE and all of the features that come with it. You can get some free credit on the system but if you are looking to set up on bare metal and don’t know much about configuring networks, firewalls or load-balancers then you may struggle with Kelsey’s guide. You can end up in a web of tutorials where you’re trying to understand a lot just to grasp one concept — Kubernetes.
The purpose of this guide is to help those who want to learn how to set up Kubernetes using the services themselves instead of using something like KubeADM, which creates the services within the cluster using container images.
Note: You’ll see me use terms such as Machine, PC, Laptop, Server, VM, Node, Controller and Worker in this tutorial; these simply refer to a physical machine or a VM with Linux installed on it. You don’t need fancy, costly servers or services; just free software and a machine with a fair amount of RAM.
Don’t let terminology confuse you!
Read the docs
I can’t stress enough how useful it is to read and understand the documentation on the main Kubernetes website to learn how to set clusters up in a variety of different ways.
Setting up the resources
I’ll be setting this up on virtual machines, via libvirt, using my home PC however you can replicate this with Virtualbox or physical servers allowing you to follow along from pretty much anywhere.
Configuring your VMs
Each VM will be set up with the following:
- 2GB RAM
- 2 CPUs
- 2 x 40GB disks for RAID1
- 1 NIC — Bridged
In production your nodes would require much higher resources and preferably hardware RAID.
Each VM has been configured with a single Bridged Adapter which is connected to my host machine’s network interface. It just allows the VMs to get an IP from my home router and be seen as physical machines on the network — see tutorials below for more info on how to set up the VMs if you’re unsure how to do this.
There are 3 controllers, 3 workers and 1 load balancer server. The best method is to install the base version of the OS you’ll use and then clone the VM so you have the total you need for this tutorial. It will save time as you won’t need to install the OS 7 different times.
Just make sure you update the hostname using `hostnamectl set-hostname some-hostname` and the hostname in the `/etc/host` file. Once done, presuming you’re using static IPs, update the IP in `/etc/netplan/00-installer-config.yaml`
Guides to creating VMs
Operating System
I’ll be using Ubuntu Server for all of my controller and worker nodes. There is nothing Ubuntu specific about this tutorial other than apt usage for a couple of packages so feel free to use any Linux version you wish. Generally speaking you should be able to translate for your distribution without any issues.
Container networking
We will use Calico. See the other options available here
Firewall
Presuming you’re at home, it’s likely your router already allows the internal communication between nodes but for external access (if you want external access), you’ll likely need to configure some port forwarding. If you’re unsure how to do this, check out portforward.com for examples.
Some info you’ll need later
The info below might be different for you depending on your network.
Networks
Kubernetes Subnet — 192.168.0.0/24 <- This is just my home network
POD_CIDR — 10.200.0.0/16 <- This is the private network the PODS will use.
SERVICE_CIDR — 10.32.0.0/16 <- This is the private network the services will use.Machines
This PC (Lab machine): 192.168.0.2
This is the PC from which most of the commands in this guide will be run when configuring SSl/TLS, configs, running kubectl and more. We’ll copy files & configs to controllers & nodes as required later.
k8s-controllers-lb: 192.168.0.101 (aka KUBERNETES_PUBLIC_IP)
k8s-controller-0: 192.168.0.110
k8s-controller-1: 192.168.0.111
k8s-controller-2: 192.168.0.112
k8s-worker-0: 192.168.0.120
k8s-worker-1: 192.168.0.121
k8s-worker-2: 192.168.0.122If you wanted to do a single node setup, you could do everything in this guide on the one node; it would work as both a controller and worker. You would just need to “untaint” the controller node.
You could also set up one controller and one worker. The number of each is pretty much limitless.
Before moving forward, I recommend you make the following changes to /etc/hosts on your lab machine and all controllers/workers so that the scp commands later will work without issue and so that the nodes can connect to one another. Replace the IPs as required.
cat << EOF | sudo tee /etc/hosts
127.0.0.1 localhost
127.0.1.1 $(hostname -s)# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters192.168.0.101 k8s-controllers-lb
192.168.0.110 k8s-controller-0
192.168.0.111 k8s-controller-1
192.168.0.112 k8s-controller-2
192.168.0.120 k8s-worker-0
192.168.0.121 k8s-worker-1
192.168.0.122 k8s-worker-2
EOF
Installing packages
This step is nice and easy.
Create a working directory on your lab machine which will hold all the configurations and TLS certs.
All commands throughout the entire series, unless otherwise specified, should be run on the lab machine from within this directory.
mkdir ~/k8s-the-hard-way-bare-metal
cd ~/k8s-the-hard-way-bare-metalGet kubectl
This is the tool used to interact with the cluster — the snippet below will always pull version 1.23.
curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.23.0/bin/linux/amd64/kubectl
chmod +x kubectl
sudo mv kubectl /usr/local/bin/Setting up the Kubernetes Cluster
This guide was originally written for v1.23.0.
I’ll try and keep it up-to-date every year or two with the latest version for as long as I’m able however, if anything doesn’t work you may want to refer to the changelogs to confirm nothing fundamental has changed in the service configurations since the writing of this article and you using it.
You’re ready to start setting up the cluster I’ve broken down the steps into separate articles to make them a little easier to chew.
Before dropping into each section, run the following on all controller and worker nodes as well as the load balancer to install some packages the cluster will need for networking and firewall rules.
Run commands at the same time by using tmux or screen for ease.
sudo apt install conntrack socat ipset -ySetting up cfssl & generating configs
Generate kubeconfigs and encryption key
Configuring controllers
Configuring the load balancer
Configuring the workers
Configure remote access
Configuring Pod Routing
Configuring DNS
Testing the cluster
Conclusion
You should now have a fully functioning Kubernetes system that you can use either in (a pretty basic) production environment or as a single/multi node testing environment.
You’ve taken a lot in today, go make yourself a brew and absorb what you’ve done. It may take a few tear-downs and restarts but you’ll get there.
We all have to start somewhere!
