Create a Kubernetes Cluster using Virtualbox — The Hard Way

Hi everyone! Have you ever thought about having a production like Kubernetes cluster privately so that you can play with Kubernetes with no fear and run various tests at ease?

To achieve this goal, we can go for following approaches:

1. Run Kubernetes cluster on local machine using mini-kube
2. Run Kubernetes cluster on one of the cloud service like GoogleCloud, AWS etc.
3. Run Kubernetes cluster on your local machine using Virtualbox and virtual machines.

The first solution has its own limitations. The major limitation of Minikube is that the local cluster has only one node, and therefore is not a great simulation of a production-level, multi-node Kubernetes cluster.

The second solution works, yet it is costly. As Learning Kubernetes and performing experiments on it is a time consuming process, non of the cloud services free trial would be enough and at the end it gonna charge us:)

The third solution is more complex that other two. Yet it provides us with significant flexibility. It is very like production level kubernetes clusters. We are able to add nodes, drain nodes, join master nodes, work with master’s etcd and use/experiment our desired Kubernetes network plugins. The other important benefit of setting up Kubernetes cluster with this approach is that as we have to setup everything from the scratch, we understand what is happening under-hood in a better way. Lets go for it then!

Setting up Kubernetes cluster in action

In the nutshell, we are going to implement the following architecture for setting up our Kubernetes cluster.

Figure 1-Cluster architecture

There are some key-points apparent in the figure 1.

• We start off by 1 master node
• we have 3 worker nodes
• Each VM has 2 network interfaces.
• One network interface is attached to default NAT network in virtualbox. the purpose of this network is to provide the node internet access.
• Another network interface is attached to a Host-Only network in virtualbox. All inside cluster communications would be through this network.

Prepare VirtualBox machines for deploying Kubernetes cluster

According to figure-1, We need a private network that nodes can communicate with each other though. To create one go to File -> Host Network Manager. Create new network there with no DHCP Enable option. Remember the network range created! we gonna use it soon. Mine is 192.168.60.1/24 .

VM’s OS and configuration:

I assume you are familiar with creating new virtual machine in virtual box. if not, no worries! just Do google it :). We gonna use following setup as of our VMs:
Master(s):
— Name: master# (number of master in the cluster)
— OS: ubuntu-server-18.04
— CPU: 2core
— Memory: 2GB
Workers:
— OS: ubuntu-server-18.04
— CPU: 1core
— Memory: 1GB

Please note each of the VMs should benefit from 2 networks. One default NAT network which provides internet access to our node, Another attached to Host Only Network created in previous step.

After setting up the VM virtual hardware, it is time to install our operating system. There is no trick on it, except we have to assign our desired IP address in the range provided by Host Only adapter manually. In my case, I will assign 192.168.60.11 to my first Master node. We will extend this lab in the next tutorials so lets reserve some IPs from 192.168.60.11 to 192.168.60.20. Consequently, worker01 node will be 192.168.60.21 and so on.

Note: You should not set gateway 192.168.60.1 for any Node, Or you will have to change the default route (The route we need for Internet connection) to the NAT interface after OS install.

Note: Among All packages the installer ask you about, only select OpenSSH package to be installed.

Having OS installed and VM up and running, you should be able to ping the IP address of the VM.

We repeat the same steps for the workers too. At the end, the final node should represent the following table

After finishing the setup process, you should be able to ping all the targets from your host machine and from inside each of the nodes.

Setup Kubernetes cluster using kubeadm tool

So far we prepared the required infrastructure to deploy our Kubernetes cluster. Lets begin setting up the cluster :). There are several ways to setup Kubernetes cluster. We are going to leverage `kubeadm` tool. kubeadm enable us to create Kubernetes cluster that will pass Kubernetes Conformance tests. kubeadm also support providing tokens, cluster upgrade etc.

Install required components

Each of the nodes need the following components available:

• kubelet: The kubelet is the primary “node agent” that runs on each node. It can register the node with the apiserver using one of: the hostname; a flag to override the hostname; or specific logic for a cloud provider.
• kubectl: The Kubernetes command-line tool, kubectl, allows you to run commands against Kubernetes clusters. You can use kubectl to deploy applications, inspect and manage cluster resources, and view logs.
• kubeadm: Using kubeadm, you can create a minimum viable Kubernetes cluster that conforms to best practices.

Step 1- Installing required components

On each nodes, add the kubernetes repository

sudo apt update && sudo apt install -y curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add
sudo apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main"

Having required repository added, install required components. At the time, the latest version of kubernets is 1.25.0. Yet as for more stability, we are going to use kubernetes version 1.22.10 for our cluster.
First, lets list what versions are available:

apt list -a kubeadm

sudo apt install -y kubeadm=1.22.10-00 kubelet=1.22.10-00 kubectl=1.22.10-00

Note: In case you copy the command above, please note the “-” inside the code. It can be copied as “ — ” which fail the command above.

It is recommended to hold the version on installed packages. We really don’t want to mess up with kubernetes auto updates :)

sudo apt-mark hold kubeadm kubelet kubectl

Check the version of installed components and it should be as version we just installed

kubead version

Next step is to install Docker on the nodes. The official document on installing docker on Ubunut can be found here. Yer for the sake of simplicity, we install it using a simple command as follow:

sudo apt install -y docker.io

You can verify installation using

docker version

Docker will not start on system startup. meanwhile it is a requisite for kubernetes cluster to be functional. Do Not forget to enable its service to make sure docker will get up and work on each node restart

sudo systemctl enable docker

Changing Docker Cgroup Driver

It is crucial to change docker cgroup Driver after install or you get error regarding “Docker group driver detected cgroupfs instead systemd” on cluster initialization. It is easy, and should be done on all nodes (master and workers).

sudo cat <<EOF | sudo tee /etc/docker/daemon.json
{ "exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts":
{ "max-size": "100m" },
"storage-driver": "overlay2"
}
EOF

Restart Docker and make sure it is up and running

sudo systemctl restart docker

You can check if the Cgroup driver is changes to systemd

sudo docker info | grep -i cgroup

Disable swap and enable ip_forward

Kubernetes cluster needs the to disabled swap for the sake of security issues and performance. To disable swap permanently on all node, run the following commands:

sudo swapoff -a

Now to disable swap permanently, open /etc/fstab and comment line containing swap by adding # in the beginning.

Now we need to enable IP_Forward in kernel

sudo modprobe overlay
sudo modprobe br_netfilter
sudo tee /etc/sysctl.d/kubernetes.conf<<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
sudo sysctl --system

Step 2- Initializing Kubernetes Master Node

Having are required components installed and configured, We shall initialize the master node. Please note that in our lab and for now, our control plane consists of only one node (master01). All the required components for control plane including etcd will be installed on master node as a docker container.

Please note that from the first place, the secondary private network was added for internal kubernetes components communications. Therefore we shall bound the api-adrvertisement-address to this network only.

sudo kubeadm init --apiserver-advertise-address 192.168.60.11 --control-plane-endpoint 192.168.60.11

it takes some time to download all the required docker images and initialize it depends on you Internet connection speed.

When the master node initialization process done, the commands to run on workers and probable other masters to join the cluster will be provided by kubeadm

To connect to our cluster using kubectl, we need to apply following changes on the remote machine.

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

lets setup kubectl installed on master machine and then run kubectl get node command to see the output.

As you can see the master node is already added to the cluster.

Copy and paste the command to join workers to the cluster somewhere. We gonna need it to join workers to our clusters

Note: All the previous steps should be applied to workers as well, except that workers dont need kubectl at all.

Having all the above steps done for workers, use the command you copied and join the workers to the cluster. After joining the workers to the cluster, the output of kubectl get node should show new joined nodes as worker.

Step 3- Installing Kubernetes Network

You probably noticed in kubectl get node command output that all the nodes are in “NotReady” state. The reason is we still did not create any Kubernetes network for the cluster. This network is different from our infrastructure network, and is responsible to create an internal network for all inside-cluster components so that they can communicate with each other through it.

We are going to use Flannel, one of most straight forward networks for kubernetes. There are other network-add on with more feature and capabilities. Yet Flannel serve us best and also dosent add up any unnecessary complexity.

To install Flannel network add-on:

1. kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml

It takes some time to download all required docker images and bring the containers up.

Congratulations! The Kubernetes cluster is up and ready to use.

Final Words:
I started to write about what I get during last 3 years working in DevOps in a series. We will go through most of experiences I get by doing. For that purpose, we need this cluster. I started to explain it in hard-way, so we can get what exactly is happening under hood. Because performing all the steps every time manually is a time consuming process, I provided an automated way to deploy the exact same cluster, which will take few minutes based on your Internet connection speed. We will use this automated way to do our experiments on Kubernetes until we start working with cloud based Kubernetes engines such as GKE.
Please provide me feed backs and ask questions to encourage me.