Multi Node Kubernetes Cluster with Vagrant, VirtualBox and Kubeadm

TL;DR, Go grab my Vagrantfile at https://github.com/ecomm-integration-ballerina/kubernetes-cluster and run vagrant up to bring a 3 node Kubernetes cluster up and running in VirtualBox.

I have been trying to setup a multi node Kubernetes cluster to try out Ballerina integrations at scale in containers.

There are many third party tools to setup a Kubernetes cluster. I’ve looked at two options, Minikube and Kubeadm. Minikube doesn’t support multi node setup, so I picked Kubeadm. Kubeadm is still in beta, do not use it in production.

I followed Kubeadm guide and setup a three node cluster (single master and two nodes) manually first and then automated in Vagrant because I can recreate the cluster easily later with no effort.

So here is how I did it. Try it, create issues in my repo and send pull requests.

Note: Step-1 to Step-8 should be done in all the Kubernetes nodes (master + nodes). Step-9 to Step-12 should be done in Kubernetes master. Step-13 should be done in Kubernetes nodes.

Step-1: Pick a base box for Kubernetes nodes

config.vm.box = "ubuntu/xenial64"
config.vm.box_version = "20180831.0.0"

Step-2: Setup a network

config.vm.network :private_network, ip: "192.168.205.10"

Step-3: Setup memory and CPUs

v.customize ["modifyvm", :id, "--memory", "2048"]
v.customize ["modifyvm", :id, "--cpus", "2"]

Step-4: Install Docker

I’m not going to use vagrant docker provisioning because it always installs the latest version, but Kubeadm requires 17.03 or older. So I’m going to use shell provisioning to install docker.

apt-get update
apt-get install -y apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
add-apt-repository "deb https://download.docker.com/linux/$(. /etc/os-release; echo "$ID") $(lsb_release -cs) stable"
apt-get update && apt-get install -y docker-ce=$(apt-cache madison docker-ce | grep 17.03 | head -1 | awk '{print $3}')

Step-5: Allow regular user to run docker commands

I’m adding the regular user (vagrant) to the docker user group so that I can run docker commands as vagrant user without sudo privileges.

usermod -aG docker vagrant

Step-6: Install Kubeadm, Kubelet and Kubctl

I’m using shell provisioning to install Kubeadm, Kubelet and Kubectl from the repositories.

    apt-get install -y apt-transport-https curl
    curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
    cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
    deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
    apt-get update
    apt-get install -y kubelet kubeadm kubectl
    apt-mark hold kubelet kubeadm kubectl

Step-7: Kubelet requires swap to be disabled

Running following bash command will disable the swap in the current login session.

swapoff -a

Step-8: Disable swap permanently

Following change will keep swap off after reboot too

sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

Step-9: Initialize Kubernetes master

Following Kubeadm command will install and configure a Kubernetes master.

# ip of this box
IP_ADDR=`ifconfig enp0s8 | grep Mask | awk '{print $2}'| cut -f2 -d:`

# install k8s master
HOST_NAME=$(hostname -s)
kubeadm init --apiserver-advertise-address=$IP_ADDR --apiserver-cert-extra-sans=$IP_ADDR  --node-name $HOST_NAME --pod-network-cidr=192.168.0.0/16

Step-10: Allow regular user to run Kubectl commands

I’m going to copy the Kubectl credentials to vagrant user’s home directory to be able to run Kubectl commands without sudo privileges.

#copying credentials to regular user - vagrant
sudo --user=vagrant mkdir -p /home/vagrant/.kube
cp -i /etc/kubernetes/admin.conf /home/vagrant/.kube/config
chown $(id -u vagrant):$(id -g vagrant) /home/vagrant/.kube/config

Step-11: Install a Pod Network

We need to install a Pod Network to enable pod-to-pod communication in our Kubernetes cluster. There are many third party Pod Network projects such as Calico, Flannel and Weave. I’ve tried to use Weave, but looks like there are some bugs that prevent a Weave pod to be scheduled on Kubenetes nodes. Then I’ve tried Calico and it works fine so far.

# install Calico pod network addon
export KUBECONFIG=/etc/kubernetes/admin.conf
kubectl apply -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml
kubectl apply -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml

Step-12: Save the Kubeadm join commands

Our Kubernetes master should be up and running by now. We can join Kubernetes nodes to master using Kubeadm join command. You can run kubeadm token create — print-join-command in Kubernetes master to get the join command that should be executed in Kubernetes nodes.

I’m going to store the output of this command in a file in Kubernetes master. Later I’m going to download this file from inside Kubernetes nodes, execute and join the cluster.

kubeadm token create --print-join-command >> /etc/kubeadm_join_cmd.sh
chmod +x /etc/kubeadm_join_cmd.sh

Step-13: Execute Kubeadm join command in Kubernetes nodes

What I’m doing here is to scp the above kubeadm_join_cmd.sh from master to nodes and execute it. I’m using sshpass to run prompt-less scp command in bash.

apt-get install -y sshpass
sshpass -p "vagrant" scp -o StrictHostKeyChecking=no vagrant@192.168.205.10:/etc/kubeadm_join_cmd.sh .
sh ./kubeadm_join_cmd.sh

That’s all, your cluster should be up and running now.

Putting it all together

Here is my full Vagrantfile gist. But be sure check my repo to get the up-to-date Vagrant file. Run vagrant up to bring up a 3 node Kubernetes cluster.

Follow me on Twitter @raj10x for latest blogs!

Raj Rajaratnam