Setting up a Kubernetes Cluster

10 min readJan 3, 2023

Kubernetes is a difficult piece of software. Even setting up the initial cluster to learn Kubernetes involves extensive debugging. To help you get started and past the setup process, I wrote a primer on getting a simple cluster up and running. This will help in getting you started on your Kubernetes journey.

Prerequisites

An intermediate understanding of AWS — Kubernetes is challenging in its own right. If you don't have intermediate knowledge of AWS IaaS solutions, parts of this may become difficult to follow.
An intermediate understanding of Linux and Bash— The cluster we'll deploy to a Ubuntu instance running in AWS. Kubernetes running on Linux leverages Linux's kernel and network.
Debugger's Mindset — Not only is Kubernetes complex, but it's constantly evolving. It's possible that the next version of Kubernetes will change this setup process, and you'll need to do extensive troubleshooting. This, for better or worse, is normal when working with Kubernetes.

Part 1: Setting Up Your AWS environment

We're going to spin up three instances to seed our cluster. One instance will be the Kubernetes control plane. The other two will be worker nodes. Of course, you can add more worker nodes if needed.

But before we get into the compute, let's first set up the Security Group and Key Pair that we'll use for our instances.

Here are the rules that we're going to use for the MyKubernetesClusterSG Security Group:

**Please note:** you must create the Security Group before referencing it as the Source for the last rule.

For the Source on the last rule, sg-098…. is MyKubernetesClusterSG. You must create the Security Group before making the last rule. I've given all resources in the Security Group full access to each other. You'll want to be more cautious when deploying into a production environment.

For the Key Pair, I named mine Kubernetes-KP:

Next, we're going to set up the Control Plane with the latest Ubuntu AMI (ami-0574da719dca65348):

Use the t2.medium instance type and the Key Pair that we've created:

Use a Public Subnet that has a route to an Internet Gateway (I'm just using a subnet in my default VPC):

Your instance should look like the one below. So go ahead and launch it:

Launch two more instances of identical instances, except they're named Kubernetes-Worker-1 and Kubernetes-Worker-2:

Please take note of your Private IP addresses for the following sections.

Part 2: Setting up the Control Plane

After we SSH into the Control Plane instance, let's change the hostname to k8s-control to make it easy to see which instance we're currently using, and map the hostnames will be creating to the IP addresses of the instances we just spun up:

#become root
sudo su

#set hostname
hostnamectl set-hostname k8s-control

#Maps hostnames to IP addresses
cat << EOF >> /etc/hosts
<insert control plane ip> k8s-control
<insert worker 1 ip> k8s-worker1
<insert worker 2 ip> k8s-worker2
EOF

#exit root
exit

#exit the server and then sign back in
exit

When you connect to the instance again, your shell will look like so:

These lines will map the specified IP addresses to the corresponding hostnames. For example, after this code is run, if you try to access k8s-control your computer, it will resolve to the server's IP address.

Next, make sure that your cluster is up to date and apt-transport-https, curl, vim, and gitare installed:

#update server and install curl vim wget
sudo apt-get update -y
sudo apt install -y apt-transport-https curl vim git

Then we're on to containerd. The containerd runtime is a daemon that manages the lifecycle of containers on a host. It is designed to be lightweight and efficient and is often used as an alternative to the Docker daemon (dockerd) for container orchestration. These commands install and configure the containerd container runtime on an Ubuntu system:

#Install containerd 
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update -y
sudo apt-get install -y containerd.io

#Configure containerd
sudo mkdir -p /etc/containerd
sudo containerd config default | sudo tee /etc/containerd/config.toml

The bash script does the following:

Creates a directory at /etc/apt/keyrings to store the Docker GPG key.
Downloads the Docker GPG key and stores it in the /etc/apt/keyrings directory using the curl and gpg commands.
Adds the Docker APT repository to the system's sources.list file using the echo and tee commands.
Updates the package cache using. apt-get update.
Installs the containerd.io package using apt-get install.
Creates a directory at /etc/containerd to store the containerd configuration file.
Generates a default containerd configuration file using the containerd config default command and stores it in the /etc/containerd directory using the tee command.

Now we need to set the cgroup driver for runc to systems, which is required for the kubelet:

#set SystemdCgroup = true within config.toml
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml

#Restart containerd daemon
sudo systemctl restart containerd

#Enable containerd to start automatically at boot time
sudo systemctl enable containerd

The SystemdCgroup configuration option determines whether containerd uses the Systemd cgroup hierarchy when creating and managing containers. When set to true, containerd will use the Systemd cgroup hierarchy. When set to false, it will use its cgroup hierarchy. By running this command, you are modifying the /etc/containerd/config.toml file to set the SystemdCgroup option to true, which will cause containerd to use the Systemd cgroup hierarchy when creating and managing containers.

Next up, we're going to install kubeadm, kubectl, and kubelet:

#install kubeadm, kubectl, kubelet,and kubernetes-cni 
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add
sudo apt-add-repository -y "deb http://apt.kubernetes.io/ kubernetes-xenial main"
sudo apt -y install kubeadm kubelet kubectl kubernetes-cni

The curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - command retrieves the GPG key for the Kubernetes package repository and adds it to the system's keyring. This is necessary to verify the authenticity of the packages that are being installed.

apt-add-repository is used to add a repository to your system's list of sources for packages. This kubernetes-xenialrepository contains packages for running Kubernetes on Ubuntu Xenial, including the Kubernetes server and command-line tools.

The apt -y install kubelet kubeadm kubectl kubernetes-cni command installs the essential packages for Kubernetes:

The kubelet is a component of a Kubernetes cluster that runs on each node in the cluster. It is responsible for maintaining the state of the pods on the node and ensuring that the containers in the pods are running.
The kubeadm is a tool for bootstrapping a cluster. It is used to create and configure the components of a cluster, such as the API server, etcd, and the control plane nodes.
The kubectl is a command-line tool for interacting with a Kubernetes cluster. It is used to manage the resources in the cluster, such as deploying applications, scaling resources, and viewing the status of the cluster.
The kubernetes-cni is a package that provides the Container Network Interface (CNI) plugins for Kubernetes. The CNI plugins provide networking services for the containers in a Kubernetes cluster.

Next, we need to set kernel parameters in preparation for running Kubernetes:

#disable swap
sudo swapoff -a

#check if a swap entry exists and remove it if it does
sudo vim /etc/fstab

#Load the br_netfilter module in the Linux kernel
sudo modprobe br_netfilter

#switch to root to 
sudo su 
echo 1 > /proc/sys/net/ipv4/ip_forward
exit

swapoff -a disables swap on all available swap devices. Swap is a space on a device (usually a disk) that stores data that would otherwise be stored in RAM. When the system runs out of available RAM, it can use swap space to continue running. However, swap can be slower than RAM, so it is often disabled in a Kubernetes cluster to improve performance and stability.

The sudo modprobe br_netfilter command loads the br_netfilter kernel module into the Linux kernel. The br_netfilter the module enables netfilter packet filtering in the network bridge, allowing the kernel to filter traffic passing through the bridge based on various criteria such as IP address, protocol, or port number.

The echo 1 > /proc/sys/net/ipv4/ip_forward command enables IP forwarding on Linux. Kubernetes uses iptables to implement network policies, and IP forwarding is required for traffic to flow between pods on different nodes.

Next, we're going to initialize the Kubernetes cluster:

#initialize kubernetes cluster
sudo kubeadm init --pod-network-cidr=10.244.0.0/16

When you run kubeadm init, it will perform the following tasks:

Set up the necessary components and infrastructure to run a cluster, including the etcd distributed key-value store, the Kubernetes API server, and the kubelet process.
Initialize the cluster's etcd database with the necessary configuration and state data.
Set up a secure connection between the API server and etcd.
Set up a secure connection between the API server and the kubelets.
Install the necessary networking components to allow pods to communicate with each other and with services.
Create a default configuration file for the kubectl command-line tool, which manages the cluster.

The --pod-network-cidr option specifies the range of IP addresses used for pod IPs in the cluster. The pod-network-cidr value must not overlap with any existing network ranges and must be large enough to accommodate the number of pods you expect to run in the cluster.

As part of the command's output, you'll get a series of commands to start your cluster. They will look like so:

#Allow kubectl to interact with the cluster
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

#export KUBECONFIG as root
sudo su
export KUBECONFIG=/etc/kubernetes/admin.conf
exit

This script does the following:

Creates a new directory at $HOME/.kube if it does not already exist.
Copies the admin.conf file from /etc/kubernetes/ to $HOME/.kube/config. The admin.conf file contains the configuration for a Kubernetes cluster, including the API server's address and the credentials needed to authenticate with the API server.
Changes the owner of the $HOME/.kube/config file to the current user ($(id -u)) and group ($(id -g)).
Exports the KUBECONFIG environment variable, which tells kubectl (the command-line tool for interacting with a Kubernetes cluster) where to find the configuration file.

You will also get a statement like the one below:

#Command to run on worker nodes
kubeadm join <control-plane-ip>:6443 --token <token> \
 --discovery-token-ca-cert-hash <hash>

You should SAVE THIS JOIN COMMAND to run on worker nodes later.

Finally, we need to set up the network for Kubernetes:

#Install CNI Flannel
kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/v0.20.2/Documentation/kube-flannel.yml

kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/v0.20.2/Documentation/kube-flannel.yml installs the CNI (Container Network Interface) Flannel plugin for Kubernetes. Flannel is a CNI plugin that provides a simple, secure, and scalable network fabric for pods in a cluster. The kubectl apply command is used to apply the configuration specified in the kube-flannel.yml file, which will install Flannel in the cluster.

After kubeadm init completes, you will have a working Kubernetes cluster ready to run applications. You can then use kubectl it to deploy and manage applications on the cluster., you should see the Control Plane node when you run kubectl get nodes:

Now it's time to set up the Worker Nodes.

Part 3: Building the Worker

Like with the Control Plane, the first step is to set the hostnames. First, map the hostnames to IP addresses:

#become root
sudo su

#set hostname
hostnamectl set-hostname k8s-worker<number of worker node>

#Maps hostnames to IP addresses
cat << EOF >> /etc/hosts
<insert control plane ip> k8s-control
<insert worker 1 ip> k8s-worker1
<insert worker 2 ip> k8s-worker2
EOF

#exit root
exit

#exit the server and then sign back in
exit

Most of the setup process for Worker Nodes has already been covered with the Control Plane section, so I'm providing the script you can quickly run through. For this first part, you can copy and paste into the terminal (after you've filled in the IP addresses for the Control Plane and Worker nodes):


#update server and install apt-transport-https and curl
sudo apt-get update -y
sudo apt install -y apt-transport-https curl

#Install containerd 
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update -y
sudo apt-get install -y containerd.io

#Configure containerd
sudo mkdir -p /etc/containerd
sudo containerd config default | sudo tee /etc/containerd/config.toml

#set SystemdCgroup = true within configs.toml
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml

#Restart containerd daemon
sudo systemctl restart containerd

#Enable containerd to start automatically at boot time
sudo systemctl enable containerd

#install kubeadm, kubectl, kubelet,and kubernetes-cni 
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add
sudo apt-add-repository -y "deb http://apt.kubernetes.io/ kubernetes-xenial main"
sudo apt install -y kubeadm kubelet kubectl kubernetes-cni

#disable swap
sudo swapoff -a

#load the br_netfilter module in the Linux kernel
sudo modprobe br_netfilter

In the second section, you'll need to have your hands on the keyboard to use vim and enter/exit root:

#check if a swap entry exists and remove it if it does
sudo vim /etc/fstab

#enable ip-forwarding 
sudo su 
echo 1 > /proc/sys/net/ipv4/ip_forward
exit

Finally, you'll want to run the command that we saved when running Kubernetes on the Control Plane:

#Command to run on worker nodes
kubeadm join <control-plane-ip>:6443 --token <token> \
 --discovery-token-ca-cert-hash <hash>

If the joins have been done correctly on both worker nodes, you'll see the following:

Wrap Up

As I mentioned in the beginning, Kubernetes is constantly evolving. So if you're encountering any issues, let me know and the comments, and I'll do my best to update this.

Thanks for reading!