How to Bootstrap Kubernetes the hard way!

What is the Purpose of this document:

Kubernetes (sometimes called K8s for short) is becoming a leading solution for container orchestration, yet it can be a bit of pain to bootstrap and to understand basic ideas that it brings to the IT world.

This tutorial is the second in a series of blog posts that will guide someone with a good system administrator to have basic understandings of K8s. I would like to install Kubernetes on VMs using minimal tools and use the kubernetes.io instructions as a reference, this could come handy if you want to get the certificate for K8s administrator.

What is Kubernetes:

Kubernetes is a portable, extensible open-source platform for managing containerized workloads and services, it uses both declarative configuration and automation. Google open-sourced the Kubernetes project in 2014. It builds upon a decade and a half of experience that Google has with running production workloads at scale, combined with best-of-breed ideas and practices from the community.

Why everyone is talking about this?

Kubernetes provides a container-centric management environment. It orchestrates computing, networking, and storage infrastructure on behalf of user workloads.

This provides much of the simplicity of Platform as a Service (PaaS) with the flexibility of Infrastructure as a Service (IaaS) and enables portability across infrastructure providers. Kubernetes, aka K8s, is an open-source cluster manager software for deploying, running and managing containers at scale.

It lets developers focus on their applications, and not worry about the underlying infrastructure that delivers them.

Kubernetes can run in any environment, for example, on-premise bare-metal cloud, or AWS. Docker is widely used nowadays for container run time environment, but K8s can use any kind of container (rkt for example).

Why did I choose kubeadm and not Minikube or Google Kubernetes Engine?

When I started reading tutorials over the net, I saw that most of them use tools like Minikube (to run Kubernetes locally), or Google Kubernetes Engine, Both these tools offer the option to “deploy” K8s easily but using them at the start can lead to troubles when trying to understand basic K8s concepts.

The problem is that you will miss a lot of knowledge derived from the pain of failure. I wanted something else, I wanted to understand how I can bootstrap a cluster myself and with this tutorial you will deploy a cluster and will understand and configure the network, it might be a bit harder at the start, but at the end, I think it will be much more beneficial for you.

But why do we need to install Docker? Aren’t we installing Kubernetes?

K8s is an orchestration tool, but it does not provide a run-time environment for running your containers. We would still need to install Docker Daemon on each machine we use. We will pack and run an application inside a Docker container, but K8s will schedule the deployments and all the rest of the operations needs for your app.

Networking:

Some basic concepts:

Pods:

A Pod is the basic building block of Kubernetes it is the smallest and simplest unit in the Kubernetes object model that you create or deploy. A Pod represents a running process on your cluster.

A Pod encapsulates an application container (or, in some cases, multiple containers), storage resources, a unique network IP, and options that govern how the container(s) should run. All the containers in a pod can reach each other on localhost.

pods need to be able to communicate with other pods whether they are running on the same localhost or separate hosts.

Let’s imagine the following network,10.116.0.0/24, so the router is 10.126.0.1 and we have two instances that are 10.116.0.2 and 10.116.0.3 respectively. Given this setup, each instance can communicate with the other via eth0. That’s how the node network works, but what about the Pods? Kubernetes assigns an overall address space for the bridges on each node, and then assigns the bridges addresses within that space, based on the node the bridge is built on.

Kubernetes adds routing rules to the gateway at 10.116.0.1 telling it how packets destined for each bridge should be routed, which node’s eth0 the bridge can be reached through. This combination of virtual network interfaces, bridges, and routing rules is usually called a pod network.

Kubernetes uses CNI (Container Network Interface) to manage and operate the pod network, there are a couple of external modules such as Calico, Flannel, and Canal to name few. I choose Calico, it looked easy to configure and I had no problems with it. For our purpose, it’s important to understand that you will first need to configure the node network, and then the pod network.

Kubernetes imposes the following fundamental requirements on any networking implementation (barring any intentional network segmentation policies):

• all containers can communicate with all other containers without NAT
• all nodes can communicate with all containers (and vice-versa) without NAT

What this means in practice is that you can not just take two computers running Docker and expect Kubernetes to work. You must ensure that the fundamental requirements are met.

Before you proceed to take time to think about the subnet and CIDR you will use for the pod network, since we are using containers, we need to understand that a node will have their own IP over the k8s internal network. You can’t just run `curl localhost:port` when your application is running on a pod, you will need to check your k8s cluster for its internal network.

You define the pods network when you use `kubeadm init`

After you configure the internal network, you can configure the load-balancer or ingress so your application can be reached from the internet. This is out of the scope for this tutorial.

What we want to achieve:

At the end of this tutorial you will have:

• One Kubernetes master
• Two Kubernetes worker nodes
• A working node network
• A working pod network

Assumptions:

I assume prior knowledge in the following areas:

• basic knowledge in Linux operating system.
• basic knowledge in networking and subnetting and routing.
• basic knowledge in containers (mainly Docker).
• Basic knowledge in c groups and namespacing.
• basic knowledge in HTTP and usage of curl.

Before you proceed:

• please decide on the master node, make sure to label your machine adequately. The hostname should represent the role of the machine in the cluster.
• please make sure that all your nodes can ping each other (the best would be to set them up in one subnet)
• due to the fact that K8S needs a fine amount of open ports, we would suggest that you will deploy your nodes in a DMZ or a security group in AWS.

What do we want to install:

kubeadm

kubeadm performs the actions necessary to get a minimum viable cluster up and running. By design, it cares only about bootstrapping, not about provisioning machines.

kubelet

The kubelet is the primary “node agent” that runs on each node. The kubelet works in terms of a PodSpec. A PodSpec is a YAML or JSON object that describes a pod. The kubelet takes a set of PodSpecs that are provided through various mechanisms (primarily through the API server) and ensures that the containers described in those PodSpecs are running and healthy.

Kubelet doesn’t manage containers which were not created by Kubernetes

kubectl

Kubectl is a command line interface for running commands against Kubernetes clusters.

Before you begin:

you will need one or more machines running one of:

Ubuntu 16.04+

Debian 9

CentOS 7

RHEL 7

Fedora 25/26 (best-effort)

HypriotOS v1.0.1+

Container Linux (tested with 1800.6.0)

We decided to go with Ubuntu, just pick any Linux distribution that you want!

Each machine should have:

• 2 GB or more of RAM per machine (any less will leave little room for your apps)
• 2 CPUs or more
• Full network connectivity between all machines in the cluster
• Disable swapfile

Open the following ports on your master node:

TCP Inbound 6443

TCP Inbound 2379–2380

TCP Inbound 10250

TCP Inbound 10251

TCP Inbound 10252

Open the following ports on your worker nodes:

TCP Inbound 10250

TCP Inbound 30000–32767

Install CRI (Container Runtime Interface):

This would be the Daemon that controls the containers that K8s will actually run.

We decided to go with Docker, but mind you there are other options.

Install Docker CE

Set up the repository:

Update the apt package index

apt-get update

Install packages to allow apt to use a repository over HTTPS

apt-get update && apt-get install apt-transport-https ca-certificates curl software-properties-common

Add Docker’s official GPG key

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -

Add docker apt repository

add-apt-repository \
“deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable”

Install docker ce

apt-get update && apt-get install docker-ce=18.06.0~ce~3–0~ubuntu

Setup daemon

cat > /etc/docker/daemon.json <<EOF
{
“exec-opts”: [“native.cgroupdriver=systemd”],
“log-driver”: “json-file”,
“log-opts”: {
“max-size”: “100m”
},
“storage-driver”: “overlay2”
}
EOF
mkdir -p /etc/systemd/system/docker.service.d
Update and restart docker
systemctl daemon-reload
systemctl restart docker

Verify the MAC address and product_uuid are unique for every node

run the following on each machine:

ip a
sudo cat /sys/class/dmi/id/product_uuid

Install the required software:

Install curl

apt-get update && apt-get install -y apt-transport-https curl

Add apt-key for google repo

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -

add the correct info

cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
Install the software
apt-get update
apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl

You can install the latest version of Docker CE for the purpose of this tutorial, kubeadm will send an error message, but installation will work!

Init the cluster, by running on the master node the following command:

kubeadm init — pod-network-cidr 10.101.0.0/16

You can choose any network you subnet you want, make sure that the CIDR and subnet make sense. Remember that this is how you set up the pod network.

To start using your cluster, you need to run the following as a regular user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Copy the command with the token, you will need it to connect the nodes to the cluster for example:

kubeadm join <master node ip>:6443 — token 16z6a5.z6csgg27mbiaf9sy — discovery-token-ca-cert-hash sha256:9f242479a67996d8307a1adfe19a620bd54f24de21473a88ab733f6c13307who am I5b0

run the command on the worker nodes

and check out the output

kubectl get nodes
NAME STATUS ROLES AGE VERSION
worker-1 NotReady <none> 15m v1.13.2
worker-2 NotReady <none> 15m v1.13.2
master-node Ready master 38m v1.13.2

if all your nodes could connect — rejoice! you have now a running k8s cluster! you can put a small sticker on your laptop.

Notice that both the worker nodes are in “NotReady”, this means that you did not configure an internal network yet. which is fine.

if you can’t connect — check your firewall config or security group if you work with AWS.

Let’s configure the internal network:

There are many options for networking but we choose Calico, as defined in their site: “Calico enables networking and network policy in Kubernetes clusters across the cloud. Calico works everywhere — on all major public cloud providers and private cloud as well”.

curl \
https://docs.projectcalico.org/v3.4/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml \ -O

open the calico.yaml with a text editor, and change the following in the calico.yaml:

POD_CIDR=”<your-pod-cidr>”

In the ConfigMap named calico-config, locate the typha_service_name, delete the none value, and replace it with calico-typha.

kubectl apply -f calico.yaml
kubectl get nodes
NAME STATUS ROLES AGE VERSION
worker-1 Ready <none> 19m v1.13.2
worker-2 Ready <none> 18m v1.13.2
master-node Ready master 42m v1.13.2

All the nodes should be ready!

You have configured the internal network for K8S! you can now rejoice again

You can put another bigger K8S sticker on your laptop!

Some links for reference:

Polar Squad

Calico

Kubernetes

Docker