Kubernetes the hard way on bare metal/VMs — V1.12

An all encompassing series designed for beginners. From VM configuration to K8S cluster building.

Kubernetes Logo

There is an updated article for 1.23 here.

Introduction

After spending the past few months learning about Kubernetes, how to use it and how to set it up, I came to realise that whilst there are some excellent tutorials scattered around the internet, many presume a variety of situations such as:

• You’re using GCE, AWS or another cloud provider
• You have access to multiple servers
• You know all about networking
• and many other things

Now I’m not here to discredit these tutorials; of course these tutorials are very useful otherwise people wouldn’t use them and I certainly wouldn’t have learned what I have up to now.

I strongly recommend tutorials such as Kelsey Hightower’s “Kubernetes The Hard Way”. It’s an excellent tutorial that takes you through every step of setting up a Kubernetes Cluster however presumes you’re using GCE and all of the features that come with it. You can get some free credit on the system but if you are looking to set up on bare metal and don’t know much about configuring networks, firewalls or load-balancers then you may struggle with Kelsey’s guide. You can end up in a web of tutorials where you’re trying to understand a lot just to grasp one concept — Kubernetes.

I want to attempt to alleviate some of the potential confusion by creating and collating some tutorials into this one series; this guide is inspired by and stands on the shoulders of Kelsey’s work and he deserves much credit! It has been translated from his step by step tutorial into something tailored to bare metal/VM setups and aimed at beginners. This allows you to start from installing Linux and finish with a fully functioning cluster. I’ve broken this down so that more advanced Linux users can skip certain sections.
I won’t pretend to be an expert on Kubernetes, but I know how easy it is to skip to the quick deploy solutions without really understanding what is going on and how to set things up. The purpose of this guide is to help those who want to learn how to set up Kubernetes without using quick solutions such as Minikube — these don’t teach you about setting up Kubernetes but instead create a cluster for you and let you start using it.

Note: You’ll see me use terms such as Machine, PC, Laptop, Server, VM, Node, Controller and Worker in this tutorial; these simply refer to a physical machine or a VM with Linux installed on it. You don’t need fancy, costly servers or services; just free software.
Don’t let terminology confuse you!

Read the docs

I can’t stress enough how useful it is to read and understand the documentation on the main Kubernetes website to learn how to set clusters up in a variety of different ways.

Setting up the resources

I’ll be setting this up on libvirt using my home PC however you can replicate this with Virtualbox or physical servers allowing you to follow along from pretty much anywhere.

Configuring your VMs

Each VM will be set up with the following:

• 2GB RAM
• 2 CPUs
• 2 x 40GB disk for a demo of using RAID in live
• 1 NIC — Bridged

In production your nodes would require much higher resources and preferably hardware RAID.

Each VM has been configured with a single Bridged Adapter which is connected to my host machine’s network interface allowing the VMs to get an IP from my home router — see tutorials below for more info on how to set the VMs up if you’re unsure.

There are 3 controllers, 3 workers and 1 load balancer server*. The best method is to install the base version of the OS you’ll and then clone the VM so you have the total you need for this tutorial. It will save time as you won’t need to install the OS 6 different times; just make sure you update the hostnames and /etc/host files.

*The load balancer will supply its own disk so you can remove any disks from this VM.

VMs used in this tutorial.

Guides to creating VMs

Using libvirt/KVM/QEMU to create VMs.

Using Virtualbox to create VMs

Operating System

I’ll be using Ubuntu Server for all of my controller and worker nodes. There is nothing Ubuntu specific about this tutorial other than apt usage for a couple of packages so feel free to use any Linux version you wish. Generally speaking you should be able to translate for your distribution without any issues.

Installing Ubuntu 18.04 Server

Network

You will use kube-router. This is a purpose built, turnkey solution that provides high performance Kubernetes networking.
See the other options available here

Firewall

Presuming you’re at home, it’s likely your router already allows the internal communication between nodes but for external access (if you want external access), you’ll likely need to configure some port forwarding. If you’re unsure how to do this, check out portforward.com for examples.

Some info you’ll need later

The info below might be different for you depending on your network.

Networks
Kubernetes Subnet — 192.168.0.0/24 <- This is just my home network
POD_CIDR — 10.200.0.0/16 <- This is the private network the PODS will use.
SERVICE_CIDR — 10.32.0.0/16 <- This is the private network the services will use.

Machines
This PC (Lab machine): 192.168.0.2
This is the PC from which most of the commands in this guide will be run when configuring SSl/TLS, configs, running kubectl and more. We’ll copy files & configs to controllers & nodes as required later.
k8s-controller-loadbalancer-maintenance: 192.168.0.101

k8s-controllers-lb-VIP: 192.168.0.210 (aka KUBERNETES_PUBLIC_IP)
k8s-controller-0: 192.168.0.110
k8s-controller-1: 192.168.0.111
k8s-controller-2: 192.168.0.112
k8s-worker-0: 192.168.0.120
k8s-worker-1: 192.168.0.121
k8s-worker-2: 192.168.0.122

If you wanted to do a single node setup, you could do everything in this guide on the one node; it would work as both a controller and worker.
You could also set up one controller and one worker. The number of each is pretty much limitless.

Before moving forward, I recommend you make the following changes to /etc/hosts on your lab machine and all controllers/workers so that the scp commands later will work without issue and so that the nodes can connect to one another. Replace the IPs as required.

cat << EOF | sudo tee /etc/hosts
192.168.0.101 k8s-lb
192.168.0.210 k8s-controllers-lb
192.168.0.110 controller-0
192.168.0.111 controller-1
192.168.0.112 controller-2
192.168.0.120 worker-0
192.168.0.121 worker-1
192.168.0.122 worker-2
EOF

Installing packages

This step is nice and easy.

Create a working directory on your lab machine which will hold all the configurations and TLS certs.
All commands throughout the entire series, unless otherwise specified, should be run on the lab machine from within this directory.

mkdir ~/k8s-the-hard-way-bare-metal
cd ~/k8s-the-hard-way-bare-metal

Get kubectl

This is the tool used to interact with the cluster — the snippet below will always pull the latest stable version.

curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.12.0/bin/linux/amd64/kubectl
chmod +x kubectl
sudo mv kubectl /usr/local/bin/

Setting up the Kubernetes Cluster

This guide was originally written for v1.12.0 and I’m currently playing with version 1.15 but there were a number of changes made in that, v.1.13 and v.1.14 that means this guide won’t work with either of those.
I’ll try and keep it up-to-date with the latest version for as long as I’m able however if anything doesn’t work, you may want to refer to the changelogs to confirm nothing fundamental has changed in the service configurations since the writing of this article and you using it.

You’re ready to start setting up the cluster I’ve broken down the steps into separate articles to make them a little easier to chew.

Before dropping into each section, run the following on all controller and worker nodes to install some packages the cluster will need for networking and firewall rules.
Run them at the same time by using tmux or screen for ease.

sudo apt install conntrack socat -y

Setting up cfssl & generating configs

Install CloudFlare’s PKI Toolkit and generate TLS certs

Generate kubeconfigs and encryption key

Generate kubeconfig files & encryption key — Drew Viles — Medium

Configuring controllers

Setting up the controllers — Drew Viles — Medium

Configuring the load balancer

Configuring a free load balancer — Drew Viles — Medium

Configuring the workers

Setting up the workers — Drew Viles — Medium

Configure remote access

Configuring remote access — Drew Viles — Medium

Configuring DNS

Setting up DNS — Drew Viles — Medium

Testing the cluster

Testing the cluster — Drew Viles — Medium

Conclusion

You should now have a fully functioning Kubernetes system that you can use either in (a pretty basic) production environment or as a single/multi node testing environment.

You’ve taken a lot in today, go make yourself a brew and absorb what you’ve done. It may take a few tear-downs and restarts but you’ll get there.

We all have to start somewhere!