Kubernetes Home Dev Lab

(Setting up K8s on Ubuntu for training)

8 min readNov 29, 2022

Kubernetes is the fastest growing container orchestration technology

(November 2022)

Do you want to set up Kubernetes at home for training or dev purposes ? Do you like Ubuntu for your home servers? Read this before you jump in to install it.

I, Alex McLintock, have been using container systems for over ten years, but not Kubernetes. The one I am most familiar with, YARN, is Java based, predates Docker and Kubernetes (k8s), and was developed as part of the Hadoop Big Data system. People didn’t like YARN because it only really did Java and did not have all the benefits of Docker. (Some people try to get it to launch python and other languages, and while it *can* I don’t believe it is very good at it).

Because of my background in Big Data I have a fair bit of computer hardware lying around. This makes building a home development laboratory quite reasonable for me. Other people may wish to learn k8s by running it in the cloud. That can make sense but is not what this article is about.

Part of my motivation to do this was that I wanted to learn about Stackable (https://stackable.tech/en/). Stackable provide easy to use packaged versions of many Open Source systems, but that can wait for future articles.

∘ Operating System Choice ?
∘ Install Guides?
· Hardware
∘ Hardware — Compute?
∘ Hardware — Networking?
∘ Hardware — Storage?
· Software
∘ Software — k8s version?
∘ Software — Pod Networking?
∘ Software — Admin ?
∘ Feedback ?
∘ Credits

Operating System Choice ?

This might not be the right way round but for me my home dev machines are either Ubuntu or MacOSX (on my laptop). I had recently upgraded to Ubuntu 22.04 so that is what I aimed for. You might find that 20.04 has a much older set of software. This typically means fewer bleeding edge features, but better tested software.

I seem to be hitting a few bugs with 22.04 which might be fixed by the time you read this, or might not even affect you.

If all you have are Windows boxes then I wish you luck. I hear that it is possible to run k8s on Windows, but it seems like trying to fight with one hand tied behind your back. I don’t recommend it.

You may be a CentOS/Fedora/RedHat fan, and I quite understand wanting that as a more supportable linux for production environments. It is your choice, but I picked Ubuntu for this article.

Install Guides?

I found a few articles which tell you how to install Kubernetes on Ubuntu. Originally I found articles talking about Ubuntu version 20.04, but you are better off with someone who has actually run the commands on the same version of the OS you are working with. I suggest that you read at least a couple of them before proceeding with the install.

This blog article of mine won’t try to replicate those articles — use them for figuring out the commands.

The above articles make some high level architecture assumptions which I hope to make clear for you so that you can make the best choices when following them.

Don’t just charge in to follow them though. I have some tips below.

Hardware

This section discusses some of the hardware considerations you need to think about before embarking on building your own Kubernetes cluster.

Hardware — Compute?

So you can run K8s on a single machine, and that may work for some, but in my opinion that isn’t much better than simply running Docker on one machine. In order to test out the distributed nature of k8s I felt it was important to have more than one worker node. I expect to be spinning up many pods running many applications and did not want to be stressing out one machine doing it. Luckily I had a couple of 32Gb i5 machines running Ubuntu which could be used for this.

Kubernetes also has the concept of master nodes which help coordinate the cluster and interact with the administrators. I decided to make my main desktop the k8s master node since it was powerful enough. I haven’t decided yet whether to also make it a worker node. I don’t want to slow it down too much when I am sitting in front of it!

Hardware — Networking?

Luckily I had a spare Gigabit switch for this. I tried to use an HP managed switch but frankly it was far too noisy for a home office so gave up. I may be buying a decent fanless switch at some point.

Anyway, if you have a few spare computers then you probably have spare networking for them too.

Hardware — Storage?

All three of my machines use small fast SSD disks for the OS, but only the master node had any decent sized HDD disks. I normally use a FreeNAS box for NFS based storage. For a K8s home lab you are going to probably provide two forms of storage:

Local Persistent Volumes — for things which don’t need to store things long term. These will be the normal disks in the worker nodes.
Remote Persistent Volumes — such as an NFS server, iSCSI server, S3 clone, or something else. None of the articles I have read so far considered this aspect.

(The FreeNAS project got renamed to TrueNAS, so if you are starting out building a new NFS server then that is what you may wish to search for).

If on the other hand your worker nodes have big disks it is possible to run an NFS server on one of them as a k8s pod. I don’t like this option, but it may work for you.

Depending on your internet connection you might wish to actually use Amazon AWS S3 as a storage mechanism. This is outside of the scope of this blog.

Software

This section looks at some of the software choices you need to make. I assume you have already chosen Ubuntu 22.04 for the host Operating System.

Software — k8s version?

If, like me, you are trying to install onto version 22.04 Ubuntu (the latest Long Term Support (LTS)at time of writing), then you will want to have this repository enabled.

deb http://apt.kubernetes.io/ kubernetes-xenial main

You are right to spot that this says it is for xenial — ie the previous LTS version of Ubuntu. In fact when I upgraded the upgrade process actually commented this out thinking that xenial had to be wrong. In fact as of Nov 2022 there is no jammy version (22.04) and this xenial one seems to work fine.

What it provides in Kubernetes version 1.25 This is important to know because there seems to have been quite a few changes between version 1.23 and 1.24 so if you have an older version of k8s from before version 1.24 you may have trouble upgrading it. What I did was to delete my old cluster and start an install from scratch.

Having said all that I have had more than one person recommend Docker Desktop to me. It is what we used to use at a company I worked at, before they changed the licence to add in a fee for such use. So you might wish to consider that — particularly if you have a Ma but check the licence terms.

Software — Pod Networking?

If you are new to k8s then you might wonder what this section is about. In essence the pods (which are the basic unit of execution — a running container image) need networking to talk to each other and there are different ways of doing this through an addon or plugin technique. You need to add some software to k8s to do this for you. The installation guides listed above assume that you want either Flannel or Calico, but there are loads more (see URL below). In my case I tried Flannel, but hit problems and moved to Calico. Using Flannel creates various virtual networking devices which I didn’t like and which might be left around if you try it.

Some useful URLS:

The bottom line is that you need to make a choice before starting your install, and only change if absolutely necessary.

Software — Admin ?

You are almost certainly going to be using kubeadm to actually create your k8s cluster, and the instructions for that exist in the pages linked to near the start of this blog. This is typically done as root since it requires elevated access rights. However you will be configuring one user to be the k8s administrator who will be able to use the following software:

You will probably be using kubectl to make changes and look at your cluster. The admin user gets given config which allows it to talk to the cluster without becoming root. Even if you use a graphical admin tool you will want to be familiar with kubectl as a fall back option.

Using Kubectl can get a bit slow at times as you issue one command to get a list of something and then cut and paste ids to get information about one of them. There are a number of tools to make this process easier. The ones I have seen recommended include

K9s — a text terminal based interface
https://k8slens.dev/desktop.html K8s Lens — a full blown admin tool which is free for individuals (eg someone training) but needs a commercial licence for anything else. I am told the free version may not stay around forever.

K9s is I think the most popular, but I have not found a recent version for Ubuntu, A quick Google came up with adverts for two more that I was not aware of, so you have choices here. I don’t think you have to decide now as the admin tools can be swapped in and out as necessary. They all talk to the admin API and require no changes to your cluster.

I am told that Lens is slow for big clusters, but that isn’t going to matter for a home training cluster.

https://geekflare.com/kubernetes-management-software/

I am almost certain that you will also be using Helm which the above article describes as “the package management for Kubernetes”. It is optional, but my tip is to start learning about it now.

Feedback ?

I hope you have found this useful. I don’t claim to know everything so feedback is welcome. Your best bet would be to get in touch via LinkedIn

Credits

Image taken from the K8s community at https://github.com/kubernetes/community/blob/master/icons/README.md

The Kubernetes logo is a registered trademark of The Linux Foundation.

All text by Alex McLintock