Crafting a Complete Kubernetes Home Lab: Building a K3s Cluster on Raspberry Pis

Part 1: Initial Setup and Deployment of a K3s Cluster on Raspberry Pis

6 min readMay 6, 2024

Introduction

Welcome to the first article of this series. This project was inspired by my experience with cloud infrastructure, aiming to recreate a similar setup of services such as Vault, Cert-Manager, External DNS, Nginx Ingress controllers, Argo CD, and more using bare-metal technology in a home environment. By leveraging the compact power of Raspberry Pis and ARM architecture, we will set up a Kubernetes environment capable of running similar services to those found in cloud systems. Initially, I was unsure if these small devices could handle the extensive services I planned to deploy, but they have proven to run smoothly. This series will cover different features, from hardware selection and initial setup to deployments involving Terraform, ArgoCD, CI/CD pipelines, Hashicorp Vault, OIDC authentication, and more. The cluster is set up purely for personal exploration, allowing me to deploy services I want to experiment with, such as CNCF graduated and incubating projects. I chose K3s and Raspberry Pi OS Lite because both are lightweight and work well based on my experience.

Goal

The goal of this first article is to set up a running K3s cluster on multiple Raspberry Pi units, each with its own static IP.

Hardware Choices and Initial Configuration

For this project, I chose Raspberry Pi 4 Model B units for their cost-effectiveness. The cluster comprises:

1 Raspberry Pi with 8 GB of RAM: This unit is intended to handle services with high RAM consumption.
3 additional nodes, each with 2 GB of RAM
A configurable router to which you can connect these devices is a valuable addition

In my setup, this configuration was sufficient for deploying the services I wanted. Feel free to add more nodes if you require additional computing power. You can also run this setup with fewer nodes if you don’t plan to use the Thanos monitoring stack, Hashicorp Vault, or other resource-intensive services.

Initial Configuration of Raspberry Pi using Raspberry Pi Imager

To configure our Raspberry Pis, we will use the simplest and most sustainable method, the Raspberry Pi Imager, available for download from the official Raspberry Pi website. Here’s how to prepare each Raspberry Pi:

Download and Install Raspberry Pi Imager
Download the latest version of Raspberry Pi Imager from the official Raspberry Pi website (v1.8.5 as of the writing of this post).
Insert the SD Card and Launch Raspberry Pi Imager
Choose the appropriate Raspberry Pi model
Choose ‘Raspberry Pi OS Lite (64-bit)’
Under ‘Other general-purpose OS’, select ‘Raspberry Pi OS Lite (64-bit)’ for its reliability and light weight.
Configure Advanced Settings
After selecting ‘Next’ in Raspberry Pi Imager v1.8.5, configure the advanced settings:

Hostname: Assign a unique hostname to identify each device.
Set up a User: Create a user that will be used for SSH connections.
Configure SSID: If you plan to use WiFi, provide your network SSID and password.
Locale Settings: Adjust the locale settings based on your timezone and preferences.

Available custom settings in Raspberry Pi Imager before flashing your SD Card

Public Key for SSH: Add your public SSH key to enable direct SSH access to your devices.

SSH configuration section in Raspberry Pi Imager

Click Save and flash the SD card then repeat the process for each Raspberry Pi unit.

Assigning Static IP Addresses to Raspberry Pis

Assigning static IP addresses to your Raspberry Pi devices ensures network consistency and simplifies cluster management.

Why Assign Static IPs?

Consistency: Ensures devices always have the same IP, making it easier to identify and access nodes.
Cluster Stability: Prevents disruption in communication between nodes due to changing IP addresses.

How to Assign Static IPs

Using Your Router: Access your router’s admin panel (typically via 192.168.0.1 or 192.168.1.1), navigate to DHCP settings or Address Reservation, and assign static IP addresses to each Raspberry Pi's MAC address for centralized and convenient management.

Using nmcli: Use the nmcli command-line tool to directly configure a static IP address on each Raspberry Pi for greater control.
This method works well on Raspberry Pi OS Lite (release of March 15, 2024, Kernel version: 6.6).

# Get your connection names
nmcli connection show

# Example output:
# preconfigured       UUID  wifi      wlan0
# Wired connection 1  UUID  ethernet  eth0

# Use the connection mapped to the device you want to configure:
# "preconfigured" for wireless connections
# "Wired connection 1" for Ethernet connections

# Example for rasp-1 modifying the Ethernet connection to take 192.168.0.156
# Gateway is 192.168.0.1
sudo nmcli c mod "Wired connection 1" ipv4.addresses "192.168.0.156/24" ipv4.method manual
sudo nmcli con mod "Wired connection 1" ipv4.gateway 192.168.0.1
sudo nmcli con mod "Wired connection 1" ipv4.dns "192.168.0.1, 8.8.8.8"
sudo nmcli con up "Wired connection 1"


# To revert to DHCP, use 
sudo nmcli c mod "<connection_name>" ipv4.method auto
sudo nmcli con up "<connection_name>"

Assigning static IP addresses ensures that your K3s cluster is stable and easy to manage.

**Simple and Effective Home Lab Network Setup**

Having a second router is optional but can be beneficial for fine-tuning your network configuration.

Configuring your Raspberry Pis to run K3s

Enabling Necessary System Settings

K3s requires specific Linux kernel features, particularly control groups (cgroups), to be enabled. These settings are crucial as they allow Kubernetes to manage resources like CPU and memory allocation:

Modify the /boot/firmware/cmdline.txt file (or /boot/cmdline.txt depending on your OS version):
Add the following parameters at the end of the existing line in the file (you’ll need elevated privileges):
cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory

cgroup_enable=cpuset: Enables CPU set management in cgroups, essential for assigning specific CPUs and memory nodes to particular processes.
cgroup_memory=1 and cgroup_enable=memory: Activate memory tracking and limiting in cgroups, ensuring that each container uses only its allotted share of memory. This prevents any single process from exhausting system resources, which is vital for maintaining the stability and efficiency of the cluster.
Reboot the system

Installing K3s on your nodes

On the primary node

The primary node will be set up as the control plane, responsible for managing the cluster and orchestrating workloads across all nodes.

# using the official K3s installation script
curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644

This script simplifies the installation of K3s, setting file permissions to allow the Kubernetes configuration to be read without elevated privileges.

Retrieve the cluster token (crucial for security, used to authenticate nodes joining the cluster):

# Run this on the primary node to retrieve the server token
sudo cat /var/lib/rancher/k3s/server/token

Adding additional nodes

# Run this on your additional Raspberry Pis
curl -sfL https://get.k3s.io | K3S_URL=https://<PRIMARY_NODE_IP>:6443 K3S_TOKEN=<TOKEN> sh -

# Be sure to 
# replace <PRIMARY_NODE_IP> by your main node IP
# replace <TOKEN> by the token retrieved on the main node

Verify the cluster node status

kubectl get nodes

High Availability and Scalability

While K3S supports setting up a high-availability environment with multiple master nodes, we will initially stick with a single control-plane node for this setup. Once a persistent storage solution is configured, additional control-plane nodes can be added later to achieve high availability.

Simplify working with your k3s cluster with these helpful utilities

While kubectl is the go-to command line tool, visualizing complex clusters is a breeze with tools like Lens.

kubectl: Powerful, scriptable, but steeper learning curve.
Lens: User-friendly interface, ideal for quick checks.

Use kubectl for automation, Lens for exploration, together they're a powerful duo!

In addition to visualization tools, there are other command-line utilities that can streamline your workflow with kubectl:

kubens: This handy tool allows you to quickly set your current namespace. Namespaces help organize resources within your cluster, and kubens saves you from repeatedly specifying the namespace in your kubectl commands.
kubectx: Managing multiple Kubernetes clusters can be cumbersome. kubectx simplifies switching between contexts (cluster configurations) with a single command.

Conclusion

This article lays the groundwork for establishing a functional Kubernetes cluster with Raspberry Pis. We have detailed the necessary hardware setups, software configurations, and system settings to ensure a successful deployment. In the next article, we’ll begin setting up Terraform to deploy essential system resources such as MetalLB, Nginx controllers, and others.