Hello k0s!

Last week Adam Parco unveiled a new open source Kubernetes distro we’ve been working on, called k0s. This announcement prompted a lively discussion around the whole topic of Kubernetes distributions, what makes them useful, and, let’s be honest, why we felt the need for another one. We feel pretty strongly about it, so we wanted to take a deeper look at the vision for k0s, its main features, and why we think they’re important.

Miska Kaipiainen outlined the high level design philosophy of k0s in his recent tweet. We wanted to create a modern, robust and versatile base layer for various use cases where Kubernetes is in play. Something that leverages pure upstream Kubernetes and is versatile enough to cover use cases ranging from local dev environments to cloud based deployments and all the way to various edge/IoT type of use cases.

Let's dig deeper. In the following sections, I'll try to describe some of key features.

Pure upstream Kubernetes and operating system independence

From the outset, we wanted to start with pure upstream Kubernetes and create a distro that’s versatile, runs on all mainstream operating systems, and can form a robust foundation for higher level solutions and products.

Starting with upstream Kubernetes makes life easier for k0s developers too. Maintaining a fork is actually quite a bit of work, and we wanted to stay away from that. Instead, we build all the needed Kubernetes components as statically-linked binaries, and the source is always pure upstream Kubernetes repositories. This lets k0s closely track upstream, and as the community releases new versions of Kubernetes, we can build new versions of k0s with fairly low effort and turnaround time.

Another huge reason to build a single binary package is that it can be built to run on any (Linux) OS kernel, without other dependencies. That means no need to blunder with RPMs, DEBs, Snaps, or what-nots before running k0s. From past experience we’ve learned that those can get both hairy to maintain, hard to reliably automate, and also huge time snitchers. Having a single package (the binary itself) for all OSes lets us focus on the “real thing” rather than environment-specific details.

Running all components as separate binaries also keeps the developers of k0s sane. If we tried to embed all bits and pieces as Golang modules, managing Go dependencies alone would become a nightmare. Instead, we can really focus on the k0s parts, and improve functionality quickly.

Included-but-swappable container runtime, networking, and storage

Docker famously touted its “batteries included, but swappable” message, and k0s has taken the same approach. It has been designed so that you will be able to swap out your CRI, CNI, and CSI.

By default, k0s includes the containerD runtime, but you could swap it out for something different. If you are a vendor or a company (or even just a user) that prefers a specific container runtime, you can use it with k0s. For example, if you needed extra security, you could swap out containerD for the FIPS validated Mirantis Container Runtime (formerly Docker Engine — Enterprise).

Similarly, k0s ships with Calico for networking and in-tree Kubernetes storage providers, but will work with any CNI networking overlay or CSI storage driver.

Swappable, scalable etcd

By default, k0s runs etcd as the datastore for the Kubernetes API. Those of us who’ve worked with etcd, however, know that it’s not the easiest solution to manage and configure as a clustered datastore.

k0s also helps by automatically scaling up the etcd cluster running on controller nodes. To increase availability by adding a new controller node, you can simply run:

k0s server <join-token>

When adding a new controller, the existing controller adjusts the etcd cluster automatically and starts the new node as a member of the cluster. Scaling down does require minimal manual intervention — that is, a simple k0s etcd leave command — as k0s cannot determine whether or not a terminated node will rejoin the cluster.

k0s also embeds kine, by Rancher, which enables it to use other data stores beyond etcd. For example, when running a simple test cluster with limited resources one could use SQLite as the datastore, or when running on a public cloud such as AWS, one could use managed MySQL as the datastore. Depending on use-case specifics, one can implement endless architectural variations.

Why k0s is different — running the control plane without pods and containers

That’s all great, but it’s also table stakes for a modern Kubernetes distribution. Here’s where it gets interesting.

In most Kubernetes distros, the control plane runs in containers and pods, which are managed by kubelet as static manifests. While this architecture does simplify the bootstrap process, it also creates a connection between the control plane and the workers of the cluster. Taints are usually used to build a boundary between the control plane and the workers, and to prevent normal workloads from landing on control plane nodes, but a taint-based boundary isn’t perfect; container workloads still connect to the overlay network, and thus could possibly move laterally in the cluster.

In k0s, by default, we create the control plane as fully isolated. There’s no kubelet or container runtime running on controllers; ALL the control plane components run in “naked” processes. In fact, the control nodes are not directly part of the cluster, so there’s zero opportunity for users to schedule pods to controller nodes, whether deliberately or accidentally.

You may have seen this phenomenon before, as this is pretty much what many managed Kubernetes services do to protect their control planes. If you’ve been using managed services such as EKS, GKE, and others, you may have noticed there are no “master” nodes in the kubectl get nodes list. Here we’re accomplishing the same isolation those services do, but with a single binary distro.

k0s as a single binary

k0s is packaged as a single binary, available for both amd64 and arm64 architectures. One way to look at it is like a self-extracting archive where we embed all the needed kubernetes binaries into the k0s binary. At runtime, we dump all these binaries onto disk and spawn the components as normal OS processes.

So why not run the Kubernetes bits in containers? For one thing, running everything in normal processes enables us to run the control plane in environments where running containers doesn’t make sense, or where it isn’t possible.

That’s a pretty bold statement. Let me explain.

Running k0s everywhere

One of the things we’ve learned from experience is that people want to run Kubernetes and containers in lots of different deployment architectures. Allowing for that kind of variety poses a general challenge to typical Kubernetes setups, because the control plane and workers still need to talk to each other, and nothing about the architecture we’ve described so far guarantees that. For example, the worker nodes could be running in an environment that doesn’t allow incoming connections.

Luckily, we’re not the first people to confront this problem. Upstream Kubernetes already has a concept called an “egress selector.”

An egress selector acts as a proxy between the api server and kubelet. When the api-server wants to connect to the kubelet it actually connects to this egress selector, which in turn forwards the connection to the kubelet.

But that is only part of the problem.

Creating a two-way connection

Remember, we also need to solve for situations where the worker node environment can’t accept incoming connections. To solve that problem, k0s uses a konnectivity server and agent to implement reverse tunnels. In other words, the connection is initiated by the worker node “calling-home” to the control plane and establishing a persistent tunnel. When the api-server calls the kubelet, the konnectivity server, running alongside the api-server on the same node, proxies that call through this bi-directional tunnel.

Now, there are some massive implications here.

What all of this means

When we add a new node to the cluster, we use a token. The token embeds all the information needed for the worker to call home. So as long as the worker can do that it can join the cluster.

All of this means that it becomes trivially easy to create a distributed cluster. Connect your laptop to the cluster. Create a cluster of IoT devices that act as worker nodes. Or create a k0s cluster that runs in the cloud and hosts multiple control planes for other k0s clusters, each hosting a set of on-premise worker nodes.

We’re just getting started

Though we’ve been working on k0s for a while, we’re just getting started. k0s is a really young project, so we expect it to evolve rapidly over the next few months as a robust and versatile way of running Kubernetes anywhere. And there’s of course some rough edges we need to fix. To achieve this goal, of course, we need your help and feedback. Let us know what you want to see in k0s, and please file issues as you try it! (Oh, and we apologize for the dismal state of the docs, we promise we’re working on it!) We’re excited to gather all community feedback and especially involvement to help realise this vision for k0s.

P.S. Exciting stuff is afoot! Remember, where I mentioned that a goal of k0s was to run on “all OSes?” What if you could type something like c:\>k0s.exe worker <token> (i.e., on a Windows Server command line)? We’ll need to overcome some obstacles before this works, but we hope to actually support Windows workers in the near-ish future.