Not an introduction to Kubernetes on-premises (bare-metal)

Lets talk about pros and cons of bare-metal Kubernetes usage in development and production. Even though Kubernetes has a lot of features which allows better workflow, it also has a lot of problems in case of bare-metal implementation (though, much less than a year or so ago).

First and foremost: if you really want to use Kubernetes in your daily workflow, it is better to use GKE/AKS/Amazon EKS solutions (or similar) due to multiple tools and services which were created, firstly, for such environments and some of theirs features won’t work on-premises or just hard to setup (especially those which related to networks and LoadBalancers and etc).

Second: do you really need Kubernetes or you need some kind of orchestration tool?

Quick comparison

Lets start with discussion about which tool to use and in which cases (and remember, we are talking here about running pretty much default setup of Docker Swarm and Kubernetes). I will compare Docker Swarm with Kubernetes and solutions based on those tools (generally used by large amount of developers and end users). I won’t talk about some obvious like quorums and how the whole cluster works inside Kubernetes and Swarm. This article not about that. But, there are some issues related to clusters and cluster management in both cases.

In both cases:

• Minimum number of master-nodes to implement HA-cluster: 3

Docker Swarm:

• Hardware requirements: low (you can use machines with 1 CPU core with low resources available — RAM and disk space — and it will work somewhat fine)
• You can control your cluster (eg, deploy/scale/remove services) from your PC/Mac as long as your machine is one of master nodes (which means you need to have direct connection between your PC/Mac and other nodes)
• You can add additional nodes to your cluster as long as they can communicate directly with each other
• No eviction policies, no monitoring (even basic one) out of box. Basically — straight orchestration without a lot of massive (though powerful) services
• Container Runtime Interface (if i may say so in this case), obviously — Docker

Kubernetes:

• Hardware requirements: higher. You need at least 2 CPU cores and 4GB RAM for your master-nodes — that is if you are running stacked control plane (which means you are running all Kubernetes services and ETCD on the same machines at the same time). In case you have different machines for ETCD and Kubernetes you will still need 2 CPU cores at least
• You can add additional nodes to your cluster even if they are behind NAT but you will need to use some CNI (network) plugins which are compatible with NAT
• You can control your cluster via kubectl no matter how far your PC/Mac behind some NAT. And that is a game changer. Also, you don’t need join cluster as a master node because kubectl uses Kubernetes API server to communicate with cluster. Out of box you can setup authentication via RBAC and self-generated keys for separate users and roles
• Eviction policies, resource limits, powerful API for plugins, basic monitoring out of box (though not powerful enough)
• Container Runtime Interfaces (CRIs): Docker, CRI-O, Containerd, frakti (official docs about CRI — https://kubernetes.io/docs/setup/cri)
• Multiple types of Ingress/Egress controllers (you will probably need to expose some services to outside — right?)

Docker Swarmor Kubernetes? Why and when.

Swarm: use than you need to setup something quick and somewhat reliable (Swarm has its own issues related to networking and deployment — at least the ones i struggled with). It will give you an ability to deploy (and scale) services across cluster and network communication between services without exposing unneeded services to outside. If you have multiple replicas of services, Swarm has load balancer which can cover some of the use cases.

Kubernetes: heavy artillery. If you need eviction policies; resource limits; custom network solutions for your pods, containers, services, deployments, etc. If you know about operators and can leverage this feature in your projects or infrastructure or both. If you need highly supported and active solution (even though with a lot of problems). Also, kubectl is awesome. In this case, use Kubernetes.

But there are a lot of problems and issues related to Kubernetes. Sometimes minor updates could break compatibility with older versions (while major updates won’t). Sometimes your eviction policies just does not work even though you had set them up with a lot of system and Kubernetes reserved resources. Sometimes CNI (network) plugin will produce errors. Sometimes some operators will stop working and due to some limited debug features you won’t be able to pinpoint the problem without outside help. In fact, a lot of problems related to Kubernetes requires outside help in some form or other.

Kubernetes Issues

Lets keep things slim and article-related and move into another article: Kubernetes Issues

Hiplabs

Hiplabs.dev

• Scalable web services.
• Mobile apps.