Kubernetes Issues
Follow up on Not an introduction to Kubernetes on-premises (bare-metal). Some issues we had experienced while working with bare-metal Kubernetescluster.
Active Development
While “active”means “bugfixes”and “new features”all the time it also means that even in GA(General Availability) stage of development you will probably encounter some inconsistent changes and behaviour. And if you are planning to use something in beta, or alphastage you will probably encounter a lot more issues. But it depends.
Also, bare-metal is not a main focus of development. Cloud providers is.
But it has it’s upsides too. Via kubectl you can drain nodes for maintenance and upgrade. Upgrade procedure for each version has migration documentation and kubeadm has built-in features to provide you best experience for rolling updates across clusters. You can perform a dry run and check difference between current version and next version (configuration and services wise). Just don’t forget to pass — config parameter. Otherwise you will end up with completely different configuration. And backups to. Do not forget about those.
And yes, you will need to upgrade your cluster quite often. And it requires a lot of time for maintenance.
Ecosystem
Usually, rich “ecosystem” means good things. But, like in case of NodeJS’s NPM, it also has a lot of problems. Due to active development and no LTS releases there is a constant race between developers of Kubernetes and community. A lot of potentially good tools are abandoned because of that. It requires a lot of work to keep your projects for Kubernetes updated. Because of this update of some operator could lead to problems across the cluster (later about this).
Basically, pin your operators, controllers, pods, deployments, etc images versions by version (ie, flux:1.10.1), not tag (like “latest”or “stable”).And upgrade only after reviewing all the changes in changelog. Also, review default installation procedure and check all images versions/tags and change them to pinned ones if needed.
Ingress
… and service discovery
That is one of the things where Kubernetes shines. First of all, lets take a step back: every thing inside Kubernetes described by yaml-configs. And that is very convenient and powerful feature of Kubernetes. I won’t describe how exactly Ingress works or how to configure it. We are here because of the issues related to Ingress.Which leads us to second point: there are a lot of ingress-plugins (solutions) which solve a lot of problems. But to make Ingress you will need to expose default ingress ports outside. And if you are using some Cloud Provider all you need to do is to specify LoadBalancer type of endpoint. Cloud Provider will do everything needed to expose those ports on some external interface(-s). But, we are speaking about bare-metal installations here. There are a bunch of solutions to make it work. But, in any case, you will need to expose some host ports via external Load Balancers (you could run something like MetalLB or use HAProxy + proxy-protocol). Every solution has its ups and downs. HAProxy could be configured so it will just proxy TCP/UDP traffic to your ingress-controllers (eg, nginx-ingress and proxy-protocol will provide an information about client’s IP address). But because of bare-metal installation you can experience some problems with integrating your cluster with CI/CD services (such as Gitlab) — because theirs main focus is Cloud Providers, not bare-metal installations. In our case we were not able to integrate Gitlab with Kubernetes cluster fully (cert-manager installation via Gitlab+ Helm was not working properly and Gitlab could not finish cluster initialisation).
Helm
Because i mentioned it in previous issue.
So you have RBAC in Kubernetes. And kubectl works with your cluster via Kubernetes API Server which uses RBAC to control user’s permissions and accounts. I think you heard about Helm. Good tool and so on. First issue which you will encounter while using Helm: it has its own ACL. But Helm also limited by RBAC in Kubernetes. So you will need accounts both in Helm and Kubernetes. Second issue related to deployments. Basically, if you deploy something via Helm and delete this deployment via Kubernetes later you will end up in the situation there Helm think that application already deployed but Kubernetes will have different opinion on that. All issues related to Helm are based on one simple problem: Helm has its own database. Based on experience and a lot of reading material available Helm is not needed in most cases. As a matter of fact, i could not think of any use case there Helm was required. I would suggest against using Helm in your own cluster.
CI/CD
While Kubernetes has a lot of solutions in this department every single one of them has its own issues (surprise!). First of all you will need to decide whether you are ready for GitOps or not. Based on that you will need to pick some solution to deliver your updates and deploy them. If you are using Gitlab and want to use theirs solution you are stuck with Helm. But you will be able to use Review Apps.
GitOps is awesome, but do you need to sync cluster state with Git bidirectionally or just Git → Cluster?
Do you keep all the manifests in one repository (for all of your projects and services) or in each repo its own manifest? Some services which are using GitOps for CI/CD workflow pipeline support only one Gitrepo. Some support bidirectional syncing (keep in mind — it is an expensive feature).
I’d recommend to use GitOps with Git → Cluster sync only. This way you always know which version of service/application currently deployed. And can easily rollback. Also it is quite transparent. But this way you will not be able to use Review Apps feature of Gitlab.
Evictions And Resource Limits
Biggest problem of them all. First of all evictions won’t work properly without specified resource limits for your pods. Problem is Kubernetesdecides whether to deploy some pod onto some node using QoS policies and available resources on that node.
There are 3 different QoS policies: BestEffort, Guaranteed, Burstable.
BestEffort means that there are no limits/requests specified in POD’s manifest.
Guaranteed means that limits = requests.
Burstable means that limits > requests (which means that your application guaranteed to have resources = requests available at all times, but may use slightly more than that).
Picture this: we have 1 node with 30% CPU + 500mb RAM available and we want to deploy our application. Lets say our application uses 300mb of RAM but it also have memory leakage issues and it could use up to a couple of GBs of RAM.
With BestEffort evictions and other policies related to resource usage won’t save us from System OOM errors which will lead to node instability and crashes. Which in turn will lead to downtime of all applications deployed on that node.
Guaranteed is the safest one. Because this way Kubernetes always knows exactly how much resources your POD will consume and Kubernetes’s scheduler will take it into consideration while deploying said POD (and evicting).
Burstable works almost like previous one but in some cases you may end up with System OOM issues which will lead to node crashes and downtime. To avoid such situations you need to specify resource requests only slightly more than limits.
One of the reasons i recommend to use Cloud Provider based Kubernetes solution is eviction policies and resource limits. We lost a lot of time tuning eviction policies and still it has some issues.
Conclusion
Some of this issues related both to bare-metal installations and Cloud Provider based ones. Some — only for bare-metal. To minimise maintenance costs (both price- and time- related) it is better to use Cloud Provider based ones.
But if you need more freedom (in some way or another) bare-metal is your answer (but remember: you will need at least a couple of DevOps who knows a thing or two about Kubernetes).
Hiplabs
- Scalable web services.
- Mobile apps.
