Common Kubernetes Errors Made by Beginners [2023]
Most problems with Kubernetes adoption ultimately stem from the complexity of the technology itself. There are unobvious difficulties and nuances of implementation and operation, and there are underutilized advantages. Additionally, as you use Kubernetes more frequently, you can become a victim to alert fatigue. As a result, you lose money.
After a few years of experience with Kubernetes and several consultations for various clients, I have noticed some common Kubernetes errors. So, I thought of selecting and discussing the 6 frequent ones.
- The selector of the labels on the service does not have a match with the pods
To function correctly as a network balancer, a service generally specifies selectors that allow you to find the pods that are part of the balancing pool. If there is no match, the service has no endpoints to forward traffic to, and an error occurs. Bear in mind that the load balancing towards the pods is of a random type.
2. Wrong container port mapped to the service
Each service has two fundamental parameters, “targetPort” and “port”, which are often confused and misused. This confusion then results in error messages claiming that the connection was refused or there was a lack of response to the request.
To avoid this error, remember that “targetPort” is the destination port in the pods, the one to which a service goes to forward traffic. The “port” parameter, on the other hand, refers to the port the service exposes to the clients.
They can be the same, so it is essential to know their meanings!
3. CrashLoopBackOff
Another frequent Kubernetes error is the crashloopbackoff error. It occurs when a pod is running, but one of its containers is restarting due to termination (usually the wrong way). In other words, the container has fallen in the loop of start-crash-start-crash.
The CrashLoopBackOff error can occur due to various reasons — the wrong deployment of Kubernetes, liveness probe misconfiguration, and init-container misconfiguration. An easy way to resolve this error is by properly configuring and deploying Kubernetes. However, you can also bypass the error by creating a separate deployment with the help of a blocking command.
4. Liveness and readiness probes
Several mistakes are made regarding probes. The first is not defining any health check for the application, which will never be restarted in case of problems and will always remain within the load-balancing pool of a service.
The second type of error concerns defining equal liveness and readiness probes by contacting the same HTTP endpoint, for example. It may be due to a misunderstanding of these types of tests. The liveness probe is linked to the concept of a healthy application, so if it fails, the pod will be restarted.
The readiness probe relates to whether or not the pod is inserted into the load-balancing pool during the entire life cycle of the pod. Defining them equal could risk, for example, the restart of an application that has temporarily slowed due to a lot of traffic.
Another critical aspect is not to make probes fail if one of the service’s dependencies is down. It would only compound the problem, which must be detected in the appropriate service.
5. Resources — Requests and Limits
Generally, newbies forget to set requests and limits for CPU and memory. This error leads to too many applications being scheduled on some worker nodes, causing them to become “overcommitted”. The final result is a low-performing application.
On the other hand, if the limits are set too low, the CPU performs poorly. If you’ve seen the OOMkill error message before, this is the problem. The ideal strategy is to allocate the right amount of CPU and memory, possibly setting requests equal to the limits (Guaranteed QoS) for critical applications.
Understanding the necessary resources is basically to test and monitor with tools like Prometheus.
6. Too many load balancer–type services
It often happens that beginners expose too many services to the outside using the load balancer type. In this case, a specific controller provided by the cloud provider creates a load balancer outside the cluster, inside the VPC. These resources can get quite expensive, primarily if they are associated with static IP addresses.
The best solution is to expose the cluster with a single load balancer that will forward the traffic to the cluster nodes, which generally have an ingress controller for routing using ingress resources.
If you enjoyed this blog post, you can encourage me to produce more content by buying me a coffee.
Final Words
Don’t expect everything to work out magically: Kubernetes is not a panacea. A lousy application will stay that way even in Kubernetes (and possibly get worse). If you are careless, the control layer will become increasingly complex, slow, and arduous. Additionally, failure to have a disaster recovery strategy could leave you vulnerable. The Kubernetes platform cannot automatically ensure isolation and redundancy. Take some time to make your application truly cloud-native.
