Unlocking Kubernetes Performance with no CPU Resource Limits
In the dynamic realm of software development, the rise of containerization has revolutionized application deployment and management. Containers, lightweight, self-contained units of software, encapsulate an application’s code, dependencies, and runtime environment, enabling seamless portability and efficient resource utilization. At the heart of this containerized world lies Kubernetes, an open-source platform that orchestrates and manages containerized applications.
When deployed to a cluster of nodes (VMs, physical machines, etc.), Kubernetes orchestrates containerized applications by creating, monitoring and managing groups of containers. It acts as a underlying control system, ensuring that containers are running and functioning correctly, and dynamically scaling them up or down based on workload demands.
Kubernetes Scheduling Fundamentals
Kubernetes containers consume two primary resources: CPU and Memory. These resources can be further defined using two key properties: Request and Limit.
- Request is what the pod is guaranteed to get, K8s will only schedule a pod on a node that can give it the requested resources (Pods are guaranteed to get the amount of CPU they request)
- Limit makes sure that the pod does not consume more resources than specified in the limit (Pods will be throttled if they exceed their limit or killed in the case they exceeds the memory limit)
Both CPU Request and Limit are set in cores or millicores, a millicore represents one-thousandth of the allocated execution time in a CPU core.. It’s best to see millicores as a way to express fractions, x millicores correspond to the fraction x/1000 (e.g. 250millicores = 250/1000 = 1/4). The value 1 represents the complete usage of 1 core (or hardware thread if hyperthreading or any other SMT is enabled).
CPU vs Memory Resources
CPU is a compressible resource, that can be dynamically adjusted to accommodate varying demands. If multiple pods request more CPU than is available, the system will prioritize the available resources proportional to the CPU requested. Given this flexibility, we should not have CPU request and limit values set identically.
Memory, on the other hand, is a non-compressible resource, if a process allocates more memory than it has requested, the only way for Kubernetes to reclaim that memory is to kill the pod.
Once you give a pod memory, you can only take it away by killing the pod
Consequently, we shouldn’t differentiate between memory request and limit. Setting memory request and limit to the exact amount of memory required ensures that processes don’t consume more memory than they need and removes the risk OOMKill due to the pod being evicted from memory it have allocated.
How is Kubernetes scheduling Pods?
The scheduler evaluates each pod’s resource requirements and identifies nodes that can accommodate those needs. This involves considering factors like CPU, memory, storage, and network bandwidth. Nodes that meet the pod’s specifications are deemed feasible candidates for scheduling.
Why Limits Can Be Counter-Productive
Pods may not be able to fully utilize the available resources, even if there is ample spare capacity. This is because the limit acts as a hard cap, preventing the pod from consuming more resources beyond that threshold.
We have recently had a scenario were a pod has a request of 1000m and a limit of 3000m. Even though the CPU utilization of the node was well below 50%, the pod was throttled below its specified limit, leading to CPU contention and performance issues. That the pods was throttled can be due to multiple reasons:
- The metrics do not really give the accurate numbers, but a aggregated view of the CPU usage (meaning that the CPU usage actually was at the set CPU limit)
- The pod is throttled due to that the CPU needs was very high and only needed a very short period of time
Pods getting throttled at Limit
Comparing two cases with a single pod running on a 1 VCPU node: Given a Pod A which executes a batch job that takes 50ms
Having two Pods deployed with no Limit
Comparing two cases with two pods running on a 1 VCPU node with the same CPU request configuration.
Limit vs Request : Matrix
Summing up the scenarios above, we end up with the following matrix (Kubernetes Requests vs Limits) describing what will happen:
Glossary
- CPU_PERIOD_US : cpu.cfs_period_us: [cfs_period in microseconds (us)]
The duration in microseconds of each scheduler period, for bandwidth decisions. This defaults to 100000us or 100ms.
- CFS_QUOTA_US : cpu.cfs_quota_us: [cfs_qouta in microseconds (us)]
The maximum time in microseconds during each cfs_period_us in for the current group will be allowed to run. For instance, if it is set to half of cpu_period_us, the cgroup will only be able to peak run for 50 % of the time.
Conclusion
CPU Requests should correspond to the actual CPU the application needs in order to fulfil the purpose.
- Therefore we must know our system and our application in order to set appropriate magnitude for the requested resource.
CPU Limit should not be set at all
References
- Github — Original Kubernetes documentation
- Github — Pods are guaranteed to get the amount of CPU they request
- Github — Pods will be throttled if they exceed their limit
- StackOverflow — It’s best to see millicores as a way to express fractions
- NetData — CPU limits should rarely be used, if used at all!
- GoogleSource — Linux Kernel: Documentation/cgroups/cpu.txt
- Robusta — For the love of god, stop using CPU limits on Kubernetes
- Medium — Kubernetes resources under the hood — Part 3
- Sysdig — Understanding Kubernetes Limits and Requests
- Robusta — Memory Request should be equal to Limit
