Linux Beyond the Basics: CPU and CPU Set Cgroups

The Art of CPU Resource Management

This blog post is part of the series Linux Beyond the Basics.

Introduction

Welcome back, Linux enthusiasts! In our ongoing series exploring the depths of the Linux kernel, we’re diving into the powerful world of cgroups (control groups). Today, we’ll focus on two specific types: cpu and cpuset, which give you fine-grained control over how your system's CPU resources are allocated.

Why CPU Resource Management Matters

Imagine a scenario where a rogue process starts consuming all your CPU power, causing other critical applications to slow down or even crash. Or maybe you want to ensure a real-time application always gets the CPU time it needs, regardless of what else is running. This is where cpu and cpuset cgroups come to the rescue!

Understanding cpu cgroups

The cpu cgroup primarily governs the share of CPU time processes receive. It's like dividing a pie – you determine how much each process (or group of processes) gets. This control is achieved through a hierarchy of three parameters that work together. There are several key parameters:

cpu.shares (The Foundation):

• This parameter acts as the foundational layer for CPU resource allocation. It sets a relative weight that determines the cgroup’s share of CPU time compared to other cgroups. The default value is 1024.
• For instance, if you have two cgroups, A and B, with A having cpu.shares set to 2048 and B set to 1024, A will get twice as much CPU time as B when both are competing for resources.
• Think of cpu.shares as the base amount of CPU time a cgroup receives under normal conditions.

Example: Say there are 3 cpu cgroups A, B and C, their cpu.shares are 1000, 2000, 3000 respectively. During a time period of 10s, A is idle waiting for I/O, B and C are busy, how would the Linux CPU scheduler distribute CPU cycles?

Answer: In the Linux Completely Fair Scheduler (CFS), CPU cycles are distributed proportionally to the cpu.shares values assigned to each cgroup. However, since cgroup A is idle, it won't receive any CPU time during this period.

Here’s how the CPU cycles would be distributed:

Calculate the total shares of active cgroups:

• B (2000) + C (3000) = 5000 shares

Determine the proportion of shares for each active cgroup:

• B: 2000 / 5000 = 40%
• C: 3000 / 5000 = 60%

Distribute the CPU cycles according to the proportions:

• If we assume a total of 100 CPU cycles in the 10-second period:
• B would receive 40 cycles (40% of 100)
• C would receive 60 cycles (60% of 100)

cpu.cfs_period_us and cpu.cfs_quota_us (The Cap)

• These parameters act as an upper limit or cap on top of the cpu.shares allocation. They work together to enforce a CPU time quota.
• cpu.cfs_period_us defines a time window (in microseconds). The default is 100000 microseconds (100 milliseconds).
• cpu.cfs_quota_us specifies the maximum amount of CPU time (also in microseconds) the cgroup is allowed to consume within the defined cpu.cfs_period_us.
• For example, if cpu.cfs_period_us is 100000 (100 ms) and cpu.cfs_quota_us is 20000 (20 ms), the cgroup is limited to using a maximum of 20% of the CPU during each 100 ms interval.

How They Work Together

• Low CPU Demand: When the overall system has plenty of CPU capacity available (low utilization), the cpu.shares value primarily determines the distribution of CPU time among cgroups. Cgroups with higher shares get proportionally more CPU time.
• High CPU Demand: As the demand for CPU resources increases, the quota mechanism (cpu.cfs_period_us and cpu.cfs_quota_us) kicks in. If a cgroup tries to consume more CPU time than its quota allows within a period, it's throttled back to respect the limit.

Key Points to Note

• The cpu.shares setting establishes a relative distribution of CPU resources among cgroups.
• The quota mechanism enforces an absolute limit on CPU time usage for a cgroup within a specific time window.
• Both mechanisms work in conjunction to ensure fair distribution and prevent monopolization of the CPU.

Use Cases

• Fairness: Ensure no single application starves others of CPU time.
• Rate Limiting: Cap the CPU usage of specific applications or services.
• Prioritization: Give more CPU time to critical processes when needed.

Understanding cpuset cgroups

While cpu cgroups manage shares, cpuset cgroups control which specific CPUs (and memory nodes) a group of processes can use. This is invaluable for performance optimization and isolation.

Key Parameters:

• cpuset.cpus: A comma-separated list or range of CPU core IDs. For instance, cpuset.cpus = 0-3 means the cgroup can only run on cores 0, 1, 2, and 3.
• cpuset.mems: A comma-separated list or range of memory node IDs (relevant for NUMA systems). For example, cpuset.mems = 0 restricts the cgroup to memory node 0.

Use Cases:

• NUMA Optimization: Bind processes to CPUs close to their memory for better performance.
• Real-time Isolation: Dedicate specific CPUs to real-time applications for guaranteed responsiveness.
• Security Hardening: Isolate sensitive processes to reduce the attack surface.

Practical Examples

Let’s walk through two examples with specific parameter settings:

1. Limiting Web Server CPU Usage with cpu cgroup:

sudo mkdir /sys/fs/cgroup/cpu/webserver
sudo echo 50000 > /sys/fs/cgroup/cpu/webserver/cpu.shares 
sudo echo 80000 > /sys/fs/cgroup/cpu/webserver/cpu.cfs_quota_us  # Max 80ms per 100ms
sudo echo 100000 > /sys/fs/cgroup/cpu/webserver/cpu.cfs_period_us  # Period of 100ms
pid=$(pgrep -f "nginx")  # Replace 'nginx' with your web server's name
sudo echo $pid > /sys/fs/cgroup/cpu/webserver/tasks

2. Assigning a Real-Time Application to Specific Cores with cpuset cgroup:

sudo mkdir /sys/fs/cgroup/cpuset/realtime
sudo echo 2,3 > /sys/fs/cgroup/cpuset/realtime/cpuset.cpus   # Assign cores 2 and 3
pid=$(pgrep -f "realtime_app")  
sudo echo $pid > /sys/fs/cgroup/cpuset/realtime/tasks

(Make sure to replace "realtime_app" with the actual name or pattern of your real-time application)

The Power at Your Fingertips

cpu and cpuset cgroups unlock a world of possibilities for optimizing resource allocation, improving system responsiveness, and enhancing security. Experiment with these tools to fine-tune your Linux system for optimal performance.

Happy cgrouping!