Linux Beyond the Basics: Cgroups

The Art of Resource Control

This blog post is part of the series Linux Beyond the Basics.

Introduction

Welcome back, Linux enthusiasts! In our ongoing series exploring the depths of the Linux kernel, we’re turning our attention to a powerful mechanism known as cgroups (control groups). If you’ve ever wondered how to keep your resource-hungry processes in check, manage containers effectively, or prioritize workloads, then cgroups are your new best friend.

What Are Cgroups?

At their core, cgroups are a kernel feature that allows you to partition and limit the system resources (CPU, memory, disk I/O, network, etc.) that a group of processes can use. Think of them as virtual cages where you can corral your processes and set rules for their behavior. This gives you fine-grained control over how your system’s resources are allocated, ensuring that critical processes get the resources they need while preventing others from hogging everything.

Why Use Cgroups?

1. Resource Management: This is the bread and butter of cgroups. You can set hard limits on how much of a particular resource a group of processes can consume, ensuring fairness and preventing resource exhaustion.
2. Prioritization: Need to make sure your database server gets priority access to the CPU? Cgroups allow you to assign relative weights to different cgroups, giving some processes preferential treatment.
3. Accounting: Cgroups meticulously track the resource usage of each group, giving you valuable insights into how your system is being utilized. This data can be invaluable for performance tuning and capacity planning.
4. Isolation: By creating separate cgroups for different workloads, you can isolate them from each other. This means if one process goes haywire, it won’t impact the others.
5. Containerization: Cgroups are a fundamental building block for container technologies like Docker and Kubernetes. They provide the isolation and resource management necessary to make containers work seamlessly.

How Do Cgroups Work?

Cgroups operate through a hierarchical structure, where the root cgroup encompasses all processes on your system. You can create child cgroups within this hierarchy, each inheriting the resource limits of its parent but capable of further refining those limits. This structure allows you to organize processes logically and apply different resource policies to different groups.

Cgroups work by tracking and controlling the resource usage of processes within each group. For example, a CPU cgroup can set limits on the CPU time that processes within that group can consume. This is achieved through various kernel mechanisms like scheduling classes, resource counters, and notification events.

How to Create Cgroups

Mount the Cgroup Filesystem: By default, the cgroup filesystem is not mounted. You’ll need to mount it to interact with cgroups. The most common location is /sys/fs/cgroup:

sudo mount -t cgroup -o none,name=systemd cgroup /sys/fs/cgroup

Note: For older systems that do not use systemd, you might use a slightly different command like sudo mount -t cgroup none /sys/fs/cgroup

Create a Cgroup: Navigate to the desired controller directory (e.g., /sys/fs/cgroup/cpu) and create a new directory for your cgroup:

sudo mkdir /sys/fs/cgroup/cpu/my_cpu_group

Set Resource Limits: Within your cgroup directory, you’ll find various files that control resource limits. For example:

sudo echo 512000 > /sys/fs/cgroup/cpu/my_cpu_group/cpu.shares  # CPU shares
sudo echo 10000000 > /sys/fs/cgroup/memory/my_cpu_group/memory.limit_in_bytes  # Memory limit  

Important: If your cgroup is also meant to control memory, you’ll need to create it under the /sys/fs/cgroup/memory hierarchy as well.

Add Processes: To add a process to a cgroup, write its PID to the cgroup.procs file within the cgroup directory:

sudo echo 12345 > /sys/fs/cgroup/cpu/my_cpu_group/cgroup.procs

Checking Cgroups of a Running Process

To see which cgroups a running process belongs to, you can inspect the /proc/[PID]/cgroup file (replace [PID] with the process ID). Each line in this file represents a hierarchy, with the format:

hierarchy-ID:controller-list:cgroup-path

For example:

2:cpu,cpuacct:/user.slice

This means the process is in the /user.slice cgroup under the cpu and cpuacct controllers.

Additional Tools

• systemd-cgls: Visualizes the cgroup hierarchy in a tree-like structure.
• systemd-cgtop: Provides a real-time view of resource usage by different cgroups.

Can Cgroups Borrow Unused Resources?

If some cgroups are not using resources to their limits, whether other cgroups can use the idle resources depends on the type of resource and the specific configuration of the cgroups. Here’s how it works for different resources:

CPU:

• The CPU resource controller can redistribute unused CPU cycles to other cgroups that need them. If a cgroup is not using its allocated CPU time, other cgroups can utilize the idle CPU cycles, provided they have tasks that are ready to run.
• The “cpu.shares” parameter is often used to specify relative CPU allocation rather than strict limits. If a cgroup has a lower load than its share, other cgroups can use the extra CPU capacity.

Memory

• The memory resource controller typically enforces hard limits. If a cgroup does not use its allocated memory, other cgroups will not automatically be able to use the idle memory beyond their set limits.
• However, if the “memory.high” limit is set and not exceeded, other cgroups can use more memory until the “memory.max” limit is hit.

Disk I/O

• The I/O resource controller can redistribute unused I/O bandwidth. If a cgroup is not using its allocated I/O capacity, other cgroups can use the available bandwidth, assuming they have I/O operations pending.
• Similar to CPU, the “blkio.weight” or “io.weight” parameters specify relative I/O allocation. Idle I/O bandwidth can be utilized by other cgroups.

Network Bandwidth

• The network bandwidth resource controller (part of the “net_cls” or “net_prio” controllers) can allocate bandwidth among cgroups. If a cgroup is not using its allocated bandwidth, the unused bandwidth can typically be utilized by other cgroups.

In summary, for resources like CPU and I/O, cgroups are designed to share idle resources dynamically, allowing other cgroups to utilize them when they are available. For memory, the sharing of idle resources is more restrictive, as it often involves hard limits that are not exceeded even if there is unused capacity. However, specific configurations and hierarchical settings can influence these behaviors.

Cgroups and Systemd: A Dynamic Duo

Systemd, the modern init system used by many Linux distributions, leverages cgroups extensively. It automatically places services and user sessions into different cgroups to manage their resource usage. This integration simplifies resource management and provides a more unified way to control and monitor processes.

Systemd’s tools like systemd-cgls (visualizing the cgroup hierarchy in a tree-like structure) and systemd-cgtop (providing a real-time view of resource usage by different cgroups) provide user-friendly ways to visualize and manage cgroups, making them more accessible to administrators.

Conclusion

Cgroups are a testament to the flexibility and power of the Linux kernel. By understanding and harnessing their capabilities, you can unlock a new level of control over your system’s resources. Whether you’re managing a bustling server environment, optimizing containerized applications, or simply taming unruly processes, cgroups offer a versatile toolkit for achieving your goals.