Cgroup introduction

(Credit: Charles Vissol)

Control group (cgroup) represents a Linux kernel design implemented to manage operating system resources.

Let’s describe a situation to understand the cgroup role:

If applications play usually nicely together, sometimes, a buggy application could consume all the available resources.

In this case, the last line of defense for the Linux kernel before crashing is the Out Of Memory (OOM) Killer process: it stops the application to free up enough resources.

Control group represents a Linux kernel design and an implementation to control resources usage and avoid this kind of situation.

What is cgroup?

Definition

From Kernel Documentation
“cgroup stands for control group and is never capitalized”
“The singular form is used to designate the whole feature and also as a qualifier as in cgroup controllers”
“When explicitly reffering to multiple individual control groups, the plural form cgroups is used”
“cgroup is a mechanism to organize processes hierarchically and distribute system resources along the hierarchy in a controlled and configurable manner”

cgroup is made of 3 parts:

1- cgroup: collection of processes bound to parameters for one (cgroup version 2) or more subsystems (cgroups version 1),

2- subsystem (controller): kernel component called resource controller or controller. The controllers are various types and are able to:

• limit CPU time on cgroup processes,
• limit memory available on cgroup processes,
• account of CPU time on cgroup processes,
• freeze execution on cgroup processes,
• resume execution on cgroup processes,

3- hierarchy: set of cgroup arranged in tree.

cgroup is bound with controllers and a Virtual File System (exists only in memory at runtime).

Here is the simplified architecture of cgroup:

The file system contains parameters to limit processes. The parameters are related to the elected controllers applicable to the processes.

Basically, cgroup is designed to track processes but with additional controllers, it can account and limit resources the cgroup can access.

Info

cgroup v1 distinguishes processes from tasks. A process consists of multiple tasks (more commonly called threads). In cgroup v2, there is only processes.

cgroup is particularly important nowadays because servers are multi-cores CPUs and full of memory: a server can now run multiple services, multiple virtual machines, multiple containers, and multiple user accounts all at the same time.

Main features of cgroup

cgroup provides control on:

1- Resources limiting:

• number of CPU shared per process.
• limits on memory per process.
• Block Device I/O per process (See udev details at the end of the article).
• Which network packets are identified as the same type (another application can enforce network traffic rules),
• …

2- Prioritization: allocates to some processes more time on the system compared to others,

3- Accounting: Accounts particular processes to monitor resources utilization for a particular tree (cgroup hierarchy: you can see what processes inside of which cgroup are consuming which types or resources),

4- Processes control: controls workload, applications and resources required. Restricts processes to specific devices, for specific users, groups or processes,

In one sentence, you can use cgroup to set limits, prioritize or isolate the hardware resources of processes. It allows you to control accurately resource usage of applications to utilize them more efficiently.

cgroup evolution

cgroup is part of the kernel design to enable you organizing processes into hierarchically ordered groups. This hierarchy (control groups tree) is defined into a cgroup virtual filesystem, mounted by default at /sys/fs/cgroup/ directory. The cgroup virtual filesystem is loaded at runtime and disappears once system shutdowns.

cgroup exists in 2 versions:

• version 1 (designed by Google, since Linux kernel 2.6.24),
• version 2 (since Linux kernel 4.17).

In recent systemd distros, systemd automatically mount the cgroupfs (cgroup file system) version 2 at /sys/fs/cgroup during the boot process. systemd and service manager utilize cgroup to organize all units and services.

You can also manage manually cgroup hierarchies by creating / removing your own sub-directories inside /sys/fs/cgroup directory.

Controllers evolution

The main difference beetwen v1 and v2 cgroup is about the Controllers: in v1, each Controller could be in a separate hierarchy. In v2, this is not possible.

Also, cgroup v1 and v2 controllers are not the same.

The importance of cgroup

cgroup is part of the foundation for containers technology, included in any modern Linux OS.

Who is using cgroup?

• systemd
• OpenVZ
• YARN (Hadoop)
• LXC
• rkt
• Docker (containerd)
• Podman
• runc
• crun
• CRI-O
• Tupperware (FB)
• …

So… using containers, you are using cgroup by default. However, because cgroup is part of the Linux design, a system administrator can, without containers:

• manage the type of workloads, the applications and the resources they require,
• enhance security because default Linux systems use cgroup but does not put any restrictions upon processes. As system administrator, you can restrict access to specific devices for specific users, groups, or processes to help lock down a system.
• tune performance through cgroup

Note

Docker Engine supports cgroup v2 since version 20.10.6 (2021-04-12) before, only cgroup v1 was supported

Important

Note also that there is notable differences between Linux and Windows regarding containerization. Windows does not implement cgroup and namespace in its design.