Cgroup introduction
Control group (cgroup
) represents a Linux kernel design implemented to manage operating system resources.
Let’s describe a situation to understand the cgroup
role:
If applications play usually nicely together, sometimes, a buggy application could consume all the available resources.
In this case, the last line of defense for the Linux kernel before crashing is the Out Of Memory (OOM) Killer process: it stops the application to free up enough resources.
Control group represents a Linux kernel design and an implementation to control resources usage and avoid this kind of situation.
What is cgroup?
Definition
From Kernel Documentation
“cgroup stands for control group and is never capitalized”
“The singular form is used to designate the whole feature and also as a qualifier as in cgroup controllers”
“When explicitly reffering to multiple individual control groups, the plural form cgroups is used”
“cgroup is a mechanism to organize processes hierarchically and distribute system resources along the hierarchy in a controlled and configurable manner”
cgroup
is made of 3 parts:
1- cgroup: collection of processes bound to parameters for one (cgroup version 2) or more subsystems (cgroups version 1),
2- subsystem (controller): kernel component called resource controller or controller. The controllers are various types and are able to:
- limit CPU time on
cgroup
processes, - limit memory available on
cgroup
processes, - account of CPU time on
cgroup
processes, - freeze execution on
cgroup
processes, - resume execution on
cgroup
processes,
3- hierarchy: set of cgroup
arranged in tree.
cgroup
is bound with controllers and a Virtual File System (exists only in memory at runtime).
Here is the simplified architecture of cgroup
:
The file system contains parameters to limit processes. The parameters are related to the elected controllers applicable to the processes.
Basically, cgroup
is designed to track processes but with additional controllers, it can account and limit resources the cgroup can access.
Info
cgroup v1 distinguishes processes from tasks. A process consists of multiple tasks (more commonly called threads). In cgroup v2, there is only processes.
cgroup
is particularly important nowadays because servers are multi-cores CPUs and full of memory: a server can now run multiple services, multiple virtual machines, multiple containers, and multiple user accounts all at the same time.
Main features of cgroup
cgroup
provides control on:
1- Resources limiting:
- number of CPU shared per process.
- limits on memory per process.
- Block Device I/O per process (See
udev
details at the end of the article). - Which network packets are identified as the same type (another application can enforce network traffic rules),
- …
2- Prioritization: allocates to some processes more time on the system compared to others,
3- Accounting: Accounts particular processes to monitor resources utilization for a particular tree (cgroup hierarchy: you can see what processes inside of which cgroup are consuming which types or resources),
4- Processes control: controls workload, applications and resources required. Restricts processes to specific devices, for specific users, groups or processes,
In one sentence, you can use cgroup
to set limits, prioritize or isolate the hardware resources of processes. It allows you to control accurately resource usage of applications to utilize them more efficiently.
cgroup evolution
cgroup
is part of the kernel design to enable you organizing processes into hierarchically ordered groups. This hierarchy (control groups tree) is defined into a cgroup virtual filesystem, mounted by default at /sys/fs/cgroup/
directory. The cgroup
virtual filesystem is loaded at runtime and disappears once system shutdowns.
cgroup
exists in 2 versions:
- version 1 (designed by Google, since Linux kernel 2.6.24),
- version 2 (since Linux kernel 4.17).
In recent systemd
distros, systemd
automatically mount the cgroupfs
(cgroup
file system) version 2 at /sys/fs/cgroup
during the boot process. systemd
and service manager utilize cgroup
to organize all units and services.
You can also manage manually cgroup
hierarchies by creating / removing your own sub-directories inside /sys/fs/cgroup
directory.
Controllers evolution
The main difference beetwen v1 and v2 cgroup
is about the Controllers: in v1, each Controller could be in a separate hierarchy. In v2, this is not possible.
Also, cgroup
v1 and v2 controllers are not the same.
The importance of cgroup
cgroup
is part of the foundation for containers technology, included in any modern Linux OS.
Who is using cgroup?
- systemd
- OpenVZ
- YARN (Hadoop)
- LXC
- rkt
- Docker (containerd)
- Podman
- runc
- crun
- CRI-O
- Tupperware (FB)
- …
So… using containers, you are using cgroup
by default. However, because cgroup
is part of the Linux design, a system administrator can, without containers:
- manage the type of workloads, the applications and the resources they require,
- enhance security because default Linux systems use
cgroup
but does not put any restrictions upon processes. As system administrator, you can restrict access to specific devices for specific users, groups, or processes to help lock down a system. - tune performance through
cgroup
Note
Docker Engine supports
cgroup
v2 since version 20.10.6 (2021-04-12) before, onlycgroup
v1 was supportedImportant
Note also that there is notable differences between Linux and Windows regarding containerization. Windows does not implement
cgroup
andnamespace
in its design.