Optimizing Docker with systemd: A Comprehensive Approach

“When work is well done, it brings joy.”

Introduction

The purpose of this article is to provide readers with a brief and understandable introduction to the capabilities of Systemd and demonstrate how these capabilities can be used to configure Docker services. At a time when the industry widely adopts container managers such as Kubernetes, Red Hat OpenShift, Amazon EKS, and Google Kubernetes Engine, this article targets those who are just starting to work with Linux and Docker. I aim to provide a quick and understandable introduction to service management using Systemd, demonstrating how to efficiently configure and manage Docker services. To ensure smooth operation and ease of management after system updates, making this article ideal for beginners in containerization.

Article Plan

1. Short description of Systemd and its functions.
2. Changing the startup of Docker and Docker Compose services: integration and configuration.
3. Detailed overview of Unit files in Systemd and their structure.
4. Important directives in the [Unit], [Service], and [Install] sections.
5. Application and examples of using other sections, particularly [Timer].
6. Network configuration: static IP, DHCP.
7. Organization of the Systemd filesystem: directory partitioning and their purposes.

Systemd is a system and service manager for Linux that replaces traditional init scripts. It plays a key role in system initialization, managing services, and dependencies between them. In this article, we explore the basic capabilities of Systemd and show how they can be applied to configure and manage the Docker service.

Introduction to Systemd

Systemd is a system and service manager for Linux, replacing traditional init scripts. It plays a key role in system initialization, service management, and dependencies between them. In this article, we explore the core features of Systemd and demonstrate how they can be applied to configure and manage Docker services.

1. Brief description of Systemd and its functions:

Systemd is a system and service manager for Linux operating systems designed to provide a better framework for expressing dependencies, allowing more work to be done in parallel during system startup, and reducing computational costs associated with shell-based scripts. It introduces the concept of “unit” files to manage resources such as services, mount points, and devices. Systemd also provides aggressive parallelization, utilizes socket and D-Bus activation for service startup, offers on-demand daemon launching, monitors processes using Linux cgroups, supports system state snapshots and restoration, and manages mount points and automatic mounting. Its goal is to standardize service configuration and behavior across Linux distributions.

2. Changing Docker Service Startup: Integration and Configuration

Changing the startup of the Docker service via systemd involves integrating and configuring a unit file to manage the Docker service as a system service. This includes creating or modifying the file /etc/systemd/system/docker.service to configure Docker startup parameters, define service dependencies, and specify behavior during system startup. For example, Docker can be configured to start with specific command-line options or automatically restart in case of failure. After making changes, it is necessary to execute systemctl daemon-reload followed by systemctl restart docker.service to apply the new configuration.

When Docker is installed on a system with systemd, it typically provides a docker.service unit file that manages the Docker daemon. This file allows systemd to control the Docker process, including starting it on system boot, restarting it on failures, and stopping it on system shutdown. By setting Docker daemon startup parameters through ExecStart in the systemd override.conf file, as well as specifying listening parameters in daemon.json, Docker will attempt to use both sets of parameters. To avoid conflict, it is preferable to use only one method for configuring listened addresses for the Docker daemon — either via daemon.json or via ExecStart in the systemd unit file. Using daemon.json is more preferred if you plan to configure additional Docker daemon parameters. Systemd provides a mechanism for overriding and adding settings to existing unit files, achieved by creating a directory named after the unit file with a .d suffix in /etc/systemd/system/, where configuration files can be placed, such as override.conf

Example: Configuring override.conf for docker.service:

Create a directory for Docker override settings:

sudo mkdir -p /etc/systemd/system/docker.service.d

This command creates a directory named docker.service.d inside the /etc/systemd/system/ directory. This directory will hold the override configuration files for the Docker service.

Create a file named override.conf in this directory to modify or add settings:

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// -H tcp://0.0.0.0:4243

The first line ExecStart= resets any existing values that may have been defined in the main docker.service unit file. The second line sets a new startup command by adding TCP port listening.

Example of the docker.service unit file:

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target docker.socket
Requires=docker.socket

[Service]
Type=notify
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always

[Install]
WantedBy=multi-user.target

This file specifies that the Docker daemon should start after the network initialization (After=network.target) and depends on the Docker socket (Requires=docker.socket), which is used to manage Docker through its API. The daemon.json file is a configuration file for the Docker daemon, allowing you to configure various Docker operations such as logging, storage, and networking settings.

Example daemon.json:

{
  "debug": true,
  "log-level": "info",
  "storage-driver": "overlay2",
  "data-root": "/var/lib/docker",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  },
  "default-address-pools": [
    {"base": "10.10.0.0/16", "size": 24}
  ]
}

Systemd service management and Docker can interact with each other, providing system administrators with flexible tools for managing containers as system services. For example, you can create a unit file for systemd that will start a Docker container when the system boots. This allows integrating Docker containers directly into the system initialization process, ensuring their automatic startup, restart in case of failures, and proper shutdown along with the system.

Example unit file for launching a Docker container (myapp.service):

[Unit]
Description=My Docker Application
Requires=docker.service
After=docker.service

[Service]
Restart=always
ExecStart=/usr/bin/docker run --rm --name myapp myimage
ExecStop=/usr/bin/docker stop myapp

[Install]
WantedBy=multi-user.target

This configuration file creates a service named myapp, which, upon system startup, launches a Docker container from the myimage image and stops this container upon system shutdown. The integration of Docker with systemd demonstrates how modern process management and application containerization systems can work together, providing powerful and flexible tools for deploying and managing applications.

Configuring a docker-compose service through systemd

Creating a systemd service to manage your docker-compose project provides automation and simplifies the process of starting and stopping containers. You can create a file named myapp.service in /etc/systemd/system/ with the following content:

[Unit]
Description= Run Docker Compose project My App
Requires=docker.service
After=docker.service

[Service]
Type=simple
ExecStart=/usr/bin/docker-compose -f /path/to/your/docker-compose.yml up
ExecStop=/usr/bin/docker-compose -f /path/to/your/docker-compose.yml down
Restart=always

[Install]
WantedBy=multi-user.target

This allows for easy launching and stopping of docker-compose projects as system services.

Automated Docker Cleanup with systemd

To automate the Docker cleanup process, you can utilize systemd timers. systemd understands which service to activate using the timer mechanism. When you create a .timer file (in our case, docker-cleanup.timer), you configure systemd to wait for a specific period of time (here — 12 hours) before activating the corresponding service. The name of the service to be activated is determined by the timer’s name. Thus, docker-cleanup.timer automatically looks for a service with a similar name, i.e., docker-cleanup.service, and triggers it. Therefore, systemd “knows” that after the specified period in the timer, it needs to activate the docker-cleanup.service, which executes the cleanup command.

Create docker-cleanup.timer to define the cleanup interval and docker-cleanup.service to execute the cleanup command:

/etc/systemd/system/docker-cleanup.timer:

[Unit]
Description=Docker cleanup timer

[Timer]
OnUnitInactiveSec=12h
OnBootSec=10min

[Install]
WantedBy=timers.target

/etc/systemd/system/docker-cleanup.service:

[Unit]
Description=Docker cleanup
Requires=docker.service
After=docker.service

[Service]
Type=oneshot
ExecStart=/usr/bin/docker system prune -af

[Install]
WantedBy=multi-user.target

This set of files will allow you to automatically clean up unused Docker objects, freeing up system resources.

3. Detailed Overview of Unit Files in Systemd and Their Structure

Unit files in systemd are configuration files that describe the behavior of services, devices, mount points, timers, and other system resources. They have the extension .service for services and other extensions depending on the type (e.g., .socket, .device). Each unit file consists of sections: [Unit] with information about the service, [Service] with details of the service’s startup and shutdown, and [Install] for auto-start settings. Important directives include ExecStart for the service start command, ExecStop for stopping it, and Restart for restarting it in case of failures.

The structure of unit files in systemd is divided into sections, each containing specific directives:

[Unit] — the main section containing information about the unit, such as description, dependencies (Requires, After, Before), startup conditions (ConditionPathExists, ConditionKernelCommandLine, etc.).

[Service] — the section for services, defining the behavior of the service on startup and shutdown, the executable files used (ExecStart, ExecStop), the restart policy (Restart), service type (Type), environment settings (Environment), and much more.

[Install] — the section defining how the unit should be installed and activated, for example, which symbolic links to create when systemctl enable is executed.

Key Directives:

• ExecStart — the command to start the service. This is the main directive that defines what the unit launches.
• ExecStop — the command to stop the service. Used for the proper termination of the service.
• Restart — specifies whether systemd should attempt to restart the service in case of failure or stop.
• After and Before — control the order of unit startup, defining dependencies between them.

Let’s consider a simple example of a unit file for a service that runs a basic web application:

[Unit]
Description=My Web Application
After=network.target

[Service]
ExecStart=/usr/bin/python /path/to/app.py
Restart=always
Environment=VAR=value

[Install]
WantedBy=multi-user.target

In this example:

• Description provides a brief description of the service.
• After=network.target indicates that the service should be started after network initialization.
• In the [Service] section, ExecStart specifies the command to start the application, Restart=always instructs systemd to restart the service on failure, and Environment sets environment variables.
• The [Install] section with WantedBy=multi-user.target directive indicates that the service should be automatically started when transitioning to multi-user mode.

Options in the [Unit] section include:

• [Timer]: For timers that replace cron tasks. Controls the timing of service execution.
• [Mount]: For mount points. Describes file system mounting parameters.
• [Automount]: Automatic mounting when accessed.
• [Socket]: For sockets. Manages socket creation for IPC or network services.
• [Path]: For tracking changes in files or directories and launching services in response to those changes.
• [Slice]: For managing process groups.

4. Important directives in the sections [Unit], [Service], and [Install].

Key directives in the [Service] section are:

• ExecStart: Command to start the service.
• ExecStop: Command to stop the service.
• ExecReload: Command to reload the service configuration.
• Restart: Restart policy for the service upon unexpected termination.
• Type: Service type, defining how systemd interacts with the service (e.g., simple, forking, oneshot).
• Environment: Setting environment variables for the service.

[Service]
ExecStart=/usr/bin/myapp
ExecStop=/usr/bin/myapp --stop
ExecReload=/usr/bin/myapp --reload
Restart=on-failure
Type=simple
Environment="VAR1=value1" "VAR2=value2"

Examples:

• ExecStart defines the command to start the application.
• ExecStop specifies how to stop the application.
• ExecReload — command to reload configuration without fully stopping the service.
• Restart=on-failure restarts the service in case of its failure.
• Type=simple assumes that the main process is the start process.
• Environment sets environment variables for the service.

Section [Install]

• Defines the behavior of the unit when it is enabled or disabled, affecting the system boot process.
• WantedBy: Indicates the targets for which the current unit will be automatically started.
• Alias: Allows the unit to be available under another name.

[Install]
WantedBy=multi-user.target
Alias=mycustomservice.service

Examples:

• WantedBy=multi-user.target enables the service in the specified target, making it active during boot in multi-user mode.
• Alias=mycustomservice.service allows the service to be accessible under another name.

5. Applications and examples of using other sections, including [Timer].

[Timer]

The [Timer] section in a systemd unit file is used to configure timers that can activate services on a schedule or in response to specific events. It is a powerful tool for task scheduling in the system, similar to cron in Unix systems, but with additional features and integration with the systemd service management system.

Key features of the [Timer] section:

• Time-based task execution: Timers can be configured to run tasks at specific times, on specific days of the week, month, or year. This is useful for regular maintenance operations such as backups, system updates, and more.
• Event-based task execution: Timers can be configured to run tasks in response to file or system events, such as changes in the content of a directory or the availability of a network resource.
• Persistence: If a timer should trigger while the system was powered off, it can be set to execute tasks immediately upon the next system boot.
• Accuracy: Allows setting a minimum delay for the timer, useful for managing task execution precision and reducing system load when multiple tasks start simultaneously.

Examples of directives in the [Timer] section:

• OnCalendar=: Sets a schedule for the timer in calendar format. For example, OnCalendar=daily will trigger the task every day at midnight.
• OnActiveSec=: Sets the timer to trigger after a specified number of seconds after the timer is activated.
• OnBootSec=: Triggers the task after a specified number of seconds after system boot.
• OnUnitActiveSec=: Triggers the task after a specified number of seconds after the associated unit enters an active state.
• OnUnitInactiveSec=: Triggers the task after a specified number of seconds after the associated unit enters an inactive state.
• Persistent=: Specifies whether the timer should execute missed tasks upon the next activation.
• AccuracySec=: Sets the maximum acceptable deviation in the timer’s start time, useful for optimizing system resources.

[Unit]
Description=Ежедневное резервное копирование

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true

[Install]
WantedBy=timers.target

This timer activates the backup.service daily at 2 AM. If the system was powered off at the time the timer was supposed to trigger, the task will be executed immediately after the next system boot due to the Persistent=true option.

Systemd timers provide a flexible and powerful means of task scheduling, allowing system administrators to efficiently manage the execution of services and scripts according to a schedule or events.

[Mount]

[Mount]
What=/dev/sdb1
Where=/mnt/external
Type=ext4
Options=defaults

Examples:

• Mounts the partition /dev/sdb1 to /mnt/external with the ext4 file system.

[Socket]

[Socket]
ListenStream=8080
Accept=yes

Examples:

Opens a socket on port 8080 and uses Accept=yes for accepting connections.

Commonly used targets in systemd:

• basic.target: Serves as a synchronization point to achieve basic system functionality. All standard services necessary for the core operation of the system should be started before this target.
• multi-user.target: Equivalent to runlevel 3 in traditional SysVinit initialization systems. This target is used for multi-user systems without a graphical user interface.
• graphical.target: Equivalent to runlevel 5 in SysVinit, designed for systems with a graphical user interface.
• sockets.target: The target that needs to be reached before starting services that depend on sockets.
• timers.target: Used for organizing and managing systemd timers.
• paths.target: For services that are activated based on changes in the file system.
• remote-fs.target: For systems that need to wait for the mounting of all remote file systems before continuing the boot process.
• local-fs.target: Used to ensure that all local file systems are mounted before starting services that depend on them.
• swap.target: Ensures that all swap partitions are activated before continuing the boot process.
• sysinit.target: Used at the beginning of the system initialization process, before basic.target. This target includes tasks necessary to initialize system services before transitioning to user tasks.
• emergency.target: Intended for booting the system in emergency mode.
• rescue.target: Provides a minimal environment for system recovery.

These targets provide a flexible mechanism for defining the boot order and dependencies between different services and system resources. Using the After= and Before= directives in the unit file allows administrators to precisely control when and in what order services should start, based on their dependencies and the need for specific resources or services.

6. Network Configuration: Static IP, DHCP.

systemd-networkd is a component of systemd designed to manage network interfaces in Linux. It allows configuring network connections of various types, such as Ethernet, Wi-Fi, VLANs, bridges, and tunnels, using simple configuration files. systemd-networkd automatically applies these settings during system boot or when a new device is connected.

Setting up a static IP address:

To configure a static IP address for a network interface, create a configuration file in the directory /etc/systemd/network/. For example, 10-static-ethernet.network:

[Match]
Name=eth0

[Network]
Address=192.168.1.10/24
Gateway=192.168.1.1
DNS=192.168.1.1

Description:

• The [Match] section specifies which interfaces this configuration file applies to. In this case, it applies to the eth0 interface.
• The [Network] section specifies the network settings for this interface, including the static IP address, default gateway, and DNS servers.

Setting up DHCP:

To automatically configure a network interface using DHCP, create a file, for example, 20-dhcp.network:

[Match]
Name=eth0

[Network]
DHCP=yes

This configuration enables DHCP for the eth0 interface, allowing automatic retrieval of IP address, gateway, and DNS servers from the DHCP server.

7. Organization of the Systemd Filesystem: Directory Structure and Their Purposes.

Systemd Directory Structure

Systemd utilizes several directories for storing unit files, each with its own purpose:

• /lib/systemd/system/: Contains unit files provided by application and library packages. These files are considered part of the distribution and are typically not meant to be edited by the system administrator. Package updates may overwrite the contents of this directory.
• /etc/systemd/system/: Intended for unit files created or modified by the system administrator. Files in this directory take precedence over those in /lib/systemd/system/ and do not change during system package updates.
• /run/systemd/system/: Used for temporary unit files created during system operation. These files disappear after reboot.
• /usr/lib/systemd/system/: Similar to /lib/systemd/system/ and also used for storing system unit files provided by application and library packages. The difference in paths depends on the Linux distribution.
• /run/systemd/system/: Used for storing temporary unit files created during system operation. Unit files in this directory take precedence over files in other directories but are not preserved after reboot.
• /etc/systemd/user/: For user-level unit files rather than system-level ones. This allows users to manage their own services without requiring superuser privileges.
• ~/.config/systemd/user/: User-specific unit files allowing for personalized service management at the individual user account level.

Splitting unit files into “system” (in /lib/systemd/system/) and “user” (in /etc/systemd/system/) directories allows for flexibility in managing service configurations and their stability.

Ensuring Stability: Changes made to files in /lib/systemd/system/ may be lost during package updates that provide these unit files. Storing the original unit files in this directory ensures that system updates do not disrupt service operation due to configuration conflicts.

Flexibility in Configuration: Placing modified or administrator-created unit files in /etc/systemd/system/ allows for overriding system service settings without the risk of them being overwritten during updates. This also facilitates backup and relocation of user configurations.

Thus, a system administrator can alter the startup parameters of a service provided by the distribution. Instead of editing the original file in /lib/systemd/system/, one can create a file with the same name in /etc/systemd/system/ and define only the parameters that need to be changed. systemd automatically supplements the settings from /lib/systemd/system/ with those defined in /etc/systemd/system/, giving the latter priority.

To fully override the behavior of a service installed in the system, the administrator can create or modify a unit file in /etc/systemd/system/. This may include changing startup parameters, dependencies, environment, and other aspects of the service. Here’s how it can be done:

1. Copying the original unit file: Start by copying the original unit file from /lib/systemd/system/ to /etc/systemd/system/. This ensures that you have a complete copy of the initial configuration for editing.

cp /lib/systemd/system/my_service.service /etc/systemd/system/my_service.service

2. Editing the unit file: Edit the copied file in /etc/systemd/system/, modifying it according to the necessary settings. You can use any text editor for this.

3. Applying changes: After editing and saving the changes in the unit file, you need to reload the systemd configuration and restart the service:

systemctl daemon-reload
systemctl restart my_service.service

4. Enabling the service (if needed): If you want the service to start automatically when the system boots, use the command “enable”:

systemctl enable my_service.service

This approach allows for complete control over service behavior in the system, providing flexibility in managing services and their dependencies. Creating and modifying unit files in /etc/systemd/system/ ensures that your changes remain effective even after system updates, providing stable and predictable configuration for critical services.

The file named override.conf in the directory /etc/systemd/system/[service-name].service.d/ is a convention, not a strict requirement. You can use any other name for the override file as long as it has the .conf extension.

Systemd reads all configuration files with the .conf extension in the directory [service-name].service.d/, merging them in alphabetical order. This allows system administrators to create multiple override files for one service, separating different aspects of the configuration using different files for better organization. For example, one file may contain security-related overrides, while another may contain performance optimizations.

If you have a service called my_service.service, you can create the following structure in /etc/systemd/system/my_service.service.d/:

• 10-security.conf — a file containing security-related settings for the service.
• 20-performance.conf — a file with performance optimization settings for the service.

override.conf — a file with general overrides for the service.

Systemd combines these files in alphabetical order. This means that settings from 20-performance.conf will be applied after settings from 10-security.conf, and both of these files can override settings from the main unit file of the service and from override.conf if it exists.

This article was written based on the following documentation:

• Debian Systemd: Official systemd documentation on Debian Wiki
• DigitalOcean: In-depth understanding of systemd units and unit files on DigitalOcean
• Arch Linux: Systemd on ArchWiki

Follow on LinkedIn