Open in app

Sign in

Write

Sign in

IBM Cloud Private — Configure direct-lvm mode for built-in Docker package

Peng Wang
5 min readApr 19, 2018

--

Device Mapper is a kernel-based framework that underpins many advanced volume management technologies on Linux. If you use Docker EE on RHEL, CentOS, or Oracle Linux, you must use the devicemapperstorage driver.

Prerequisites

  • devicemapperstorage driver is the only supported storage driver for Docker EE and Commercially Supported Docker Engine (CS-Engine) on RHEL, CentOS, and Oracle Linux.
  • devicemapperis also supported on Docker CE running on CentOS, Fedora, Ubuntu, or Debian.
  • Changing the storage driver makes any containers you have already created inaccessible on the local system. Use docker save to save containers, and push existing images to Docker Hub or a private repository, so that you not need to re-create them later.

Configure direct-lvm mode for docker production environment
Production hosts using the devicemapperstorage driver must use direct-lvm mode. This mode uses block devices to create the thin pool. This is faster than using loopback devices, uses system resources more efficiently, and block devices can grow as needed. However, more set-up is required than loop-lvm mode.

Configure direct-lvm mode automatically:

  1. Install the following packages:
    — RHEL / CentOS: device-mapper-persistent-data, lvm2, and all dependencies
    — Ubuntu / Debian: thin-provisioning-tools, lvm2, and all dependencies

2. ICP build-in Docker can manage the block device for you, simplifying configuration of direct-lvmmode. Like the following configuration in /etc/docker/daemon.json. This is appropriate for fresh Docker set-ups only.

{
  "storage-opts": [
    "dm.directlvm_device=/dev/vdb",
    "dm.thinp_percent=95",
    "dm.thinp_metapercent=1",
    "dm.thinp_autoextend_threshold=80",
    "dm.thinp_autoextend_percent=20",
    "dm.directlvm_device_force=false"
  ]
}

3. Install ICP build-in docker.

4. Verify that Docker is using the new configuration using docker info.

$ docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 17.09.0-ce
Storage Driver: devicemapper
 Pool Name: docker-thinpool
 Pool Blocksize: 524.3kB
 Base Device Size: 10.74GB
 Backing Filesystem: xfs
 Data file: 
 Metadata file: 
 Data Space Used: 20.45MB
 Data Space Total: 10.2GB
 Data Space Available: 10.18GB
 Metadata Space Used: 49.15kB
 Metadata Space Total: 104.9MB
 Metadata Space Available: 104.8MB
 Thin Pool Minimum Free Space: 1.019GB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.135-RHEL7 (2016-09-28)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-327.10.1.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.2 (Maipo)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.703GiB
Name: test03.novalocal
ID: OCFC:TPGC:QYMC:6S6Z:OIH3:H53K:ZTDS:JCNK:QWYL:VIJ2:IO2N:QPW4
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Configure direct-lvm mode manually:

  1. Identify the block device you want to use. The device is located under /dev/ (such as /dev/vdb) and needs enough free space to store the images and container layers for the workloads that host runs. A solid state drive is ideal.
  2. Stop Docker.
    $ sudo systemctl stop docker
  3. Install the following packages:
    — RHEL / CentOS: device-mapper-persistent-data, lvm2, and all dependencies
    — Ubuntu / Debian: thin-provisioning-tools, lvm2, and all dependencies
  4. Create a physical volume on your block device from step 1, using the pvcreate command.
$ sudo pvcreate /dev/vdb
  Physical volume "/dev/vdb" successfully created.

5. Create a docker volume group on the same device, using the vgcreate command.

$ sudo vgcreate docker /dev/vdb
  Volume group "docker" successfully created

6. Create two logical volumes named thinpool and thinpoolmeta using the lvcreate command. The last parameter specifies the amount of free space to allow for automatic expanding of the data or metadata if space runs low, as a temporary stop-gap. These are the recommended values.

$ sudo lvcreate --wipesignatures y -n thinpool docker -l 95%VG
  Logical volume "thinpool" created.
$ sudo lvcreate --wipesignatures y -n thinpoolmeta docker -l 1%VG
  Logical volume "thinpoolmeta" created.

7. Convert the volumes to a thin pool and a storage location for metadata for the thin pool, using the lvconvert command.

$ sudo lvconvert -y --zero n -c 512K --thinpool docker/thinpool --poolmetadata docker/thinpoolmeta
  WARNING: Converting logical volume docker/thinpool and docker/thinpoolmeta to thin pool's data and metadata volumes with metadata wiping.
  THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
  Converted docker/thinpool to thin pool.

8. Configure autoextension of thin pools via an lvm profile. Specify thin_pool_autoextend_threshold and thin_pool_autoextend_percent values. The example below adds 20% more capacity when the disk usage reaches 80%.

$ sudo cat /etc/lvm/profile/docker-thinpool.profile
activation {
  thin_pool_autoextend_threshold=80
  thin_pool_autoextend_percent=20
}

9. Apply the LVM profile, using the lvchange command.

$ sudo lvchange --metadataprofile docker-thinpool docker/thinpool
  Logical volume docker/thinpool changed.

10. Enable monitoring for logical volumes on your host. Without this step, automatic extension does not occur even in the presence of the LVM profile.

$ sudo lvs -o+seg_monitor
  LV       VG     Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Monitor  
  thinpool docker twi-a-t--- 19.00g             0.00   0.03                             monitored

11. If you have ever run Docker on this host before, or if /var/lib/docker/ exists, move it out of the way so that Docker can use the new LVM pool to store the contents of image and containers.

$ mkdir /var/lib/docker.bk
$ mv /var/lib/docker/* /var/lib/docker.bk

12. Edit /etc/docker/daemon.json and configure the options needed for the devicemapper storage driver. If the file was previously empty, it should now contain the following contents

{
    "storage-driver": "devicemapper",
    "storage-opts": [
    "dm.thinpooldev=/dev/mapper/docker-thinpool",
    "dm.use_deferred_removal=true",
    "dm.use_deferred_deletion=true"
    ]
}

Or append the following configuration for ExecStart in /lib/systemd/system/docker.service

--storage-driver=devicemapper --storage-opt dm.thinpooldev=/dev/mapper/docker-thinpool --storage-opt dm.use_deferred_removal=true --storage-opt dm.use_deferred_deletion=true

13. Start Docker.

$ sudo systemctl daemon-reload
$ sudo systemctl start docker

14. Verify that Docker is using the new configuration using docker info.

$ docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 17.09.0-ce
Storage Driver: devicemapper
 Pool Name: docker-thinpool
 Pool Blocksize: 524.3kB
 Base Device Size: 10.74GB
 Backing Filesystem: xfs
 Data file: 
 Metadata file: 
 Data Space Used: 20.45MB
 Data Space Total: 20.4GB
 Data Space Available: 20.38GB
 Metadata Space Used: 61.44kB
 Metadata Space Total: 213.9MB
 Metadata Space Available: 213.8MB
 Thin Pool Minimum Free Space: 2.039GB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.135-RHEL7 (2016-09-28)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-327.10.1.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.3 (Maipo)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.703GiB
Name: icp03.novalocal
ID: P6H5:VIS3:6WH6:KWSM:ZK7D:D6PO:OA6D:IV7C:2NG2:I2SQ:NMC5:5VZJ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

If Docker is configured correctly, the Data file and Metadata file is blank, and the pool name is docker-thinpool.

Docker

--

--

Peng Wang

Written by Peng Wang

0 Followers

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams