Orient Me: Rethink your Docker storage driver configuration

Nico Meisenzahl
Sep 4, 2017 · 3 min read

I did some research around Docker storage drivers, CfC and Orient Me the last days because I had some issues in different Orient Me installations. Finally, I came to the conclusion that the Docker storage driver configuration which is installed by the Orient Me setup might need to be customized for a production environment.

Why?

An Orient Me installation uses the overlay storage driver which is the default configuration when installing Docker from the official repository using yum. The overlay storage driver is default since Docker 1.13.1 (if the kernel supports it). The support for overlay was also added with RHEL 7.3 which is a requirement for running Orient Me. But, overlay with REHL 7.3 is only supported with XFS as the backing filesystem. This might be the first hook for some installations out there. But more important: If you check the Docker documentation you will find the information that devicemapper is the recommended and tested/supported storage driver which should be used with RHEL/CentOS and Docker CE. The same is mentioned for Docker EE. While researching this I got the same response from the IBM Cloud private team. They mentioned that devicemapper storage driver should be used with CfC/ICp on RHEL or CentOS.

After changing the storage driver to devicemapper on one of my test environment two of my known issues were fixed:

  • Issues related to CFC indices cleaner cronjob scheduling (Stoeps blogged about it here)
  • Increased CPU usage of the Docker daemon (dockerd) process

How to?

There is no migration path available which allows you to move your containers and images from the overlay to the devicemapper storage driver. Therefore the fastest solution is to reinstall Orient Me services and CfC. To uninstall Orient Me please use the deployCfC.sh which will also uninstall Docker itself.

Reinstall Docker-CE 17.03 (or the supported EE version) and all dependencies which are required to run CfC 1.1:

yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager — add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum-config-manager — disable docker-ce-edge
yum-config-manager — enable docker-ce-stable
yum makecache fast
yum -y — setopt=obsoletes=0 install docker-ce-17.03.2.ce-1.el7.centos
systemctl enable docker.service
systemctl start docker

Check if Docker is installed and running:

systemctl status docker
docker info
docker run hello-world

Prepare Docker:

  • Stop the docker daemon using systemctl stop docker
  • Backup and move your /var/lib/docker directory

Configure the volume:
Because of performance reasons do not use loop-lvm in production. I will describe how to configure direct-lvm. Therefore you need to add or pick a physical device which we will use as thinpool to store your containers and images. In my case, it will be /dev/sdb.

pvcreate /dev/sdb
vgcreate docker /dev/sdb
lvcreate — wipesignatures y -n thinpool docker -l 95%VG
lvcreate — wipesignatures y -n thinpoolmeta docker -l 1%VG
lvconvert -y — zero n -c 512K — thinpool docker/thinpool — poolmetadata docker/thinpoolmeta

You can configure to extend the logical volumes automatically if needed. You will find more information here.

Reconfigure Docker:
With Docker 17.06 there is simplified configuration available. In our case, we need to configure direct-lvm manually. Create or edit /etc/docker/daemon.json with following content:

{
“storage-driver”: “devicemapper”,
“storage-opts”: [
“dm.thinpooldev=/dev/mapper/docker-thinpool”,
“dm.use_deferred_removal=true”
]
}

Please add dm.use_deferred_deletion=true in case you are using a 4.x kernel.

Start Docker and verify your configuration:

systemctl start docker
docker info
docker run hello-world

Please verify following docker info output:

Storage Driver: devicemapper
Pool Name: docker-thinpool
Pool Blocksize: 524.3 kB
Base Device Size: 10.74 GB
Data file:
Metadata file:

Reinstall CfC and Orient Me using deployCfC.sh:
You need to include — skip_docker_deployment to be sure that the installer will not reinstall Docker again.

Update 17/09/28

IBM will officially support the devicemapper storage driver with future Orient Me versions as mentioned here.

ICS & IBM Connections

Stories related to IBM Connections and other ICS topics by Nico Meisenzahl.

Nico Meisenzahl

Written by

Senior Consultant @panagenda, @soccnx & @DockerRosenheim team member. Blogger, speaker & IBM Champion. Loves DevOps, K8s. His desk is a ping pong table.

ICS & IBM Connections

Stories related to IBM Connections and other ICS topics by Nico Meisenzahl.