Docker Data Volume Snapshots and Encryption with LVM and LUKS
Ask anyone I work with, I love Docker. It makes my job simpler in a lot of ways. One thing it doesn’t do so well out of the box is managing data. You get one volume driver: “local”. This means all your Docker volumes are directories in /var/lib/docker/volumes. Don’t get me wrong, they work just fine, but I had two needs that were not met: volume snapshots to facilitate backups, and at-rest volume encryption to satisfy data security requirements. It just so happens the Linux Logical Volume Manager (LVM) provides those features and more. (I know that ZFS is another option for these features, but my organization uses CentOS 7, and their kernel does not support ZFS.)
As you might expect, I’m not the first one to have theses needs. Some fine folks have put together a Docker LVM data volume plugin with snapshots and thin provisioning built right in. Thin provisioning is a must-have for snapshots, as over-allocation means they take up very little storage resources. I have submitted a pull to the docker-lvm-plugin project that adds the ability to encrypt your volumes with Linux Unified Key Setup (LUKS). For those interested in the cryptographic technologies is use, currently the “cryptsetup” tool defaults LUKS to a 256 bit keyed AES cipher in XTR mode.
See It in Action
I’m going to run through setting it all up on a CentOS 7 VM so you can see what is involved in making it happen. I followed this guide to set up the VM on my workstation. First I will set up the LVM storage, then I will install Docker, and finally I will demonstrate setting up LVM data volumes, including encryption, and making snapshots of the volumes.
LVM Setup
I added a second 50GB virtual SATA disk to my CentOS VM through the VirtualBox Manager.
[blake@linux-centos-67 ~]$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 8G 0 disk
├─sda1 8:1 0 500M 0 part /boot
└─sda2 8:2 0 7.5G 0 part
├─centos-root 253:0 0 6.7G 0 lvm /
└─centos-swap 253:1 0 820M 0 lvm [SWAP]
sdb 8:16 0 50G 0 disk
sr0 11:0 1 1024M 0 romI need two thin pools for Docker: one will be the storage driver for images, containers, etc. and the other will hold data volumes managed by docker-lvm-plugin. Docker provides good documentation on setting up LVM as the storage driver.
[blake@linux-centos-67 ~]$ sudo pvcreate /dev/sdb
Physical volume "/dev/sdb" successfully created
[blake@linux-centos-67 ~]$ sudo vgcreate docker /dev/sdb
Volume group "docker" successfully created
[blake@linux-centos-67 ~]$ sudo lvcreate -L 15G -T docker/internal
Logical volume "internal" created.
[blake@linux-centos-67 ~]$ sudo lvcreate -L 30G -T docker/volumes
Logical volume "volumes" created.
[blake@linux-centos-67 ~]$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 8G 0 disk
├─sda1 8:1 0 500M 0 part /boot
└─sda2 8:2 0 7.5G 0 part
├─centos-root 253:0 0 6.7G 0 lvm /
└─centos-swap 253:1 0 820M 0 lvm [SWAP]
sdb 8:16 0 50G 0 disk
├─docker-internal_tmeta 253:2 0 16M 0 lvm
│ └─docker-internal 253:4 0 15G 0 lvm
├─docker-internal_tdata 253:3 0 15G 0 lvm
│ └─docker-internal 253:4 0 15G 0 lvm
├─docker-volumes_tmeta 253:5 0 32M 0 lvm
│ └─docker-volumes 253:7 0 30G 0 lvm
└─docker-volumes_tdata 253:6 0 30G 0 lvm
└─docker-volumes 253:7 0 30G 0 lvm
sr0 11:0 1 1024M 0 romInstall Docker
I installed docker as instructed in the docs, adding the configuration to use the “docker-internal” thin pool for storage.
[blake@linux-centos-67 ~]$ sudo tee /etc/yum.repos.d/docker.repo <<-'EOF'
> [dockerrepo]
> name=Docker Repository
> baseurl=https://yum.dockerproject.org/repo/main/centos/7/
> enabled=1
> gpgcheck=1
> gpgkey=https://yum.dockerproject.org/gpg
> EOF
[dockerrepo]
name=Docker Repository
baseurl=https://yum.dockerproject.org/repo/main/centos/7/
enabled=1
gpgcheck=1
gpgkey=https://yum.dockerproject.org/gpg
[blake@linux-centos-67 ~]$ sudo yum install docker-engine
...
Installed:
docker-engine.x86_64 0:1.12.3-1.el7.centosDependency Installed:
audit-libs-python.x86_64 0:2.4.1-5.el7 checkpolicy.x86_64 0:2.1.12-6.el7 docker-engine-selinux.noarch 0:1.12.3-1.el7.centos libcgroup.x86_64 0:0.41-8.el7 libseccomp.x86_64 0:2.2.1-1.el7
libselinux-python.x86_64 0:2.2.2-6.el7 libsemanage-python.x86_64 0:2.1.10-18.el7 libtool-ltdl.x86_64 0:2.4.2-21.el7_2 policycoreutils-python.x86_64 0:2.2.5-20.el7 python-IPy.noarch 0:0.75-6.el7
setools-libs.x86_64 0:3.3.7-46.el7Complete!
[blake@linux-centos-67 ~]$ sudo mkdir /etc/docker
[blake@linux-centos-67 ~]$ sudo tee /etc/docker/daemon.json <<-'EOF'
> {
> "storage-driver": "devicemapper",
> "storage-opts": [
> "dm.thinpooldev=/dev/mapper/docker-internal",
> "dm.use_deferred_removal=true",
> "dm.use_deferred_deletion=true"
> ]
> }
> EOF
{
"storage-driver": "devicemapper",
"storage-opts": [
"dm.thinpooldev=/dev/mapper/docker-internal",
"dm.use_deferred_removal=true",
"dm.use_deferred_deletion=true"
]
}
[blake@linux-centos-67 ~]$ sudo systemctl enable docker.service
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
[blake@linux-centos-67 ~]$ sudo systemctl start docker
[blake@linux-centos-67 ~]$ sudo docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 1.12.3
Storage Driver: devicemapper
Pool Name: docker-internal
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file:
Metadata file:
Data Space Used: 11.8 MB
Data Space Total: 16.11 GB
Data Space Available: 16.09 GB
Metadata Space Used: 102.4 kB
Metadata Space Total: 16.78 MB
Metadata Space Available: 16.67 MB
Thin Pool Minimum Free Space: 1.611 GB
Udev Sync Supported: true
Deferred Removal Enabled: true
Deferred Deletion Enabled: true
Deferred Deleted Device Count: 0
Library Version: 1.02.107-RHEL7 (2016-06-09)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-327.36.3.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 993 MiB
Name: linux-centos-67.clarabridge.net
ID: JBH2:5RQU:N5V2:SLXB:HZZO:2QE4:3JH6:ZTVT:USI7:LJGC:ON3C:BM2U
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
127.0.0.0/8
[blake@linux-centos-67 ~]$ sudo docker run --rm hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
c04b14da8d14: Pull complete
Digest: sha256:0256e8a36e2070f7bf2d0b0763dbabdd67798512411de4cdcf9431a1feb60fd9
Status: Downloaded newer image for hello-world:latestHello from Docker!
This message shows that your installation appears to be working correctly.To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bashShare images, automate workflows, and more with a free Docker Hub account:
https://hub.docker.comFor more examples and ideas, visit:
https://docs.docker.com/engine/userguide/
Install docker-lvm-plugin
Currently docker-lvm-plugin is only distributed as GO source, so you will need to compile it yourself.
[blake@linux-centos-67 ~]$ sudo yum install golang git golang-github-cpuguy83-go-md2man
...
Installed:
golang.x86_64 0:1.6.3-1.el7_2.1 git.x86_64 0:1.8.3.1-6.el7_2.1 golang-github-cpuguy83-go-md2man.x86_64 0:1.0.4-2.el7_2Dependency Installed:
cpp.x86_64 0:4.8.5-4.el7 gcc.x86_64 0:4.8.5-4.el7 glibc-devel.x86_64 0:2.17-106.el7_2.8 glibc-headers.x86_64 0:2.17-106.el7_2.8 golang-bin.x86_64 0:1.6.3-1.el7_2.1 golang-src.noarch 0:1.6.3-1.el7_2.1
kernel-headers.x86_64 0:3.10.0-327.36.3.el7 libmpc.x86_64 0:1.0.1-3.el7 mpfr.x86_64 0:3.1.1-4.el7
libgnome-keyring.x86_64 0:3.8.0-3.el7 perl.x86_64 4:5.16.3-286.el7 perl-Carp.noarch 0:1.26-244.el7 perl-Encode.x86_64 0:2.51-7.el7 perl-Error.noarch 1:0.17020-2.el7
perl-Exporter.noarch 0:5.68-3.el7 perl-File-Path.noarch 0:2.09-2.el7 perl-File-Temp.noarch 0:0.23.01-3.el7 perl-Filter.x86_64 0:1.49-3.el7 perl-Getopt-Long.noarch 0:2.40-2.el7
perl-Git.noarch 0:1.8.3.1-6.el7_2.1 perl-HTTP-Tiny.noarch 0:0.033-3.el7 perl-PathTools.x86_64 0:3.40-5.el7 perl-Pod-Escapes.noarch 1:1.04-286.el7 perl-Pod-Perldoc.noarch 0:3.20-4.el7
perl-Pod-Simple.noarch 1:3.28-4.el7 perl-Pod-Usage.noarch 0:1.63-3.el7 perl-Scalar-List-Utils.x86_64 0:1.27-248.el7 perl-Socket.x86_64 0:2.010-3.el7 perl-Storable.x86_64 0:2.45-3.el7
perl-TermReadKey.x86_64 0:2.30-20.el7 perl-Text-ParseWords.noarch 0:3.29-4.el7 perl-Time-HiRes.x86_64 4:1.9725-3.el7 perl-Time-Local.noarch 0:1.2300-2.el7 perl-constant.noarch 0:1.27-2.el7
perl-libs.x86_64 4:5.16.3-286.el7 perl-macros.x86_64 4:5.16.3-286.el7 perl-parent.noarch 1:0.225-244.el7 perl-podlators.noarch 0:2.5.1-3.el7 perl-threads.x86_64 0:1.87-4.el7
perl-threads-shared.x86_64 0:1.43-6.el7 rsync.x86_64 0:3.0.9-17.el7Complete!
[blake@linux-centos-67 ~]$ mkdir go
[blake@linux-centos-67 ~]$ export GOPATH=${HOME}/go
[blake@linux-centos-67 ~]$ go get github.com/kalahari/docker-lvm-plugin
[blake@linux-centos-67 ~]$ cd $GOPATH/src/github.com/kalahari/docker-lvm-plugin
[blake@linux-centos-67 docker-lvm-plugin]$ git fetch origin
[blake@linux-centos-67 docker-lvm-plugin]$ git checkout feature/LUKS
Branch feature/LUKS set up to track remote branch feature/LUKS from origin.
Switched to a new branch 'feature/LUKS'
[blake@linux-centos-67 docker-lvm-plugin]$ go get ./...
# github.com/kalahari/docker-lvm-plugin/vendor/github.com/Microsoft/go-winio
vendor/github.com/Microsoft/go-winio/file.go:45: undefined: syscall.Overlapped
[blake@linux-centos-67 docker-lvm-plugin]$ make
go-md2man -in man/docker-lvm-plugin.8.md -out docker-lvm-plugin.8
/usr/bin/go build -o docker-lvm-plugin .
[blake@linux-centos-67 docker-lvm-plugin]$ sudo make install
install -D -m 644 etc/docker/docker-lvm-plugin /etc/docker/docker-lvm-plugin
install -D -m 644 systemd/docker-lvm-plugin.service /usr/lib/systemd/system/docker-lvm-plugin.service
install -D -m 644 systemd/docker-lvm-plugin.socket /usr/lib/systemd/system/docker-lvm-plugin.socket
install -D -m 755 docker-lvm-plugin /usr/libexec/docker/docker-lvm-plugin
install -D -m 644 docker-lvm-plugin.8 /usr/share/man/man8/docker-lvm-plugin.8
Note that I installed docker-lvm-plugin from my own fork, using the branch with the LUKS feature. Once my pull request is accepted by the maintainers you will be able to install from their repo. You can safely ignore the error on the “go get” command that installs dependencies.
The only configuration docker-lvm-plugin needs is the name of the volume group it should use.
[blake@linux-centos-67 ~]$ sudo perl -pi -e 's/VOLUME_GROUP=$/VOLUME_GROUP=docker/' /etc/docker/docker-lvm-plugin
[blake@linux-centos-67 ~]$ cat /etc/docker/docker-lvm-plugin
# Update volume group (vg) name in this file. A volume group name is needed
# to create logical volumes using docker lvm plugin. You can choose an existing
# volume group name by listing volume groups on your system using `vgs` command
# OR create a new volume group using `vgcreate` command.
# e.g. vgcreate volume_group_one /dev/hda where /dev/hda is your partition or
# whole disk on which physical volumes were created.
VOLUME_GROUP=docker
[blake@linux-centos-67 ~]$ sudo systemctl enable docker-lvm-plugin
Created symlink from /etc/systemd/system/multi-user.target.wants/docker-lvm-plugin.service to /usr/lib/systemd/system/docker-lvm-plugin.service.
[blake@linux-centos-67 ~]$ sudo systemctl start docker-lvm-pluginCreate an LVM Data Volume
Now that docker-lvm-plugin is running, we can create LVM data volumes and attach them to containers.
[blake@linux-centos-67 ~]$ sudo docker volume create --driver lvm --name example1 --opt size=100M --opt thinpool=volumes
example1
[blake@linux-centos-67 ~]$ sudo docker run -d --name example1 -v example1:/mnt/example1 centos:7 tail -f /dev/null
Unable to find image 'centos:7' locally
7: Pulling from library/centos
08d48e6f1cff: Pull complete
Digest: sha256:b2f9d1c0ff5f87a4743104d099a3d561002ac500db1b9bfa02a783a46e0d366c
Status: Downloaded newer image for centos:79a549e2b8305932f603d97887572aa02ba4aaf3d9fb4dd57246212452dd7501c
[blake@linux-centos-67 ~]$ sudo docker exec -it example1 bash -c 'echo foo > /mnt/example1/foo.txt'
[blake@linux-centos-67 ~]$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 8G 0 disk
├─sda1 8:1 0 500M 0 part /boot
└─sda2 8:2 0 7.5G 0 part
├─centos-root 253:0 0 6.7G 0 lvm /
└─centos-swap 253:1 0 820M 0 lvm [SWAP]
sdb 8:16 0 50G 0 disk
├─docker-internal_tmeta 253:2 0 16M 0 lvm
│ └─docker-internal 253:4 0 15G 0 lvm
│ └─docker-253:0-1222218-d83d27759c43c5deb055395179c51d9a4d6c7b7eb190addb991ca1f60bbd975c 253:10 0 10G 0 dm /var/lib/docker/devicemapper/mnt/d83d27759c43c5deb055395179c51d9a4d6c7b7eb190addb991ca1f60bbd975c
├─docker-internal_tdata 253:3 0 15G 0 lvm
│ └─docker-internal 253:4 0 15G 0 lvm
│ └─docker-253:0-1222218-d83d27759c43c5deb055395179c51d9a4d6c7b7eb190addb991ca1f60bbd975c 253:10 0 10G 0 dm /var/lib/docker/devicemapper/mnt/d83d27759c43c5deb055395179c51d9a4d6c7b7eb190addb991ca1f60bbd975c
├─docker-volumes_tmeta 253:5 0 32M 0 lvm
│ └─docker-volumes-tpool 253:7 0 30G 0 lvm
│ ├─docker-volumes 253:8 0 30G 0 lvm
│ └─docker-example1 253:9 0 100M 0 lvm /var/lib/docker-lvm-plugin/example1
└─docker-volumes_tdata 253:6 0 30G 0 lvm
└─docker-volumes-tpool 253:7 0 30G 0 lvm
├─docker-volumes 253:8 0 30G 0 lvm
└─docker-example1 253:9 0 100M 0 lvm /var/lib/docker-lvm-plugin/example1
sr0 11:0 1 1024M 0 romHere I created a 100MB thin volume named “example1” and attached it to a container with the same name. I use “tail -f /dev/null” to keep the container running in the background. Then I wrote some text to a file on the data volume. You can see my container in the “docker/internal” thin pool and my volume in the “docker/volumes” thin pool.
Snapshot
Now I’ll take a snapshot of the volume, and show that even when the contents of the volume change, the snapshot stays constant.
[blake@linux-centos-67 ~]$ sudo docker volume create --driver lvm --name example1_snapshot --opt snapshot=example1
example1_snapshot
[blake@linux-centos-67 ~]$ sudo docker exec -it example1 cat /mnt/example1/foo.txt
foo
[blake@linux-centos-67 ~]$ sudo docker exec -it example1 bash -c 'echo bar > /mnt/example1/foo.txt'
[blake@linux-centos-67 ~]$ sudo docker exec -it example1 cat /mnt/example1/foo.txt
bar
[blake@linux-centos-67 ~]$ sudo docker run -it --rm -v example1_snapshot:/mnt/example1 centos:7 cat /mnt/example1/foo.txt
fooEncryption
To create an encrypted volume, you will need a key file. This file contains a binary passphrase used to unlock your encrypted data volume. I like to keep my key file in a RAM disk, this way it will not be stored anywhere on the system. You will need to arrange for storage of your key file, perhaps encrypted on a thumb drive, making it available when creating an encrypted data volume, and when creating a container that uses the volume. Each encrypted volume can also use a different key file if you choose.
[blake@linux-centos-67 ~]$ sudo mkdir -p /var/run/key
[blake@linux-centos-67 ~]$ sudo mount -t ramfs -o size=4M ramfs /var/run/key
[blake@linux-centos-67 ~]$ sudo dd if=/dev/urandom of=/var/run/key/luks-volume-example.bin bs=1024 count=16
16+0 records in
16+0 records out
16384 bytes (16 kB) copied, 0.00336676 s, 4.9 MB/s
[blake@linux-centos-67 ~]$ sudo yum install cryptsetup
...
Installed:
cryptsetup.x86_64 0:1.6.7-1.el7Complete!
[blake@linux-centos-67 ~]$ sudo docker volume create --driver lvm --name example2 --opt size=100M --opt thinpool=volumes --opt crypt=luks --opt keyfile=/var/run/key/luks-volume-example.bin
example2
[blake@linux-centos-67 ~]$ sudo docker run -d --name example2 -v example2:/mnt/example2 centos:7 tail -f /dev/null
3e353925b7829c7fc580c3431a2d552db9a59e01abd5ae7ca56ed07aba3a3b2a
[blake@linux-centos-67 ~]$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 8G 0 disk
├─sda1 8:1 0 500M 0 part /boot
└─sda2 8:2 0 7.5G 0 part
├─centos-root 253:0 0 6.7G 0 lvm /
└─centos-swap 253:1 0 820M 0 lvm [SWAP]
sdb 8:16 0 50G 0 disk
├─docker-internal_tmeta 253:2 0 16M 0 lvm
│ └─docker-internal 253:4 0 15G 0 lvm
│ ├─docker-253:0-1222218-d83d27759c43c5deb055395179c51d9a4d6c7b7eb190addb991ca1f60bbd975c 253:10 0 10G 0 dm /var/lib/docker/devicemapper/mnt/d83d27759c43c5deb055395179c51d9a4d6c7b7eb190addb991ca1f60bbd975c
│ └─docker-253:0-1222218-b561ff9657c6b65e7485b309a6107e29151676c0cffbda97e8c0944661b11510 253:13 0 10G 0 dm /var/lib/docker/devicemapper/mnt/b561ff9657c6b65e7485b309a6107e29151676c0cffbda97e8c0944661b11510
├─docker-internal_tdata 253:3 0 15G 0 lvm
│ └─docker-internal 253:4 0 15G 0 lvm
│ ├─docker-253:0-1222218-d83d27759c43c5deb055395179c51d9a4d6c7b7eb190addb991ca1f60bbd975c 253:10 0 10G 0 dm /var/lib/docker/devicemapper/mnt/d83d27759c43c5deb055395179c51d9a4d6c7b7eb190addb991ca1f60bbd975c
│ └─docker-253:0-1222218-b561ff9657c6b65e7485b309a6107e29151676c0cffbda97e8c0944661b11510 253:13 0 10G 0 dm /var/lib/docker/devicemapper/mnt/b561ff9657c6b65e7485b309a6107e29151676c0cffbda97e8c0944661b11510
├─docker-volumes_tmeta 253:5 0 32M 0 lvm
│ └─docker-volumes-tpool 253:7 0 30G 0 lvm
│ ├─docker-volumes 253:8 0 30G 0 lvm
│ ├─docker-example1 253:9 0 100M 0 lvm /var/lib/docker-lvm-plugin/example1
│ ├─docker-example1_snapshot 253:11 0 100M 0 lvm
│ └─docker-example2 253:12 0 100M 0 lvm
│ └─luks-example2 253:14 0 98M 0 crypt /var/lib/docker-lvm-plugin/example2
└─docker-volumes_tdata 253:6 0 30G 0 lvm
└─docker-volumes-tpool 253:7 0 30G 0 lvm
├─docker-volumes 253:8 0 30G 0 lvm
├─docker-example1 253:9 0 100M 0 lvm /var/lib/docker-lvm-plugin/example1
├─docker-example1_snapshot 253:11 0 100M 0 lvm
└─docker-example2 253:12 0 100M 0 lvm
└─luks-example2 253:14 0 98M 0 crypt /var/lib/docker-lvm-plugin/example2
sr0 11:0 1 1024M 0 rom
[blake@linux-centos-67 ~]$ sudo cryptsetup luksDump /dev/docker/example2
LUKS header information for /dev/docker/example2Version: 1
Cipher name: aes
Cipher mode: xts-plain64
Hash spec: sha1
Payload offset: 4096
MK bits: 256
MK digest: 51 e5 ad 1d 28 ef 31 79 d0 54 6c b7 23 ec 01 99 6f 6c d2 b5
MK salt: c0 64 c3 7e bd 83 61 9c 09 75 f5 92 3a 6e 34 63
6f 90 cc 8b f8 1b 60 f1 73 01 a8 72 34 7e b6 97
MK iterations: 66000
UUID: c03a7de8-419b-4153-a99a-287d8f9382edKey Slot 0: ENABLED
Iterations: 262026
Salt: 9f a8 bf 2d 5e e1 ab 5b 0b 1a c4 24 ee bb b9 a7
ea af 14 00 19 25 36 3c d5 92 d1 29 b9 56 a9 64
Key material offset: 8
AF stripes: 4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED
Now I have an encrypted data volume mounted into a container. You can snapshot an encrypted volume, just like the plain volume. It will use the same key file as the source volume, because the snapshot happens below the encryption.
If you have any feedback, leave it below, or ping me on twitter.
