Speed up docker build with c5d instance family using NVMe based instance store.
It is good practice in the CI process to reuse docker cache as much as possible and hight iowait is a common problem on build instance in this case when running many dockers build in parallels for example.
Let’s me share stories about increasing CI build without affecting the budget. I’ve decided to change instance type c5.xlarge → c5d.xlarge. In the first look, you can say that it definitely affects the budget as they have a different price but let’s describe it in more details.
C5.xlarge cost $0.17 per Hour
C5.dxlarge cost $0,192 USD per Hour
0,192–0.17 = $0.022 USD per Hourc5dx.large on 2 cents more expensive then c5.xlarge per hour but with c5d instance family, you will get100 GB of NVMe based SSD instance store.
As GP2 EBS per GB per month cost 0,119 USD it will cost $0,020 for 100 GB per Hour.
0,119 *100/30/24 = $0,020 USD per hourAs you can see, using c5d.xlarge you just pay for instance store the same price as it would be EBS GP2 volume with the same size. Instance store is ephemeral and due to this it is not good for root volume, storing important date or artifacts but it definitely a good place for some cache because at least, it is really FAST, or just much faster than GP2.

I suppose you already knew how to mount the volume, if not, please follow this guide.
Next Step — change docker root directory to a new volume. Create file
vim /etc/docker/daemon.json and add to it a few lines with the name of a directory which is the mount point for NVMe volume( in my case I’ve mounted the device to /volume dir):
{"data-root": "/volume/docker"}
Save changes and reload docker daemon:
sudo systemctl restart dockerOK, it is time to measure disk performance. Fio was used in my case. You can install fio with a standard Linux package manager. Example for Debian-based system:
sudo apt install fioI’ve run fio command on the directory which is on 100 GB NMVMe and GP2 SSD with the same size.
fio — randrepeat=1 — ioengine=libaio — direct=1 — gtod_reduce=1 — name=test — filename=random_read_write.fio — bs=4k — iodepth=64 — size=4G — readwrite=randrw — rwmixread=75The results are in IOPs:
100 GB NVMe based SSD:
read iops: 42649
write iops: 14248
As you can see, we have in 40-times faster storage and positive impact on build time with minimal effort, without increasing budget and any burst of IOPs.
Another good article which describes IOPs as a bottleneck: https://pythonspeed.com/articles/slow-ci-aws-ec2/
Thanks for reading.
