Docker “no space left on device” INode and Overlay Problems
Sometimes when you are doing DevOps its hard to tell what is really going on, especially when modern deployment architectures are intended to be so abstracted.
$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 253544 392 253152 1% /dev
tmpfs 255820 509 255311 1% /run
/dev/xvda1 524288 520542 3746 100% /
tmpfs 255820 1 255819 1% /dev/shm
tmpfs 255820 3 255817 1% /run/lock
tmpfs 255820 16 255804 1% /sys/fs/cgroup
tmpfs 255826 4 255822 1% /run/user/112
tmpfs 255826 4 255822 1% /run/user/1000
So what this means is that you may have enough space on your device, but the filesystem does not have enough namespace resources for docker to manage its cache.
Jan 13 01:08:39 ip-172-31-20-231 systemd: Starting Docker Application Container Engine...
Jan 13 01:08:39 ip-172-31-20-231 docker: time="2017-01-13T01:08:39.430346584Z" level=error msg="Failed to GetDriver graph overlay2 /var/lib/docker"
Jan 13 01:08:39 ip-172-31-20-231 docker: time="2017-01-13T01:08:39.430742193Z" level=fatal msg="Error starting daemon: error initializing graphdriver: driver not supported"
Jan 13 01:08:39 ip-172-31-20-231 systemd: docker.service: Main process exited, code=exited, status=1/FAILURE
Jan 13 01:08:39 ip-172-31-20-231 systemd: Failed to start Docker Application Container Engine.
Jan 13 01:08:39 ip-172-31-20-231 systemd: docker.service: Unit entered failed state.
Jan 13 01:08:39 ip-172-31-20-231 systemd: docker.service: Failed with result 'exit-code'.
The Easy Fix in AWS, Change Docker Image Directory To Mounted Volume
So this could be done differently, some may argue better way. It is possible to expand the root device by snapshotting and then building a new AMI against an expanded snapshot size, but for me the easy fix was to add a volume on the instance and then point the docker home dir at my new 100G volume.
After you attach an Amazon EBS volume to your instance, it is exposed as a block device. You can format the volume with any file system and then mount it. After you make the EBS volume available for use, you can access it in the same ways that you access any other volume.
mount /dev/xvdf1 /docker
tar -cvf docker_20170112.tar /var/lib/docker
mv /var/lib/docker ./newmount/.
ln -s /docker/newmount/docker/ /var/lib/docker
service docker start
ls -al /docker/newmount/docker/
So there you have it, just cop out and do the easy thing! Why not, its easy to manage the space needed just for docker this way. If you have a good reason not to do it this way comments please!