No space left on device @ dir_s_mkdir, your server UP again in five minutes
The secret of inodes..
Did you wake up with your server down and the inbox full of AirBrake/NewRelic notifications about:
No space left on device @ dir_s_mkdir..
Yes, you’r right, the partition is not full! Then why?
It means you’re running out of Inodes. If you ever run into such trouble most likely you have too many small or 0-sized files on your disk (maybe cache), and while you have enough disk space, you have exhausted all available Inodes.
How can you identify the reason, in few steps:
# Check the available disk space
$ df -h
# Check the available Inodes
$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 958749 386 958363 1% /dev
tmpfs 961714 276 961438 1% /run
/dev/xvda1 524288 179858 344430 35% /
none 961714 2 961712 1% /sys/fs/cgroup
none 961714 4 961710 1% /run/lock
none 961714 1 961713 1% /run/shm
none 961714 2 961712 1% /run/user
/dev/xvdb 1966080 1555499 410581 80% /mnt
# Localize the files
$ for i in /*; do echo $i; find $i |wc -l; done
and localize the specific directory
$ for i in /home/*; do echo $i; find $i |wc -l; done
Cool, check the results with df -i command!