No space left on device @ dir_s_mkdir, your server UP again in five minutes

The secret of inodes..

Did you wake up with your server down and the inbox full of AirBrake/NewRelic notifications about:

No space left on device @ dir_s_mkdir..

Yes, you’r right, the partition is not full! Then why?

It means you’re running out of Inodes. If you ever run into such trouble most likely you have too many small or 0-sized files on your disk (maybe cache), and while you have enough disk space, you have exhausted all available Inodes.

How can you identify the reason, in few steps:

# Check the available disk space

$ df -h 

# Check the available Inodes

$ df -i
Filesystem      Inodes   IUsed  IFree IUse% Mounted on
udev 958749 386 958363 1% /dev
tmpfs 961714 276 961438 1% /run
/dev/xvda1 524288 179858 344430 35% /
none 961714 2 961712 1% /sys/fs/cgroup
none 961714 4 961710 1% /run/lock
none 961714 1 961713 1% /run/shm
none 961714 2 961712 1% /run/user
/dev/xvdb 1966080 1555499 410581 80% /mnt

# Localize the files

$ for i in /*; do echo $i; find $i |wc -l; done

and localize the specific directory

$ for i in /home/*; do echo $i; find $i |wc -l; done

Cool, check the results with df -i command!

Like what you read? Give Alex Di Mango a round of applause.

From a quick cheer to a standing ovation, clap to show how much you enjoyed this story.