Is your Linux server disk space maxing out, but still isn’t? Check your iNodes!
Just a handy money-saving tip here.
Ever had a mysterious problem on your Linux server where you’re getting “disk space full” but don’t seem to have maxed out your disk space? Do you have a bunch of small files on your drive? You may be maxing out your iNode usage.
This maxing out is a question of quantity, rather than size. There is an iNode allocated to each file in the Linux filesystem, so if you’ve got a very large number of small files (we’re talking hundreds of thousands or more), the iNode allocation will start to suffer. Obviously one solution is to increase your volume size, but before you dish out those $$, consider that you may just need to remove some 100,000 redundant files and you won’t need to invest more money into a bigger volume.
When you’re checking your disk, use df -ih instead of df -h.
I discovered that iNodes, not disk space, were actually the reason my disk was full.
/$ df -ih
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/xvda1 512K 512K 0 100% /
When iNode usage is 100%, the amount of options to investigate this are slightly limited as the server considers disk space completely unavailable. You can’t apt-get install tools of any kind to assist you with your quest to find all those files.
I ended up trying this command, to look for the directories with most files:
/$ sudo find . -xdev -type f | cut -d “/” -f 2 | sort | uniq -c | sort -n
However, as disk space is full, sorting doesn’t work, so I added this:
/$ find . -xdev -type f | cut -d “/” -f 2 | sort — buffer-size=10G | uniq -c | sort -n
The buffer-size parameter attempts to put the whole search in memory instead of saving it to a temp file. This quickly gave me a list of root folders and recursively the total amount of files within those folders. Going into that folder under / and re-running the command gave me a nice list of folders, each which had thousands of files. I could safely delete a bunch of those folders as I knew they were old and redundant (actually left there by the previous server guy) so, after removing some 30,000 files from that directory my df -ih command looked like this instead:
/$ df -ih
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/xvda1 512K 304K 209K 60% /
Voila! Problem solved. Only 60% usage, and that was just by deleting a few redundant directories that had thousands of unneeded files.
So, when you see your disk is full, but df -h doesn’t agree, remember to check your iNode usage before you increase your disk space.
