Linux troubleshooting: Disk analysis
Disk usage
First things first, let’s start easy by taking a look at the disk capacities. This is done quickly with the df command. Use the -h (human readable), or -m (MB) flags to get the sizes represented in a format that is comfortable for you.
$ df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 3.2G 2.3M 3.2G 1% /run
/dev/nvme0n1p2 468G 33G 412G 8% /
tmpfs 16G 175M 16G 2% /dev/shm
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
/dev/nvme0n1p1 511M 5.3M 506M 2% /boot/efi
tmpfs 3.2G 1.3M 3.2G 1% /run/user/1000What to look for?
Use% The disk space consumption in % is easy enough to give you an idea of which partition is running out of space.
Don’t get confused with the
tmpfsfile system mount points, they are a Linux in-memory file system that is used for lighting fast file reading. Often times you don’t need to bother, but there are cases where you may have mounted/tmpusingtmpfs. If you have an application filling uptmpfsthen, you have an issue because that is in practice filling up memory on the host.
Inode usage
If you’re not familiar with inodes, this is Wikipedia’s description of what they are.
The inode (index node) is a data structure in a Unix-style file system that describes a file-system object such as a file or a directory. Each inode stores the attributes and disk block locations of the object’s data. File-system object attributes may include metadata (times of last change, access, modification), as well as owner and permission data. [Wikipedia, 2022–05–04]
To be concise; they store metadata about files. What sometimes happens is that you run out of inodes before you run out of disk space. This would typically happen when you have a lot of small files on your file system.
In order to figure out if you have enough inodes, run df -i.
$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
tmpfs 4089522 1372 4088150 1% /run
/dev/nvme0n1p2 31227904 450482 30777422 2% /
tmpfs 4089522 116 4089406 1% /dev/shm
tmpfs 4089522 5 4089517 1% /run/lock
/dev/nvme0n1p1 0 0 0 - /boot/efi
tmpfs 817904 150 817754 1% /run/user/1000What to look for?
Just like in the case of df -h, you’d want to just take a look at the IUse% column to figure out if any file system is running out of inode capacity.
You cannot increase the number of inodes on your existing filesystem. To do that you’d need to reformat the filesystem and — depending on which filesystem and tool you use — set a higher inode value.
To solve the immediate issue of running out of inodes, you’d have to remove files or move files to another partition.
Debugging I/O wait
If you are experiencing I/O wait issues like we saw in the CPU analysis story, then you should start trying to identify which device is having the problem. iostat can be useful for this purpose.
With a larger monitor, you can simply run iostat -x to get stats for all I/O devices. If you have more limited space (like I have in this blog), then you could try appending -d <device> and/or --pretty.
$ iostat -xd nvme0n1 --pretty
Linux 5.13.0-40-generic (tsunami) 05/04/2022 _x86_64_ (8 CPU)r/s rkB/s rrqm/s %rrqm r_await rareq-sz Device
38.94 1600.03 11.10 22.18 0.22 41.09 nvme0n1w/s wkB/s wrqm/s %wrqm w_await wareq-sz Device
19.07 303.45 16.34 46.14 1.18 15.91 nvme0n1d/s dkB/s drqm/s %drqm d_await dareq-sz Device
0.00 0.00 0.00 0.00 0.00 0.00 nvme0n1f/s f_await aqu-sz %util Device
1.41 0.51 0.03 2.64 nvme0n1
💡 Tip! Use iostat -xd <device> 1 to get per second updates. This is helpful for monitoring trends.
What to look for?
r_await / w_await is indicating I/O wait time happening when reading/writing data to the disk. If this happens, then you likely have too much data getting written to the device.
Monitoring r/s and w/s is also a good idea. This can give you a hunch of the characteristics of the read / write patterns. Is the read/write load constant or does it fluctuate?
Which process is writing heavily?
When the question starts with “Which process […]”, you can suspect the answer to start with pidstat [...]. That holds true even in this case. To get a top list of processes that are using the disk most, run pidstat -d | head -20.
$ pidstat -d | head -20
Linux 5.13.0-40-generic (tsunami) 05/04/2022 _x86_64_ (8 CPU)UID PID kB_rd/s kB_wr/s kB_ccwr/s iodelay Command
0 1 -1.00 -1.00 -1.00 9 systemd
0 141 -1.00 -1.00 -1.00 142 kworker/u16:3-
0 428 -1.00 -1.00 -1.00 11 loop4
0 523 -1.00 -1.00 -1.00 2 loop12
0 828 -1.00 -1.00 -1.00 3 acpid
What to look for?
kb_rd/s / kb_wr/s indicate heavy readers / writers to the disk. This can help you single out which process is causing I/O waits. If you don’t see any apparent culprits, but still have I/O waits in the CPU reports, then you could have an issue with a lot of processes writing to the same partition and in total they eat up the capacity of the disk.
iodelay is a value I find difficult to read as it’s measured in clock ticks. This is a number that — depending on the CPU frequency — can become quite large. If this number isn’t by a large factor higher than the other processes, you generally have no need to worry. Focus on the reads/writes instead.
Too many open files — debugging file descriptor limits
Finally, let’s touch on file descriptors. They aren’t directly tied to the disk analysis, but close enough to make space for in this story.
In Unix and Unix-like computer operating systems, a file descriptor is a unique identifier for a file or other input/output resource, such as a pipe or network socket. File descriptors typically have non-negative integer values, with negative values being reserved to indicate “no value” or error conditions. [Wikipedia, 2022–05–04]
You may come across issues of Too many open files, that typically manifest themselves in application level logs. This means that the process is hitting it’s limit for the number of open files that is allowed. Remember that in Linux sockets, actual files and pipes are all files, so having a high number of open files is common.
To check how many open files are allowed, run cat /proc/<pid>/limits.
$ cat /proc/7023/limits
Limit Soft Limit Hard Limit Units
...
Max open files 1024 1048576 files
...The soft limit indicates the current limit, while the hard limit indicate what this limit can be at most, if you edit it.
To see how many files are open for a pid, count the files in /proc/<pid>/fd like this:
$ ls -1 /proc/7023/fd | wc -l
15If you want to set other limits, update the /etc/security/limits.conf.
The Linux troubleshooting series:
