Detailed performance analysis (especially in production environment) can be really difficult. However, you don’t need sophisticated tools to understand what’s happening with performance of your system at a high level.
In this post I will describe a few system and process parameters checking that can help finding and localizing performance problems. Also I will list some tools that can aid further investigation.
- Process user time
Time that is spent running user-mode code of the process. There are many ways to measure this parameter, but the easiest is to use standard UNIX time utility:
> time ls -all
…
real 0m0.005s
user 0m0.001s
sys 0m0.002sSo if you’re seeing unexpected user time for the process you focus on— any good profiler for your language runtime can show you detailed information.
- Process system time
Time that is spent in the kernel executing system calls. Usually tools that measure process user time also measure system time (e.g. atop , time). System call tracers (strace, dtrace) can help you gathering more info about process system calls.
- CPU idle time and IOWait
CPU idle time (top’s idle %) shows what percentage of time the processor is doing nothing. IOWait is a subcategory of idle time and indicates that idle state is caused by waiting for I/O.
Idle percentage and IOWait percentage can be seen in top ( % idle and % wa respectively).
Also you can inspect I/O parameters for running processes using iotop utility.
- Process memory usage
There are few parameters that show process memory usage, but most important are:
- resident set size (RSS) — how much memory is allocated for the process and is in RAM.
- virtual memory size (VSZ) — all memory that process can access including memory that is swapped.
Using RSS and VSZ values you can determine the excessive memory usage and (if you’re lucky) some memory leaks, but if this is not enough — use vmstat and valgrind or tools provided by your language runtime.
Actual values of RSS and VSZ can be inspected using top and it’s variants (htop, atop).
This list is not exhaustive, but performance problems immediately affect it’s parameters and listed utilities can help identifying and eliminating them.
