Troubleshooting High CPU and Memory Leaks in Java Processes
--
Introduction
A common problem encountered in any Java application is high CPU utilization and memory leaks. The general solutions are either killing the process by starting all over again or increasing the heap size. For this recurring problem, the only way to get rid of it entirely is to identify the threads of a Java Process over-consuming CPU and find the root cause of the problem. This article will guide you through the basic steps to troubleshoot the given problem on a Linux machine and identify memory leaks, if any, using the Memory Analyzer plugin of Eclipse IDE.
Why do Java Applications consume high CPU?
Java applications may utilize high CPU resources for several reasons; some reasons are listed below:
- Poorly designed application code with inefficient loops, for example, recursive method calls or poor usage of collections.
- Insufficient memory in JVM can also reflect high CPU usage, leading to more time spent in Garbage Collection.
- A JVM can show high CPU usage due to incoming workload. The server capacity may not be enough to handle the incoming requests, in which case the Java application may be functioning and trying to keep up with the workload.
Challenges
The key challenge to troubleshoot high CPU and memory leaks is to find the right set of tools and Linux commands.
A list of some common tools and commands are as follows:
- jmap for memory footprint.
- jstack for thread dump.
- top for CPU and memory utilization preview.
- jvisualvm (open source) and jprofiler (licensed) tools for monitoring application, memory footprint, thread dump, etc
- https://fastthread.io/ — online tool for thread dump analysis.
- Memory Analyzer eclipse plugin for memory footprint.
Troubleshooting Guide
- Identity the Java Process that consuming a high CPU. The top is the general Linux command to achieve this task. The following screenshot reveals that PID 53984 has maximum CPU usage i.e. 1024.3
2. After identifying the Process ID, the next step is to list down the child threads for PID 53984 using the top command shown below.
top -n 1 -b -H -p <pid> > <output location>
where
-n no. of iterations
-b batch-mode operation
-H thread mode
-p process id
Sample output of the above command.
53990 hadoop 20 0 32.3g 18g 73m R 94.8 15.3 464:51.36 java
53991 hadoop 20 0 32.3g 18g 73m R 92.9 15.3 464:58.08 java
53992 hadoop 20 0 32.3g 18g 73m R 92.9 15.3 464:51.82 java
53993 hadoop 20 0 32.3g 18g 73m R 92.9 15.3 464:58.51 java
53994 hadoop 20 0 32.3g 18g 73m R 92.9 15.3 464:59.24 java
53995 hadoop 20 0 32.3g 18g 73m R 92.9 15.3 464:50.42 java
53996 hadoop 20 0 32.3g 18g 73m R 92.9 15.3 464:56.86 java
53997 hadoop 20 0 32.3g 18g 73m R 92.9 15.3 465:01.49 java
53998 hadoop 20 0 32.3g 18g 73m R 90.9 15.3 464:55.99 java
3. Convert the first column PIDs to hexadecimal values. The converted values are shown within the parenthesis below:
53990(0xd2e6) hadoop 20 0 32.3g 18g 73m R 94.8 15.3 464:51.36 java
53991(0xd2e7) hadoop 20 0 32.3g 18g 73m R 92.9 15.3 464:58.08 java
53992(0xd2e8) hadoop 20 0 32.3g 18g 73m R 92.9 15.3 464:51.82 java
53993(0xd2e9) hadoop 20 0 32.3g 18g 73m R 92.9 15.3 464:58.51 java
53994(0xd2ea) hadoop 20 0 32.3g 18g 73m R 92.9 15.3 464:59.24 java
53995(0xd2eb) hadoop 20 0 32.3g 18g 73m R 92.9 15.3 464:50.42 java
53996(0xd2ec) hadoop 20 0 32.3g 18g 73m R 92.9 15.3 464:56.86 java
53997(0xd2ed) hadoop 20 0 32.3g 18g 73m R 92.9 15.3 465:01.49 java
53998(0xd2ef) hadoop 20 0 32.3g 18g 73m R 90.9 15.3 464:55.99 java
4. Next, we capture the jstack of the parent process id (53984), as shown below, and look up the hex values listed in step 3 with parent PID’s jstack file to identify the child threads consuming high CPU.
Sample output of command jstack 53984 > 53984.txt
“VM Thread” os_prio=0 tid=0x00007fccac09e800 nid=0xd2f3 runnable
“GC task thread#0 (ParallelGC)” os_prio=0 tid=0x00007fccac032800 nid=0xd2e6 runnable
“GC task thread#1 (ParallelGC)” os_prio=0 tid=0x00007fccac034800 nid=0xd2e7 runnable
“GC task thread#2 (ParallelGC)” os_prio=0 tid=0x00007fccac036000 nid=0xd2e8 runnable
“GC task thread#3 (ParallelGC)” os_prio=0 tid=0x00007fccac038000 nid=0xd2e9 runnable
“GC task thread#4 (ParallelGC)” os_prio=0 tid=0x00007fccac03a000 nid=0xd2ea runnable
“GC task thread#5 (ParallelGC)” os_prio=0 tid=0x00007fccac03b800 nid=0xd2eb runnable
“GC task thread#6 (ParallelGC)” os_prio=0 tid=0x00007fccac03d800 nid=0xd2ec runnable
“GC task thread#7 (ParallelGC)” os_prio=0 tid=0x00007fccac03f000 nid=0xd2ed runnable
“GC task thread#8 (ParallelGC)” os_prio=0 tid=0x00007fccac041000 nid=0xd2ee runnable
“GC task thread#9 (ParallelGC)” os_prio=0 tid=0x00007fccac043000 nid=0xd2ef runnable
5. The threads identified in step 4 will be called the ParallelGC threads. These threads perform Garbage collection but cannot reclaim heap memory that is causing high CPU utilization.
6. The next step is to collect the heap dump of the PID 53984. This is achieved by using the following Linux command:
jmap -dump:live,format=b,file=heap.bin 53984 > 53984_heap
7. Analyze the 53984_heap file using the Memory Analyzer plugin which can be installed from Eclipse’s Marketplace (Navigate to Help option then select Eclipse Marketplace).
8. The following are screenshots of the memory leak report corresponding to PID 53984:
Observations
- Approx. 4.4 GB heap space is consumed by stale instances of java.net.FactoryURLClassLoader.
- Parallel GC threads consume high CPU time from jstack file (53984.txt), which implies that GC is not functioning properly for those stale objects in heap space.
Solutions
A few short-term solutions for memory leaks include increasing the heap space. This solution can postpone the issue for a while, but it might reappear if the load to Java Process is high.
Some of the promising approaches are to do a deep dive into the source code of the Java application and identify the areas that are causing stale instances in heap memory.
Conclusion
In conclusion, this article walks readers through:
- Potential reasons why Java Process consumes high CPU
- A troubleshooting guide to resolve the issue
- List of some common tools used for given solutions
- Linux commands to debug
- How to correlate high CPU with a memory leak
