How can we analyze JAVA memory issues (java.lang.OutOfMemoryError: GC Overhead limit exceeded) using ECLIPSE MAT and Jprofiler: III
In my previous blog posts, I talked about java.lang.OutOfMemoryError:Java heap space issues and how can we utilize ECLIPSE MAT and Jprofiler tools to analyze the heap dumps and find the cause of the issue. And also focused on different features of both tools and how we can use those to analyze the heap dumps. Please make sure to read those two blog posts before reading this one because we have discussed some fundamentals and keywords on those articles and you need to have an understanding of those before diving into this one[1][2]. In this blog post, I will continue the java.lang.OutOfMemoryError article series and will discuss the java.lang.OutOfMemoryError: GC Overhead limit exceeded issue.
java.lang.OutOfMemoryError: GC Overhead limit exceeded
This is the explanation from the Oracle docs[3]. The detail message “GC overhead limit exceeded” indicates that the garbage collector is running all the time and Java program is making very slow progress. After a garbage collection, if the Java process is spending more than approximately 98% of its time doing garbage collection and if it is recovering less than 2% of the heap and has been doing so far the last 5 (compile time constant) consecutive garbage collections, then a java.lang.OutOfMemoryError is thrown. This exception is typically thrown because the amount of live data barely fits into the Java heap having little free space for new allocations.
It simply says that the garbage collector cannot free up enough memory after 5 GC cycles. It could be due to not having enough memory allocation for the application. If that is the case, you can increase the heap memory allocation to the recommended values using xms and xmx JVM flags.
But if you have configured enough heap memory, this could be due to a memory leak in your Java application code. As an example, if your application code execution goes into an infinite loop due to a code bug and it generates a huge amount of objects in the heap and the GC is unable to clean those, java.lang.OutOfMemoryError: GC Overhead limit exceeded error will be thrown and the heap dump file will be generated in the location specified in -XX:HeapDumpPath.
As usual, most of the time it could be due to huge memory usage from a single object and if that is the case, you can simply identify the problematic objects from the dominator tree in the ECLIPSE MAT or you should check the biggest objects section in the Jprofiler.
But sometimes, you will not be able to find any clue just by looking at the dominator tree on MAT or the biggest objects in the Jprofiler. Then you need to focus on leak suspects, top components, and top consumers sections in the mat memory analyzer and see whether those reports indicate the problem.
As we discussed before, if the java.lang.OutOfMemoryError: GC Overhead limit exceeded error is thrown due to an infinite loop in your code and it may generate thousands or millions of different kinds of objects and not a single object which consumes the most of the memory. In that kind of situation, the dominator tree in MAT or the biggest objects section in the Jprofiler may not indicate the problem.
Also, in this kind of situation, you need to focus on the actual heap dump file size and the heap usage sizes that are shown in the tools. If the actual heap dump file size and heap usage size which are shown in the tools have a huge difference, you need to focus on unreachable objects as well.
Sometimes a heap dump contains objects that are eligible for garbage collection and those would be removed at the next garbage collection. These are objects which are unreachable from the garbage collection roots. By default, Memory Analyzer and jprofiler remove these objects as part of the parsing process as normally they are not crucial in determining a memory leak. If the heap dump is generated as a result of an OutOfMemoryError then the Java virtual machine will usually have performed a garbage collection operation to attempt to free space for the new object, so there will not be those objects in the heap dump file. There is a chance that the garbage collection was not performed, for example, if the object to be allocated was so huge that it was obvious to the JVM that no amount of garbage collection could free enough space. As we discussed before, let’s assume that your Java application went into the infinite loop which triggered an execution path that generates a huge number of objects and those objects are eligible for the gargable collection after one iteration of the loop. But In a single GC iteration, if the garage collector is unable to remove all of those objects, it probably will result in a java.lang.OutOfMemoryError: GC Overhead limit exceeded error and generated heap dump may have those unreachable objects. In order to check the unreachable objects in the heap dump, ECLIPSE MAT has an excellent feature where it can generate all the memory reports and charts with the unreachable objects along with live objects.
In order to enable unreachable objects in MAT, you need to go to the Window → Preferences
Then go to the memory analyzer section, select the Keep unreachable object tick, and click apply.
In order to apply those changes, you need to restart the MAT and open the heap dump file again. Please note that you need to copy the heap dump file into a new location before opening it again from the MAT or you need to delete the previously generated index files. Otherwise, it will use those previous files to generate reports.
Then you can check the dominator tree, leak suspects and top consumer section to check whether you can get new information from the heap about the problem.
Below I have shared two screenshots that show the memory usage from the heap dump before and after enabling the unreachable objects(heap dump file size is 600MB). As you can see, there is a clear difference of heap usage.
Before enabling the unreachable objects.
After enabling the unreachable objects.
Also, after enabling the unreachable objects, if you go to the dominator tree, it will show the unreachable objects along with the live objects.
This way you will be able to check where there are unreachable objects that consume most of the memory that resulted in throwing java.lang.OutOfMemoryError: GC Overhead limit exceeded error and generating the heap dump file.
Please note that the above-discussed points are valid for high memory usage situations where the application does not generate any OutOfMemoryError and heap dump but it consumes a high amount of memory. As we discussed in previous blog posts, you need to generate a heap dump file using the jmap command while your application consumes more memory and you can use MAT and Jprofiler tools to analyze the generated dump file to find the cause of the issue.
Ok then, that’s it for today. Happy blogging!!!
[3]https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/memleaks002.html
