Speeding up Java heap dumps with GNU Debugger

A Java Heap Dump is a snapshot of the memory of a JVM process at a specific moment. Heap dumps can be taken in several file types, however the most-used type is HPROF binary format. The following are the typical information included in a heap dump (may be more or less depending on the heap dump type).

• All objects: Class, fields, primitive values, and references
• All classes: Class loader, name, super class, and static fields
• Garbage collection roots: Objects defined to be reachable by the JVM
• Thread stacks and local variables: Call-stacks of threads at the moment of the snapshot, and information about local objects on a frame by frame basis

Uses of heap dumps

Heap dumps are used to troubleshoot memory problems in your applications. The most common and primary way to generate a heap dump is using jmap which is a command line option bundled with the JDK. Also there are several other approaches [2] you can try.

jmap — primary approach to take heap dumps

Let’s first look at how we use jmap to take a heap dump.

First we need to obtain the Process ID (PID) of the Java process that we are going to investigate.

ps aux | grep java

Now invoke jmap with the PID (say 21540), the heap dump format you want, and the destination of the heap dump file you want

jmap -dump:format=b,file=/opt/tmp/heapdump.bin 21540

downtime — the problem of primary approaches

As IBM Knowledge Page [1] explains, typically, a full garbage collection is triggered before the heap dump is written, so a heap dump contains information about the remaining objects in the heap. We know that full GC takes a considerable amount of time. Also it takes a lot of time for jmap to do internal conversions and finally present you a beautiful HPROF binary file. Altogether it’s a lot of downtime!

And if you have practically used jmap, you must know how this process takes several minutes to sometimes few hours to finish taking a heap dump — based on the size of your application (Larger the application, larger the memory you allocate for heap. Then, larger the heap size, larger the downtime it takes for taking heap dump).

If you do this during the local development, you won’t care much about the downtime (as long as your company provides enough coffee, PlayStation, or even comfortable bean bags). But imagine doing this in a staging environment or even at the production when things go really bad. You literally cannot use your application until the heap dump is captured (Nobody will be happy to hear it, right?).

GNU Project Debugger (GDB) — our savior

As the GNU project page [3] explains, GDB allows us to see what’s going on inside another program while it executes or what another program was doing at the moment it crashed. We can use this GDB command line tool to take heap dumps from JVM processes with few steps, more importantly within seconds (obviously faster than the traditional jmap approach).

Step 1 — Install GDB

First you need to install GDB since it’s an external tool.

For Ubuntu/Debian users, run:

sudo apt-get install gdb

For RedHat/CentOS users, run:

sudo yum install gdb

Step 2 — Take a core dump

You now need to obtain the PID of the JVM process as usual.

ps aux | grep java

Now we can attach GDB to the JVM process indicated by the PID we obtained (say 21540). This makes the JVM process stop executing and responding until JDB is detached from it (But trust me, it takes only few seconds. Compared to jmap, it’s less time and pain). Since you interactively type these command and run on terminal, make sure you do things fast too.

sudo gdb — pid 21540

Just in case, this command doesn’t work, run gdb –help in the terminal and see the correct option for attaching debugger to running process PID.

Then tell GDB to dump the core to a specific file for that JVM process, then detach the process, and finally quit.

(gdb) gcore /tmp/jvm.core
(gdb) detach
(gdb) quit

Now please do restart your application to get it up and running again as usual.

Step 3 — Convert core dump to heap dump format

We can use jmap to convert our core dump file to the heap dump format.

sudo jmap -dump:format=b,file=jvm.hprof /usr/bin/java /tmp/jvm.core

There’s a small catch I need to mention. Running jmap on the same server that your application runs will have impact on the performance of both programs. Therefore the recommended approach is to SCP your core file to an idle server or local machine with the same Java version used in your application’s server. (Java version used to create dump file should be same as the Java version used in jmap).

Taking Java heap dump with GNU Debugger (GDB)

Step 4 — Analyze the heap dump

There are several great tools such as jhat (basic tool), VisualVM (default choice), Eclipse Memory Analyzer (good enough) [4] and HeapHero (impressive tool with advanced features and deep-learning based suggestions) [5] to analyse heap dump files and have better insights. I suggest you to try them and get used to the one you feel comfortable because the choice mostly will depend on the visual experiences they provide.

If you know more hacks and tricks to play with JVM, please feel free to share them in the comments section. And stay excited with this blog to see more awesome posts in the future!!

References

1. Understanding heap dumps — https://www.ibm.com/support/knowledgecenter/en/SS3KLZ/com.ibm.java.diagnostics.memory.analyzer.doc/heapdump.html
2. 7 Ways to Capture Java Heap Dumps — https://dzone.com/articles/how-to-capture-java-heap-dumps-7-options
3. GDB: The GNU Project Debugge — https://www.gnu.org/software/gdb/
4. Eclipse Memory Analyzer — https://www.eclipse.org/mat/
5. HeapHero — http://heaphero.io/

Originally published at platformengineer.com on September 21, 2018.