GDB: Debug native part of java application (C/C++ libraries and JDK)

Why debugging of C/C++ code may be required for Java developers?

I worked with a few java projects that used native libraries created by another team from the same organization. Usually, we invoked C++ code from java.

The issue with C++ code invoked from Java is that it is usually not visible from java. We only see a top-level interface with JNI/JNA but don’t know what is going under the hood.

As a result, we can’t get a lot of information from java debugger and profiler that we use daily.

In this post, I’ll describe a gdb debugger that is able to work with the native code. As an example, we will build a C++ library for Linux (.so-file), invoke from Java and debug it.

As low-level part of JDK is also written in C++, we will take a look at how to debug JDK native code also.

GDB

GDB or GNU Debugger is a command-line debugger that comes with most Linux distributions and supports lots of processors. GDB supports both remote and local mode.

It is important to note that as of now GDB doesn’t support debugging of Java code (on https://www.gnu.org/software/gdb/ you won’t find Java in the list of supported languages). If you want to debug java code from command-line you can try JDB. It looks similar to GDB but has less functionality.

Having said that GDB doesn’t allow you to debug java-code, it can perfectly debug a native part (written not in java) of Java application. If it is important for you to debug both Java and native code “from the same IDE”, you can try to use Netbeans or Eclipse. Netbeans uses default debugger for Java code and GDB for native code. However, for IDE users switch between Java and the native code won’t be visible.

Of course, command-line debugging may look weird at first for people familiar with IDEs. But it has some benefits. One of them is the ability to run on a remote host. Although remote debug is built-in in java, debugging an application running in another part of the globe using IDE remote debuggers could be very sloooooow. GDB from the other hand runs on the target host. This is especially useful when you need to evaluate some code during debugging.

Prepare native code

All code is available in my github repo.

To be able to see a native code in the debugger, the code should be compiled in a special way. Information about method names and variables should be included in the library or comes as a separate package.

Often the easiest way to include debug info is to add it to the resulted library. Let’s write a simple java application and a C++ library.

We define an interface of native methods and load the library. The code won’t run, because the library doesn’t exist right now. We will build it soon.

Method’s names are self-describing: nativePrint — prints constant string to stdout, nativeSleep — sleeps for ms milliseconds, nativeAllocate — allocates memory for an array of size n, nativeCrash — crashes application (we will simulate a crash at the end of the article to check what information is available for investigation).

Let’s generate a c++ header file (interface) for methods defined in JNIDemoJava.java.

# generate c++ header file and put to cpp folder
/usr/lib/jvm/jdk-11.0.1/bin/javac java/jnidemo/JNIDemoJava.java -h cpp/

After we got a definition of C++ methods we can implement them:

Our C++ implementation is ready and we can create a shared library:

run “./scripts/buildLib.sh”

One important option is -g. It tells GCC compiler to include debug information in the library. It increases the size of the library and allows us to see source code during debugging.

GDB debug

Let’s try to debug our code. You can start gdb and java app together or you can attach gdb to the running app. Let’s check 2nd option as starting gdb together with the java app may require modification of startup scripts (modification is quite simple).

Note: I had an issue with default ptrace settings on Ubuntu described here. I fixed it with next command echo 0 > /proc/sys/kernel/yama/ptrace_scope.

Let’s start debug:

# start java app
# find application PID
jps
# start gdb with application PID
gdb -p 1234
# gdb will pause our application
# add a breakpoint to our code (all java packages has Java prefix)
(gdb) break Java_jnidemo_JNIDemoJava_nativeAllocate
    Breakpoint 1 at 0x7f5e77dfe944: file src/cpp/JNIDemo.c, line 35.
# resume application and wait when it will stop at breakpoint
(gdb) cont
    Continuing.
    [Switching to Thread 0x7f5ee5ef5700 (LWP 4052)]
Thread 2 "java" hit Breakpoint 1, Java_jnidemo_JNIDemoJava_nativeAllocate (
        env=0x7f5edc013340, obj=0x7f5ee5ef4980, objNumber=100)
        at src/cpp/JNIDemo.c:35
    35     return internalNativeAllocate(env, objNumber);
# step into our "internalNativeAllocate" function using "s" command
(gdb) s
internalNativeAllocate (env=0x7f5edc013340, objNumber=100)
    at src/cpp/JNIDemo.c:20
20     jclass classDouble = (*env)->FindClass(env, "java/lang/Double");
# after stop at breakpoint we can do different operations (check    # stacktrace, variables, registers, threads, etc.). Let's print     # value of objNumber parameter
(gdb) print objNumber 
$6 = 100
# we can try to debug jdk code. In this case we don't have debug     # symbols and won't get a lot of information (but info about        # threads, registers, stacktrace is still available
(gdb) set step-mode on
(gdb) set step-mode onQuit
(gdb) s
24         jmethodID midDoubleInit = (*env)->GetMethodID(env, classDouble, "<init>", "(D)V");
(gdb) s
0x00007f5ee44fc320 in jni_GetMethodID ()
   from /usr/lib/jvm/jdk-11.0.1/lib/server/libjvm.so
#exit
quit

I won’t describe all gdb commands here as list is quite big.

Debug JDK

We just got some experience in debugging native libraries. Our next step is to debug the native part of JDK (a big part of jdk is written in C++).

As an example we will debug writeBytes(…) native method that is invoked as a part of a well-known System.out.println(…) call:

#FileOutputStream.java
...
private native void writeBytes(byte b[], int off, int len, boolean append)
    throws IOException;
...

First, we need to download and compile JDK with extra parameters to preserve debug symbols.

#clone jdk from mercurial
hg clone http://hg.openjdk.java.net/jdk/jdk 
#configure and make build
bash ./configure --with-target-bits=64 --with-debug-level=slowdebug --disable-warnings-as-errors --with-native-debug-symbols=internal
make clean
make all

Let’s start and debug our app with gdb using jdk from build/linux-x86_64-server-slowdebug folder. Now we can see the source of writeBytes method (Java_java_io_FileOutputStream_writeBytes). If we debug our app with usual jdk we won’t see the source. This also allows us to see variables names.

(gdb) list Java_java_io_FileOutputStream_writeBytes
64     writeSingle(env, this, byte, append, fos_fd);
65 }
66 
67 JNIEXPORT void JNICALL
68 Java_java_io_FileOutputStream_writeBytes(JNIEnv *env,
69     jobject this, jbyteArray bytes, jint off, jint len, jboolean append) {
70     writeBytes(env, this, bytes, off, len, append, fos_fd);
71 }
72 
(gdb)

Automatically start GDB when an application crashes

When java app crashes somewhere in the native code, Linux produces core-dump. This file contains full memory snapshot with information about threads and another useful information. We can analyze this file with different tools and gdb is one of them.

However, in some production system core dumps may be disabled. One of the reasons is their size. Let’s says that your app uses 80 GB of RAM (Java heap + objects allocated by native code) then each crash will create an 80 GB core dump file. If you have some system/script that checks if your app is alive every 5 minutes and restarts it if it is dead, you can run out of disk space quite fast.

In this case, it is useful to automatically invoke GDB when an application crashes. It will be much easier to understand which code caused the issue, check variables state and core dump file isn’t required. To do this we should just add -XX:OnError=”gdb — %p" option to our app.

To demonstrate this scenario I created the nativeCrash method in our library (it tries to access memory that wasn’t properly allocated).

Let’s start this app with -XX:OnError=”gdb — %p” option. Immediately after start application will crash and we will get gdb in the terminal:

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f7348cba806, pid=10055, tid=10057
#
# JRE version: OpenJDK Runtime Environment (10.0.2+13) (build 10.0.2+13-Ubuntu-1ubuntu0.18.04.4)
# Java VM: OpenJDK 64-Bit Server VM (10.0.2+13-Ubuntu-1ubuntu0.18.04.4, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# C  [libJNIDemo.so+0x806]  Java_jnidemo_JNIDemoJava_nativeCrash+0x1c
#
...
(gdb)

From the code above, we see that the issue is with Java_jnidemo_JNIDemoJava_nativeCrash method and it is invoked from thread with id 10057.

To check the source that failed and values assigned to it’s variable we can do next: find thread gdb-id, switch to this thread, switch to the frame where we expect the issue. Here how we can do this:

# find a thread that caused an error
(gdb) info threads
  Id   Target Id         Frame 
* 1    Thread 0x7f73ad6f2380 (LWP 10055) "java" 0x00007f73ac8aed2d in __GI___pthread_timedjoin_ex (threadid=140134807701248, thread_return=0x7ffc2cdb9338, 
    abstime=0x0, block=<optimized out>) at pthread_join_common.c:89
  2    Thread 0x7f73ad6f0700 (LWP 10057) "java" 0x00007f73acfca6c2 in __GI___waitpid (pid=10098, stat_loc=0x7f73ad6eefcc, options=0)
    at ../sysdeps/unix/sysv/linux/waitpid.c:30
  3    Thread 0x7f73a960e700 (LWP 10058) "GC Thread#0" 0x00007f73ac8b66d6 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, 
    futex_word=0x7f73a40295a8)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:205
...
# switch to this thread
(gdb) thread 2
...
#6  0x00007f73ac0ee228 in signalHandler(int, siginfo_t*, void*) ()
   from /usr/lib/jvm/java-11-openjdk-amd64/lib/server/libjvm.so
#7  <signal handler called>
#8  0x00007f7348cba806 in Java_jnidemo_JNIDemoJava_nativeCrash (
    env=0x7f73a4012a00, obj=0x7f73ad6ef980) at src/cpp/JNIDemo.c:11
...
# switch to the frame with our code
(gdb) frame 8
#8  0x00007f7348cba806 in Java_jnidemo_JNIDemoJava_nativeCrash (
    env=0x7f73a4012a00, obj=0x7f73ad6ef980) at src/cpp/JNIDemo.c:11
11       printf( "%c\n", s[0] );

We found the code that caused the issue. If required we can print values of different variables.

P.S. Note about segfaults

When you will start debugging a large application, you may notice that the execution of gdb suddenly stops. The reason is that gdb automatically stops app when the segfault occurs. This makes sense for some applications but not for Java apps. JDK uses different tools that may produce segfaults (speculative memory load, NullPointerException, etc.). JDK handles SIGSEGV internally, but gdb has no idea about this. This is why we need to force gdb to ignore them.

(gdb) handle SIGSEGV nostop noprint pass