Native Memory — The Silent JVM Killer
I recently worked on a tricky issue where our application was terminated ungracefully twice in one week after being enabled in a high-volume production environment. All signs pointed to some sort of memory issue but there was little concrete data to assist in identifying the root cause. What followed was a couple of weeks of bewildering frustration, an existential crisis or two and a hard-earned lesson with a satisfactory conclusion. If you want to hear about my journey to getting to the bottom of the problem, how you can do it in your own applications and the steps to take to avoid it in the first place then read on.
Running Out of Memory
Our application was shut down twice in one week and it was clear from our monitoring that the JVM heap usage spiked in the moments prior to termination however we never got close to full heap usage. At the time we were running our JVM with Xms=500M and Xmx=45G on a VM with 60GB memory available. The other processes on the VM had very small memory footprints so we met the local recommendation of allowing 20% memory overhead above heap size to allow for the JVM’s use of native memory and any performance testing done had not indicated we needed anything more than this. Since we were not getting an OutOfMemoryException in the application logs and our metrics were being polled only once a minute we postulated…
