Debugging Spring Boot Applications for Memory Leaks
An efficient approach to diagnosing memory leaks in Spring Boot applications
If you have been a Java developer for long enough you would have come across a memory leak or two. It is ironic that you have to talk about memory leaks in a language designed with automatic memory management as a core feature.
But then again, ‘leaked memory’ in Java is more like ‘forgotten-about-memory’ rather than memory that has leaked out of the heap. Java memory leaks happen because of references held to objects prevent them from being garbage collected.
There are many causes for memory leaks in Java, which is not the focus of this article. (Here’s an excellent article discussing the common reasons for memory leaks.) What I am focusing on in this article is how to track down existing memory leaks especially in a Spring Boot microservice.
Early signs
There could be many signs of memory leaks. The worst would be your application crashing with a dreaded OutOfMemoryError. But the problem may not go that far. If you have a decent monitoring system implemented it should alert you in advance.
We had an internal Spring Boot application that we suspected had a memory leak. This application is very simple and does nothing more than some very basic CRUD operations. The following set of graphs show the memory behavior for this application in one of our test environments.
As you can see the heap size keeps growing for several days, and most of that seems to be in the Tenured Gen space. What that means is most of the retained objects are long-lived objects that survive several garbage collection cycles. (This DZone article gives a good overview of the Java memory model.)
Monitoring tools
Above graphs are from New Relic. New Relic and AppDynamics are the leaders in infrastructure monitoring space with all the bells and whistles one could wish for. But even a much simpler free tool like Visual VM could provide enough insight into Java heap to recognize leak patterns such as above.
Ok, there is a memory leak. But where exactly is it? What’s causing it?
Now comes the hard(er) part. We know some objects are getting accumulated in the heap, but which objects are they? First step would be to have a look at the code to see if there are any obvious candidates. But in our example application, we couldn’t find any problems with a code review.
One could potentially use a memory profiling tool to track down the memory leak but that is going to be a tedious job. There is a better way — analyse a heap dump!
Heap Dump
A heap dump is a snapshot of JVM memory. Typically it would be a large binary file with .hprof extension that is roughly the same size as the heap of the application at the moment it is taken.
For a Spring Boot microservice, the easiest way to obtain a heap dump is the heapdump Actuator endpoint, if that is enabled. So you could do a curl or simply open in a browser the URL https://<servicehost>/actuator/heapdump, which will download the heap dump file.
Otherwise, there are several ways to take a heap dump as described here. Be mindful that the JVM may pause for a short while during this operation.
Which tool to analyse the heap dump?
There are several tools available for this step and my favorite is Eclipse Memory Analyzer Tool (MAT). But let’s use VisualVM for now. I will explain why later.
The heap dump from our example service gives this following object histogram when opened with VisualVM.
When the objects are sorted by size you get a char[] at the top holding over 700Mb memory. Not very helpful is it? A char[] could be anything.
Now let’s sort it by Retained Size. Retained Size can be thought of as the size of the entire object graph under a particular object. This article has a good detailed description of retained size vs shallow size etc. in memory.
Not quite the Eureka moment yet. So there are 8090 ConcurrentHashMaps at the top holding about 1.2Gb worth of memory under it. Fourth line may be a better candidate — one object holding about 1.1Gb of memory.
Let’s try the Dominators Preset.
There! o.s.s.w.a.s.RegisterSessionAuthethenticationStrategy indeed is the culprit. In the heap, an object is considered to dominate another object if all references to the second object goes through the first object. So this is exactly what we want — the root object of the object graph holding the unusually large amount of memory.
After looking at the code again, it turned out that the problem was a misconfiguration in security for the service. A RegisterSessionAuthenticationStrategy when added to the application configuration, would create a session for each request received and add it to a registry holding them in a ConcurrentMap. But nothing was removing them from this registry. API documentation recommends o.s.s.w.s.HttpSessionEventPublisher should be configured for this purpose — but actually, this service does not need to store the HTTP session details at all as it is supposed to be a stateless microservice.
Mystery solved!
Eclipse MAT
Earlier I said Eclipse MAT is my preferred tool for this kind of analysis. And here’s why.
MAT Leak Suspects report will do the above analysis and spell it out for you! And you can do a lot more detailed analysis with this tool, including comparing two heap dumps. You may not always get a clear suspect as in this example. In such situations comparing two heap dumps taken a reasonable time interval apart would be the next step. Eclipse MAT documentation has loads of valuable information not just on the tool but on memory analysis in general.
It is always good to understand the basics and elements of this type of analysis, and that is the reason I did not jump straight into MAT in this article.
heaphero.io
heaphero.io is a handy free online tool which can perform memory leak analysis and more. How it works is, you upload a heap dump file to this site and it will give you a very nice detailed report. BUT…you are shipping off your heap with all its data to a third party! So you have to be mindful of the security implications.
Some Final Words
There are many causes for memory leaks in Java and many symptoms. And there are also many different ways to go about diagnosing such memory leaks. In this article, I discussed one relatively simple approach that has served me well in the past.
Thanks for reading!
