Understanding Garbage Collection Cycle Optimization — Garbage Jackson

By: Taranjit Kaur, Software Engineer I, and Neha Chopra, Senior Software Engineer at Chegg

Garbage collection is the process that is responsible for performing automatic memory management. When Java programs run on the JVM, objects are created on the heap, which is a portion of memory dedicated to the program. Eventually, some objects will no longer be needed. The garbage collector finds these unused objects and deletes them to free up memory.

Problem Statement

Our application was facing a huge amount of latency in both processing the request-response and with RAM issues. When we analyzed the production heap dump using JVM heap dump analyzer, we figured out that Jackson objects were consuming most of the RAM. Jackson is a simple, Java-based library that serializes Java objects to JSON and vice versa. The garbage collector was busy collecting the objects, which introduced latency in our system, as cycle times were reduced down from every five seconds to three seconds. In short, the garbage collector was busy collecting the huge number of Jackson’s objects created by our application.

Analysis — Why?

Because this is not a likely behavior in any application, we wanted to understand the reason for the huge number of Jackson objects created. After thorough research, we determined that we were generating a new Object Mapper instance for every request made to our application.

What is Object Mapper?

This is the Object Mapper. The Jackson library is solid and mature, and it is used for the serialization and deserialization of Java POJO object’s into JSON and vice versa.

Scope in SpringBoot

There are currently six types of instance scopes available in the spring framework:

• Singleton — a single instance is cached in the container and used for all the requests.
• Prototype — different instances are returned every time when requested from the container.
• Request — different instances are returned for each request.
• Session — the same instance of the bean is returned for the entire session.
• Application — the instance whose lifecycle is bound to the current web application.
• WebSocket — this creates it for a particular WebSocket session.

Code Smell — Object Mapper Instantiation

As discussed in the Analysis section, a chunk of Code Smell was introduced in the application, which was generating Object Mapper instances in a request scope for every request made to our application.

Here is the code snippet below:

The ThreadLocal class enables you to create variables that can only be read and written by the same thread. Thus, even if two threads are executing the same code, and the code has a reference to the same ThreadLocal variable, the two threads cannot see each other’s ThreadLocal variables. Hence, for every request in our application, we were creating a separate static ObjectMapper object.

Here’s another way of making the same mistake:

After Effects

To see the effects of the approach used above, let’s do some math. (Don’t worry, it’s simple.)

• We know that for every request, a new Jackson object is created.
• Let’s say, X number of requests are received by the single server in the distributed computed environment, which in turn instantiates the X number of Object Mapper objects in heap memory.
• Now assume that we have 1M requests coming to the single server in a day, which will create 1Mn instances of Object Mapper in our spring container in Request Scope. Let’s assume our Object Mapper size is something around 3 bytes.
• In one day, the memory allocated will be 1M*3B = 3MB.
• In one month, the memory allocated will be 3MB*30 = 90MB
• In one year, the memory allocated will be 90MB*12 = 1080MB (~1GB)…and can be X GB over X years or more in the case that requests are increased.
• In a distributed environment, there are multiple servers running for the application — for five servers, each is catering 1M requests per day. Therefore, in one day, 5M objects will be created, which is huge. And the cluster memory percentage would be increased.
• The diagram below represents the GC cycles.

Static objects are created in permanent generation, and any reference to these variables has a tendency to skip many cycles of GC. Of course, you can set a static variable to null, and thus remove, the reference to the object on the heap, but that doesn’t mean the garbage collector will collect it (even if there are no more references). Furthermore, the Object Mapper instance was created in a static reference, therefore skipping a large number of GC cycles and regaining permanency in memory. Consequently, an enormous amount of objects were created for Jackson’s objects.

As we know, Java GC destroys the objects that are no more in use/TTL expire. With this large number of Jackson objects not in use, GC cycle frequency will increase, and frequent GC cycles will be initiated.

After Effects — The Bigger Picture

Due to this, The impact was a dip in the performance of the application when the heap size increased to a significant amount and the JVM needed to perform the garbage collecting process more frequently than before. Therefore, the main processing was impacted.

Solution

Since the only work required by the Jackson object is to deserialize or serialize the JSON objects to POJOs and vice versa, and the process is thread-safe, it’s recommended to instantiate such objects in a better way — creating an Object Mapper bean in a singleton scope, where a single instance that is cached in the container will be available for all the requests coming to our application in a distributed environment.

To summarize, changing configuration is not thread-safe, so that should only be done in the bean initialization phase.

Here is the code snippet is below:

Now let’s do some math again to see how the heap size comes out using the above approach.

1. For the application server, one Object Mapper bean will be created during the instantiation phase of the application.
2. For 1M requests in a day, the same Object Mapper would be used.
3. The lifecycle of the bean will be managed by Spring.
4. In a distributed system, for five servers running for an application, only five beans will be created in the heap memory, unlike the 5M objects that were being created earlier. The cluster memory percentage would be reduced significantly.

Yes, It Made A Difference!

Below are some images of the statistics that we collected during the investigation phase of our application. First are the initial statistics that actually helped identify the root cause of the issue. See below.

The above shows the time intervals at which GC cycles were initiated. The “Avg Interval Time” is around three seconds (i.e. after every three seconds, GC was initiated and run, in order to dump the unused objects and free up the memory).

The size of the heap dump was around 210 TB — this data was collected for a year, out of which the major portion of memory taken was by Jackson objects, which is significant.

After we changed the implementation and used Singleton scope for initializing Jackson objects, the GC cycles were updated from three seconds to five seconds, as shown below.

The heap size reduced from 201 TB to around 137 TB.

In situations like this, we often neglect the small things while writing code for things like instantiations of objects using relevant design patterns. They could, however, turn out to be essential and make an impact over time if not done correctly.

The biggest learning we uncovered is to pay attention to the details while writing and reviewing code! We plan to keep this in mind as we continue to iterate and learn.

For more great content like this, check out Chegg Engineering!