When Jackson becomes a parallelism bottleneck
So you know Jackson and you want to understand how it may be hampering the performance of your multi-threaded application?
Or maybe you don’t know what it is exactly, but you’ve heard about its name in some known projects, and now you want to know more about it?
Either way, let’s get down together on this road and try to answer all those questions!
A quick introduction to Jackson
Jackson comprises a set of open-source projects used for parsing JSON in Java. It is widely used for its simplicity, with which you can convert from JSON to Java objects and back with a myriad of flexible choices.
The simplest example of using Jackson is probably somewhere along the following snippets of code:
class Person { private String name; private int age; public Person(String name, int age) { this.name = name; this.age = age; } public String getName() { return name; } public int getAge() { return age; }}Person john = new ObjectMapper().readValue( “{ \”name\”: \”john\”, \”age\”: 42 }”, Person.class);
Problems ahoy
Using Jackson is dead simple, right?
Well, there are a few considerations to it, widely publicized, if you wish to be efficient and avoid unnecessary performance penalties — — you can find a good summary in this official Jackson documentation.
For e.g., you should reuse the ObjectMapper instance across JSON conversions throughout your code. The Jackson Javadocs state that those instances are thread-safe, and therefore it should be alright to do it.
However, some time ago I’ve stumbled upon an undesirable side-effect of following this practice: my highly parallel complex system was not scaling up its performance as I used more threads (with a machine set up with enough cores to accommodate all those threads in parallel).
I eventually figured it had to be Jackson’s fault, or somewhere close to it, since my system was configurable enough that I could use other serializers/converters for my objects; whenever I did so, the performance scaled up as expected: for e.g., going from 1 to 2 threads yielded almost twice the throughput for the system.
Diagnosing the root cause
Having already established a narrow list of suspects is a long way to go when investigating performance issues. Still, having a large complex system at hands would make it quite hard to dive into the root cause. Thus, a typical approach — — and a highly recommended one — — is to create a minimal working example that reproduces the issue.
Due to my belief that the issue lied somewhere close to Jackson, I created a micro-benchmark application using JMH that performed JSON conversion round-trips.
The key benefit of this framework is to avoid typical benchmarking pitfalls, such as calling a function (which you wish to benchmark) and ignoring its output — — thus allowing the Java Virtual Machine to optimize the compiled code to simply not call the function if it can prove it has no side-effects.
Armed with this small application, I profiled with the super useful Your Kit Profiler. Soon after, I started to get feedback from the application, as I enabled increasing levels of instrumentation on the Java bytecode, until the following became obvious:
The screenshot above — — taken straight from YourKit — — shows exactly that most of the time was being spent by the application waiting to grab a lock, which was shown to be held by the synchronized block underlying monitor used in CharsToNameCanonicalizer#makeChild(int). As you may see by the package name, that’s Jackson code right there.
So, in short, there was a sequential bottleneck within Jackson. Within each ObjectMapper instance there’s quite a handful of state kept between conversions to allow for caching computations and avoid repeating work when recurring calls contain similar JSON (for example).
Getting it fixed: the power of open-source
Having got to this point could be just the tip of the iceberg in other circumstances: we knew what the problem was, but it could be a nightmare to get it fixed!
However, that was not the case. As said above, Jackson is a set of open-source projects. So, I headed out to the corresponding project and raised an issue.
It did not take long before a Jackson maintainer was interacting with me and working out a possible solution for the bottleneck.
As we usually say: “The rest is history”. You can see that the new Jackson release was helpful enough that I did not need a work-around to the performance issue; although one was clearly easy to employ in this situation, as it sufficed to use a ThreadLocal variable so that each thread — — calling into Jackson — — used its own ObjectMapper (thus being reused, but scoped within a specific thread).
Just another day
This is a small example of some of the interesting performance challenges we go through daily at Feedzai. If you enjoy having to squeeze the last drop of performance from your applications, then you’ll have a blast doing it in teams that have been focusing on real-time fraud detection for the past 8 years. Check out our current job openings at Feedzai Careers.
