So I Heard Jil Is the New JSON Serializer Champion 🏆
I’m always interested in improving performance were I can especially if it’s a quick win and I recently read about Jil and was impressed by the benchmark results I've see online.
So I decided to create a small experiment with Newtonsoft.Json (v8.0.3) and Jil (v2.1). I’ll let you read more about them on their respective sites. Also see the References section for other benchmarks, etc.
The tests were conducted in the context of a typical (but controlled) day-to-day serialization task. I compared pre and post “warm up” serialization times for both serializers. A warm up is as explained by kevin-montrose:
The first time Jil is used to serialize a given configuration and type pair, it will spend extra time building the serializer. Subsequent invocations will be much faster, so if a consistently fast runtime is necessary in your code you may want to “prime the pump” with an earlier “throw away” serialization.
Both Newtonsoft.Json and Jil spend some time warming up in their first run so it is worth reporting (see Chart 1). It’s more important to look at post warm up because this is the condition they're primarily used in. In any case we can run a throwaway serialization as part of a live deployment.
Experiment and Findings
- Tests were written in C# .Net console application.
- System.Diagnostics.Stopwatch was used to measure timer ticks (ElapsedTicks).
- Both the Jil and Newtonsoft.Json serializer instances.
- A single TestObj instance was the dependent variable of the experiment and serialized in each iteration.
- 10k iterations/measurements were taken per serializer.
- Measurements logged after each iteration.
I found most measurements taken for Jil were <1ms so I opted for ticks as:
A tick is the smallest unit of time that the Stopwatch timer can measure.
I attempted to log the timings accurately by lowering the pre-test overheads. To do this I simply initialised TestObj, and both serializer instances before running RunSerializer. Logging only took place in the iteration loop and logged after the timer has been stopped. Full source code can be found here.
I found that post warm up, Newtonsoft.Json took 5 times more ticks than Jil. Both took noticeable time to warm up in their first runs with Jil taking slightly longer. On average, after warm up, each serialization for Jil (19.42) showed a considerable saving in ticks over Newtonsoft.Json (53.56). Jil has a higher max after warm up because of the infrequent dramatic spikes (see Chart 2) but a much lower median (9.00) than Newtonsoft.Json (51.00).
Some further analysis over the 10k iterations showed (see Chart 2) that, the measured ticks stabilised and reduced for Jil. Newtonsoft.Json was also consistent throughout. Jil had some consistent looking intervals of spikes which are unexplained but maybe @kevinmontrose can shed some light on this. It is also worth noting these spikes are present in other runs of the experiment I conducted but overall these spikes are short and show no significant change overall to the results (see Table 1 average and median columns). Newtonsoft.Json also shows some spike patterns too in a similar fashion to Jil but less dramatic. It could be a caching limit is reached and reset after a certain number of iterations, although it seems no rebuilding process is taking place after the spike so this is unlikely to be the case.
Conclusions
In this experiment I compared both Newtonsoft.Json and Jil’s JSON serialization times by experimenting with similar conditions faced by my web service. I wanted to find out which of the two performed the fastest under these conditions and I found Jil performed 5 times better.
There are other variables to consider when making a switch to new technology. Other things to consider are, but not limited to, effort to switch over, feature sets, maturity of the software and realistic performance gains to the user (“Does the customer care?”). I won’t address each of these points here as it is not the purpose of this article.
But I will mention that the better JSON serializers out there are still pretty young, Jil being the younger of the two but both are used in large scale solutions (Jil at StackOverflow and Newtonsoft.Json supports ASP.NET CoreCLR and is used by many reputable tech companies), Newtonsoft.Json has a richer feature set and Jil discounts some features in the pursuit of speed.
Jil aims to be the fastest general purpose JSON (de)serializer for .NET. Flexibility and “nice to have” features are explicitly discounted in the pursuit of speed.
For straight serialization (and deserialization maybe) Jil would be a good fit because it’s built for speed, but when a lot of manipulation and querying is needed on JSON data then Newtonsoft.Json wins hands down for the features it has in that area. Both can be used in a web service in the parts where their benefits are needed.
What’s next?
I did some quick tests with different instances of TestObj which included DateTime and Guid and the conclusions reported here still held. I want to test Jil with some real variable data instances as opposed to the single instance used here. I also want to look at this in a profiler before deciding on whether to make a switch.
The source code used to run the tests can be found here.
Computer Spec
Spec taken from Speccy:
- Operating System: Windows 10 Home 64-bit
- CPU: Intel Core i7 5820K @ 3.30GHz
- RAM: 16.0GB Dual-Channel Unknown @ 1066MHz
- Software: Visual Studio Community 2015
