JSON Serialization Libraries Performance Tests
JSON serialization is a key factor in web application operations. Rest APIs heavily rely on JSON serialization (HTTP response) and deserialization (HTTP request).
Most of us have used Newtonsoft.Json library (known also as Json.NET) for serialization in our Rest APIs since it was integrated into ASP.NET, but how many of us thought about the efficiency of this library?
Fact 1: .NET string type is encoded with UTF-16.
Fact 2: HTTP protocol text encoded with UTF-8.
Fact 3: Newtonsoft actually takes UTF-8 text and uses it with .NET string. In other words, use UTF-8 text as a UTF-16 text.
Both UTF-8 and UTF-16 are variable length encodings. However, in UTF-8 a character may occupy a minimum of 8 bits, while in UTF-16 character length starts with 16 bits.
The fact that Newtonsoft uses UTF-8 text as a UTF-16 text might indicate about performance issues.
Test Criteria
We decided to make a performance test for JSON serialization libraries. Each library will be tested by 3 aspects for both serialization and deserialization actions:
- Execution time
- Memory allocation
- CPU usage
Execution time and memory allocation will be measured with BenchmarkDotNet NuGet package.
CPU usage will be measured with PerformanceCounter which provided by .NET System.Diagnostic library.
BenchmarkDotNet
BenchmarkDotNet helps you to transform methods into benchmarks, track their performance, and share reproducible measurement experiments. It’s no harder than writing unit tests! Under the hood, it performs a lot of magic that guarantees reliable and precise results thanks to the perfolizer statistical engine. BenchmarkDotNet protects you from popular benchmarking mistakes and warns you if something is wrong with your benchmark design or obtained measurements. The results are presented in a user-friendly format that highlights all the important facts about your experiment.
Here is an example of code using BenchmarkDotNet:
In the above attachment, there is a serialization dedicated class for the Jil serialization library. [Benchmark] attribute on top of a method is an indication for BenchmarkDotNet to perform an execution time test for the method. The benchmarked method will be triggered more than 50,000 times in order to perform statistical calculations.
It is important to isolate the actual serialization/deserialization action inside the benchmarked method so BenchmarkDotNet will measure the action itself, without any pre/post configuration noise.
In order to prepare the configuration prior to the BenchmarkDotNet run, [GlobalSetup] the attribute should be placed on top of a method that will be triggered right before the benchmarked methods.
It is possible to place [MemoryDiagnoser] attribute on top of a class in order to measure memory allocation for all benchmarked methods inside this class.
BenchmarkDotNet should be run against release code to ensure all optimizations are included.
At the end of its run, output will be displayed:
In the image above we can see the mean value that reveals execution time for Jil serialization action was 321.8 nanoseconds (10^-9). In addition, it provides another statistical results such as Error and Standard Deviation which are out of my scope, but it is definitely nice to have information.
At the far right column, you can see the allocated column that tells how much memory allocated per single operation (method execution): 576 B. In addition, it provides with garbage collector generations (0, 1, and 2) collections per 1000 operations, another nice-to-have information.
PerformanceCounter
The library System.Diagnostic provides us with PerformanceCounter object that allows us to measure CPU usage between time intervals. See in the image:
Explanation:
- In the first line I defined PerformanceCounter to monitor process time by passing the current process name. It is recommended to get PerformanceCounter sleep before starting measuring.
- The first for (outer) loop is just a way to gather a certain value of CPU usage samples and get their mean value. This will increase the statistical significance of the result.
- Inside the second for (inner) loop, there is an execution of JSON serialization 1000 times. Right after it is done, CPU usage is extracted from PerformanceCounter object by calling its method NextValue().
- After outer while finished its run CPU usage sum variable should be divided by a number of samples in order to get the mean value of all samples and then it should be divided by the number of machine logical processor in order to normalize the data, otherwise percentage value could be higher than 100%.
Data Types
Serialization libraries will be tested against two types of objects:
- Simple class
- Complex class
Simple Class — contains 3 primitive variables:
Complex Class — contains ~30 variables, some of them are custom objects (such as address) and Enums. One of its variables is a list of Simple Class. Inside Complex Class contractor the Simple Class list variable is populated with 2000 Simple Class instances, which makes is complex enough for our research.
Scenarios to Test
Each serializer library will be testes in the following 8 scenarios:
- Serialization success if a simple class
- Serialization success of a complex class
- Serialization Error of a simple class
- Serialization Error of a complex class
- Deserialization success if a simple class
- Deserialization success of a complex class
- Deserialization Error of a simple class
- Deserialization Error of a complex class
Serialization libraries
- Jil — A fast (self-proclaimed) JSON (de)serializer, built on Sigil with a number of impressing optimization tricks.
- FastJSON — Polymorphic JSON Serializer.
- NetJSON — A recommended serialization library found over the network.
- Newtonsoft.Json —Maybe the most popular serialization library. Was integrated into ASP.NET even though it was 3rd party.
- Protobuf-net — Protocol buffers are Google’s language-neutral, platform-neutral, extensible mechanism for serializing specific structured data — think XML, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages.
- ServiceStack — .NET serializer to JSON, JSV, and CSV.
- SpanJson — A serialization library that utilizes
Span<T>to reduce memory allocations. - Utf8Json — Very fast serialization library with a very good memory allocation.
- System.Runtime.Serialization.Json — Microsoft-developed serializer that was integrated in previous ASP.NET versions until Newtonsoft.Json replaced it.
- System.Text.Json — The brand new serializer by Microsoft. Supposedly faster and better than Newtonsoft.Json. Integrated by default with the new ASP.NET Core 3 projects.
Serialization Success — Results
Just provide simple class and complex class to the serialization class of each library.
Execution Time
Memory Allocation
CPU Usage
Deserialization Success — Results
Just provide simple class and complex class to deserialization class of each library.
Execution Time
Memory Allocation
CPU Usage
Serialization Error— Results
In order to fail serialization, we need to make an endless loop of serialization operations. Therefore, instead of simple and complex classes, I am going to use now four classes. Here is how to do it:
- SimpleParent Class: Similar to Simple Class but contains one more member, SimpleChild Class.
- SimpleChild Class: Contains only one member, SimpleParent class. So it will be enough to pass SimpleChild instance to serialization class and it will cause an endless loop which will trigger an exception that will be handled inside the benchmarked method.
- ComplexParent and ComplexChild Classes are built with the same rules of SimpleParent and SimpleChild classes.
Execution Time
Memory Allocation
CPU Usage
** Utf8Json and NetJSON are causing a stack-overflow exception that cannot be handled so I skipped their serialization error tests.**
Deserialization Error— Results
Get Simple Class and Complex Class serialized first and then put illegal characters at the beginning and at the end of these JSON strings. That should fail the deserialization
Execution Time
Memory Allocation
CPU Usage
Results Overview — Success Actions
Results Overview — Error Actions
TL;DR;
- SpanJson displays the most remarkable results, but among many pros, there is one big con: SpanJson got support with only .NET Core 3.1 and NET5.0. So if you are looking for a library for another platform, you should skip this one.
System.Runtime.Serialization.Jsondisplay the worst performance results alongside with FastJson, these both libraries should not be considered.- If execution time is the most crucial aspect of your application, you should consider SpanJson, NetJson*, Utf8Json*.
- If memory allocation is the biggest key factor for you then SpanJson, NetJson*,
System.Text.Jsonand Protobuf-Net should be considered. - In my considerations, I am giving more weight to the data of the first two tables (Serialization and Deserialization success) because most of the serialization/deserialization actions are results with success, so these results will be the performance cost of using those libraries in our web applications most of the time.
* NetJson and Utf8Json are causing stack-overflow exceptions that cannot be handled in the Serialization Error scenario so they both not measured for this scenario. But, keep in mind that such scenario of parent-child object loop is not common at all, I just used it because it was my only option to cause an error for all serialization libraries. So I wouldn’t rule them out because of this issue.
Last words
This research is just a reminder for you to challenge your use of the default serializer you are using. And most important, it reminds you that Json.NET (Newtonsoft.Json) has lots of features that have piled up over time that you are probably not using. It’s likely that moving to a different library could improve your application’s performance significantly.
It seems like Microsoft improved their serialization performance by moving to System.Text.Json from System.Runtime.Serialization.Json but it is not enough compared to the less shiny libraries, such as SpanJson, NetJSON, Utf8Json, and Protobuf-Net which display remarkable results.
Just Eat Takeaway.com is hiring! Click here to learn more about our job opportunities
