Go Benchmark : JSON v. ProtoBuf

If you’ve ever heard of ProtoBuf you may be thinking that the results of this benchmarking experiment will be obvious, JSON < ProtoBuf. My interest was in how much they actually differ in practice. How do they compare on a couple of different metrics, specifically serialization and de-serialization speeds, and the memory footprint of encoding the data. I was also curious about how the different serialization methods would behave under small, medium, and large chunks of data.

So first off, some data structures are necessary to test against. I made them using the protobuf definition syntax, and the protoc-gen-go tool. I added some diversification in the data types, but I have doubts it has much real impact.

There are 2 different tests per serialization type:

• Memory footprint of the final byte array after a Marshal
• Benchmark the speeds of the Marshal and Unmarshal operations

Results

Results for data size tests and speed benchmarks of JSON v. Protobuf

These are the raw results of running the tests and benchmarks on my laptop. Although they are not 100% accurate because of resource contention on my machine, I think it is sufficient for our purposes. We are not looking for scientific precision here, just some basic observations.

Analysis

Lets get the first few observations out of the way.

• Protobuf is faster per operation in every action
• Protobuf is smaller
• Protobuf doesn’t get dramatically slower as data size increases

Memory Footprint

First thing that we notice right away is that Protobuf is smaller than JSON, though not so much so that we should condemn JSON for its large size. If someone pointed out this size difference to me, I don’t think it would sway me to stop using JSON. However, an interesting thing to note above is that the amount of memory allocated by Protobuf is closer to the actual size that is used. This is most clear when we look at the larger data sizes.

This shows that the two libraries have different methods for traversing through the struct when encoding. I noticed that as you progressively increase the size of the data, the amount of memory allocated under Protobuf increases by small increments, whereas JSON doubles the allocated size when increasing. This behaviour is not obvious from this benchmark, but I’ll come back to this idea and do a further examination.

Marshal Speeds

Both libraries experience slow downs during the encoding of the structs as they increase in size. JSON time increases quite significantly from the small to the medium data size, and only doubles for medium to large data. Protobuf doesn’t seem to have the same level of slow down. Overall, the results of this section are the least surprising.

Unmarshal Speeds

Now seeing this was a bit of a shock. Particularly because JSON seems to take a serious hit when the data size increases. At the large data size we are looking at times of 14000ns+ ns/op. This is almost 3 times slower than the medium data size. The most interesting part here is just how fast Protobuf is in this case. With fewer allocations and nothing slower than 1000 ns/op. Now this is a result I’m impressed by.

Conclusion

In my first version of these tests, I was just looking at a single data structure of one size. While changing that structure I noticed the speeds of the two types reacted very differently to data size increases. Thats when I added 3 variations in data size. In this short test I tried not to make any grand assumptions about the way this works or why Protobuf is faster. Sometime in the future I would like to come back to this and see a more comprehensive examination of the speeds with variable data sizes.

In regards to whether you should use Protobuf or JSON, I’ll say that it’s not up to me. It depends on your situation. It is very hard to argue against the convenience of JSON. All languages have great JSON support, especially Javascript where it’s the language’s object type. Protobuf offers more than just speed though. It also provides a typed schema for your messages, which could be valuable for safety in the messages you send. So you really have to decide for yourself and your situation. This will only tell you one fact, Protobuf is smaller and faster.

You can find the source code for the tests here. Try changing the data yourself and seeing the results.