Hi Dmitry,
Thanks for the benchmarks. I’m one of the current mantainers of Scala Pickling. You’ve tried out this example with an old version of
Scala Pickling. When using Scala Pickling 0.11.0-M2, I get these results:
Benchmark Mode Cnt Score Error Units
PicklingBenchmark.deserialization_events_1k avgt 10 1094455.489 ± 13273.369 ns/op
PicklingBenchmark.deserialization_events_2k avgt 10 2015868.481 ± 40486.775 ns/op
PicklingBenchmark.deserialization_events_4k avgt 10 4538038.419 ± 424100.764 ns/op
PicklingBenchmark.deserialization_events_64k avgt 10 97526595.789 ± 9963613.430 ns/op
PicklingBenchmark.deserialization_events_8k avgt 10 10945550.818 ± 1050460.634 ns/op
PicklingBenchmark.deserialization_site_1k avgt 10 1038916.144 ± 25246.795 ns/op
PicklingBenchmark.deserialization_site_2k avgt 10 1826954.949 ± 376641.686 ns/op
PicklingBenchmark.deserialization_site_4k avgt 10 3630649.873 ± 209418.607 ns/op
PicklingBenchmark.deserialization_site_64k avgt 10 59468895.062 ± 6477382.669 ns/op
PicklingBenchmark.deserialization_site_8k avgt 10 7511857.898 ± 1091546.135 ns/op
PicklingBenchmark.serialization_events_1k avgt 10 4754260.519 ± 421141.496 ns/op
PicklingBenchmark.serialization_events_2k avgt 10 9302045.974 ± 1652369.240 ns/op
PicklingBenchmark.serialization_events_4k avgt 10 19543863.571 ± 2547904.312 ns/op
PicklingBenchmark.serialization_events_64k avgt 10 601843327.050 ± 319789216.090 ns/op
PicklingBenchmark.serialization_events_8k avgt 10 48920593.979 ± 7062061.817 ns/op
PicklingBenchmark.serialization_site_1k avgt 10 1401528.801 ± 305977.721 ns/op
PicklingBenchmark.serialization_site_2k avgt 10 2232491.542 ± 458695.841 ns/op
PicklingBenchmark.serialization_site_4k avgt 10 4535664.284 ± 1220460.654 ns/op
PicklingBenchmark.serialization_site_64k avgt 10 85861940.888 ± 21306100.374 ns/op
PicklingBenchmark.serialization_site_8k avgt 10 11108586.150 ± 2470581.958 ns/op
PicklingBenchmark.deserialization_events_1k sample 8892 1125841.475 ± 6813.511 ns/op
PicklingBenchmark.deserialization_events_2k sample 4743 2111510.022 ± 16605.272 ns/op
PicklingBenchmark.deserialization_events_4k sample 2068 4844839.118 ± 62895.047 ns/op
PicklingBenchmark.deserialization_events_64k sample 109 95416808.514 ± 3040686.109 ns/op
PicklingBenchmark.deserialization_events_8k sample 939 10717596.217 ± 183904.086 ns/op
PicklingBenchmark.deserialization_site_1k sample 9061 1104361.567 ± 7430.188 ns/op
PicklingBenchmark.deserialization_site_2k sample 6262 1597540.405 ± 10047.919 ns/op
PicklingBenchmark.deserialization_site_4k sample 3221 3108117.539 ± 23855.039 ns/op
PicklingBenchmark.deserialization_site_64k sample 183 56281457.311 ± 765828.248 ns/op
PicklingBenchmark.deserialization_site_8k sample 1388 7218007.793 ± 102108.126 ns/op
PicklingBenchmark.serialization_events_1k sample 2135 4691878.790 ± 33905.144 ns/op
PicklingBenchmark.serialization_events_2k sample 1155 8684761.212 ± 81567.722 ns/op
PicklingBenchmark.serialization_events_4k sample 498 20221145.960 ± 459502.673 ns/op
PicklingBenchmark.serialization_events_64k sample 30 445050606.933 ± 21888685.354 ns/op
PicklingBenchmark.serialization_events_8k sample 221 46196652.597 ± 896680.238 ns/op
PicklingBenchmark.serialization_site_1k sample 6817 1468022.142 ± 9573.794 ns/op
PicklingBenchmark.serialization_site_2k sample 4578 2187230.896 ± 18928.888 ns/op
PicklingBenchmark.serialization_site_4k sample 2248 4464157.153 ± 52100.353 ns/op
PicklingBenchmark.serialization_site_64k sample 124 86037140.645 ± 4723507.896 ns/op
PicklingBenchmark.serialization_site_8k sample 871 11546337.139 ± 314237.408 ns/op
These results are fruit of an ongoing, huge refactoring of Scala Pickling and of course there are still some stuff to be done. I don’t know if it would be as fast as Boopickle and other candidates, because Scala Pickling has too many levels of indirections (different pickle formats, ability to serialize any class hierarchy, etc) and automates a lot of stuff that the rest don’t. Also, please note that during this refactoring the performance has not been tuned. There’s a paper that shows real-world benchmarks **before** the refactoring, you can find it in the README of the repo.
Also, updating this page with the results of jmh would increase the certainty
of these benchmarks, because as you know timing nanoseconds manually is not faithful: it includes a lot of garbage that is incorrectly timed during the samples. I’m glad that you have a new benchmark setup using JMH. Hope you update the results soon.
This post has been really helpful to assess the current situation of all the serialization frameworks in Scala. I think that the community will be really grateful for it. On a side note, could you comment on the hardware used for the benchmarks?
