Practical programming advice
Parsing 90GB JSON Files Using Streams in .NET
Examples using both System.Text.Json and Newtonsoft.
Searching for how to parse a huge JSON file (90GB) doesn’t really turn up encouraging results. Some answers are outright wrong, misleading, saying you shouldn’t do it at all, or even that it can’t be done.
I came across StackOverflow and Quora answers such as “you definitely need to load the whole file onto disk”, or “You should never do it because of the way JSON is designed, you need to load it all into memory before you can parse it”.
Well, we’ll see about that. Hint: none of those statements are true.
I typically steer clear of writing tutorials as most of you have probably already noticed. But I’ll make an exception for this case due to its interesting feats.
The actual client project that inspired this article is written in Java, but I’ve translated parts of our solution to .NET as that’s my preferred platform. The approach is more or less the same, anyway.
🔔 Want more articles like this? Sign up here.
What we’ll look into.
Depending on your current skill level and experience with JSON, streams, and zip files…