Practical programming advice

Parsing 90GB JSON Files Using Streams in .NET

Examples using both System.Text.Json and Newtonsoft.

Geek Culture
Published in
7 min readAug 20, 2022

--

Searching for how to parse a huge JSON file (90GB) doesn’t really turn up encouraging results. Some answers are outright wrong, misleading, saying you shouldn’t do it at all, or even that it can’t be done.

I came across StackOverflow and Quora answers such as “you definitely need to load the whole file onto disk”, or “You should never do it because of the way JSON is designed, you need to load it all into memory before you can parse it”.

Well, we’ll see about that. Hint: none of those statements are true.

I typically steer clear of writing tutorials as most of you have probably already noticed. But I’ll make an exception for this case due to its interesting feats.

The actual client project that inspired this article is written in Java, but I’ve translated parts of our solution to .NET as that’s my preferred platform. The approach is more or less the same, anyway.

🔔 Want more articles like this? Sign up here.

What we’ll look into.

Depending on your current skill level and experience with JSON, streams, and zip files…

--

--

Nicklas Millard
Geek Culture

I mostly write to "future me" sharing what I learn and my opinion on software development practices. youtube.com/@nmillard | open for contracts in May 2024.