Member-only story

Streams In Julia: Working With Data On Disk

Working with data that’s still in a file with Julia.

Emma Boudreau
chifi
7 min readJan 7, 2025

--

It is no secret that one of Julia’s biggest shortcomings as a language comes in the form of memory usage. Julia offers high-level dynamic syntax, a robust type system, and is one of the few languages fast enough to join the ‘Petaflop club’, but this is not without cost. The main trade-off for all of these enormous benefits is memory usage. While it is a trade-off that is worth having, it is still a trade-off. Considering that Julia starts itself by using 80MB of memory (most languages use 1–15,) memory is almost always going to be the limiting factor for any Julia project. Memory is the limitation we need to consider most highly when working in the language. I discussed this much more thoroughly in my article late last year, in which we discussed some ways to mitigate this problem in Julia:

While all of these tips help to mitigate the memory usage of data that is inevitably read into memory, another important avenue to consider is keeping that data out of memory in the first place. While it is…

--

--

Emma Boudreau
Emma Boudreau

Written by Emma Boudreau

i am a computer nerd. I love art, programming, and hiking. https://github.com/emmaccode

Responses (1)