Member-only story

Building Out Of Memory File Readers In Julia

Demonstrating a slowly improving out-of-memory concept for Data Science.

Emma Boudreau
chifi
10 min readJan 7, 2025

--

Julia is a powerful language that balances performance and usability incredibly well, making it an excellent choice for scientific computing, data analysis, and beyond. However, one of the language’s well-known shortcomings is its memory usage, and this is often felt when working with large data-sets. One of the projects I have had looming in the background for a while creates a unique solution to this by creating lazy data-structures that are able to make stacking changes to data in a pipeline and then call those changes on command as the structure is generated. This is the main feature of one of my planned packages: AlgebraFrames .

Though the project has gone through several naming schemes, the purpose and usage of the package has remained consistent. Just like the other core packages in the Chifi ecosystem, this package is meant to spawn an entire ecosystem with much more expansive capabilities than the base package has to offer. Despite the rather slow…

--

--

Emma Boudreau
Emma Boudreau

Written by Emma Boudreau

i am a computer nerd. I love art, programming, and hiking. https://github.com/emmaccode

No responses yet