Member-only story
Building Out Of Memory File Readers In Julia
Demonstrating a slowly improving out-of-memory concept for Data Science.
Julia is a powerful language that balances performance and usability incredibly well, making it an excellent choice for scientific computing, data analysis, and beyond. However, one of the language’s well-known shortcomings is its memory usage, and this is often felt when working with large data-sets. One of the projects I have had looming in the background for a while creates a unique solution to this by creating lazy data-structures that are able to make stacking changes to data in a pipeline and then call those changes on command as the structure is generated. This is the main feature of one of my planned packages: AlgebraFrames
.
Though the project has gone through several naming schemes, the purpose and usage of the package has remained consistent. Just like the other core packages in the Chifi ecosystem, this package is meant to spawn an entire ecosystem with much more expansive capabilities than the base package has to offer. Despite the rather slow…