The Techlife
Published in

The Techlife

Top 2 tricks for compressing and loading huge datasets

Running out of memory? Huge data frame? here are some solutions

ML data sets are typically quite huge and hardware resources are also quite constrained. In this article, I will be listing an example of common effective ways to reduce the size of datasets.

1. Convert to parquet (for pandas…



Keep up to date with the latest Big Tech News, the latest advances in AI & programming tutorials

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Mostafa Ibrahim

Software Eng. University College London Computer Science Graduate. Passionate about Machine Learning in Healthcare. Top writer in AI