Forgetful pandas 🐼

Pandas memory reports can mislead — learn how to see the real usage and save memory!

Source: pixabay.com

The pandas library is THE tool for data cleaning, data prep, and data analysis in Python. Once you find your way around it’s expansive API, it’s a joy to work with. 🎉

Pandas stores your data in memory, which makes operations zippy! 🏎 The downside is that a large dataset might not fit into your machine’s memory, grinding your work to a halt. ☹️

Often, it’s handy to know how much memory your pandas DataFrame occupies. I’ve been working with pandas and teaching it to students for several years. I even wrote a little book on getting started with it. I had heard rumors that the memory usage shown by default wasn’t always accurate. 🤔 But there’s so many data things to explore and so little time, that I only recently got around to investigating. I was shocked to learn how huge the memory undercount can be. 😲

In this article I’ll show you a few ways to get the true memory usage for your pandas objects. Then I’ll share eight solutions for when your data won’t fit into memory. 🎉

Red pandas are cute, too. Source: pixabay.com