Member-only story
Don’t Crash Your App: Load Records from the Database in Batches for Better Performance
Save your Python app’s performance by efficiently loading query
This article is about optimizing the communication between your Python app and a database so that your app runs smoothly and your database server doesn’t melt. This article addresses a common inefficiency: the habit of loading all data from a query at once.
When faced with queries that returns a lot of records it’s often impractical or even impossible to load all returned records. Instead of loading all results in memory and then processing row-by-row, in this article we’ll find out how to load many small chunks. Instead of loading 1 million records and processing we’ll load a 400 batches of 2.500 records each! This way your app doesn’t have to load all results in memory which has clear benefits:
- Enhanced memory usage
- Better perceived response times
- Reduces database stress
We will also take a look under the hood and dive into the technical details of this technique, showing you exactly how it works behind the scenes. Let’s code!
