Efficient memory management when training a deep learning model in Python

How to use big data on a small computer using Tensorflow, Python and iterators

Sisyphus work; Licensable by Wiki commons license https://commons.wikimedia.org/wiki/File:Friedrich_John_nach_Matth%C3%A4us_Loder_Sisyphus_ubs_G_0825_II.jpg

The amount of data in the current business world is increasing every day. There are new data sources to merge, more rows to append and new columns to concatenate.

Unfortunately, for a typical data scientist or a machine learning engineer, the pace at which one can buy a new laptop, or computer or expand a cloud service is usually not on par with the pace of new data coming in. One’s hardware may feel like Sisyphus: trying to keep the rock not crushing him (memory in the computer not get filled up) while still maintaining his work (training a deep learning model).

In this article, I will present a logic about how to think about fitting big amounts of data into small amounts of RAM.

All the codes and examples can be found in my repo:

GitHub - Eligijus112/big-data-ml: Project to showcase how to use TF data generators

The objective we will be trying to solve is to create a model that predicts the fare amount of a…