Memory Management and Handling Out of Memory Issues in Spark

This blog is written and maintained by students in the Professional Master’s Program in the School of Computing Science at Simon Fraser University as part of their course credit. To learn more about this unique program, please visit {sfu.ca/computing/pmp}.

Ref: https://2r4s9p1yi1fa2jd7j43zph8r-wpengine.netdna-ssl.com/files/2017/06/01_06.png

Spark Memory Management

To write programs in spark efficiently and with high performance, you will have to go over the memory management in spark. Most of the people either increase the cores, increase the memory of the executor and driver or play around with file size and .repartition() function, but to understand how spark works under the hood, we will dive deep into spark’s memory management.

Memory management inside one node

Memory management inside executor memory.

The first part of the memory is reserved memory, which is 300 Mb. This memory is not used by the spark for anything. So, when you give 1Gb of memory to spark.executor.memory, then 700 Mb is only available for operations. If you want to change the reserved memory, then you have to change the source…