Photo by israel palacio on Unsplash

In this series of articles, I will walk you through a brief overview of the exciting new changes that the Apache Spark engine has gone through with its 3.0 release, more specifically the Adaptive Query Execution (AQE).

In this article, we will look into the motivations, challenges, and solutions that AQE seeks to solve. Also, we will address one topic in more detail — dynamically coalescing shuffle partitions.

So what is Apache Spark?

If you are reading this article, chances are that you already know what Apache Spark is (and if you do, please skip this intro). However, I wouldn’t be methodical enough if I…

João Pedro Santos

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store