Nethaji Kamalapuram – Medium

Nethaji Kamalapuram

Nethaji Kamalapuram

Spark Unit Testing

Spark Unit Testing with PySpark

May 16

May 16

Nethaji Kamalapuram

Spark Dynamic Resource Allocation

Spark Scheduling: Allocation Strategies

May 16

Spark Dynamic Resource Allocation

May 16

Nethaji Kamalapuram

Spark Speculative execution

What is Spark Speculative Execution?

May 16

Spark Speculative execution

May 16

Nethaji Kamalapuram

Spark Broadcast variables and Accumulators

Broadcast Variables:

May 16

Spark Broadcast variables and Accumulators

May 16

Nethaji Kamalapuram

Spark Data Caching, Repartition, Coalesce, Data Frame Hints

Let's deep dive into the data caching concept in spark, why it's needed, different caching techniques and parameters, what is best fit for…

May 15

Spark Data Caching, Repartition, Coalesce, Data Frame Hints

May 15

Nethaji Kamalapuram

Spark Dynamic Partition Pruning

Dynamic Partition Pruning for Optimized Joins

May 15

Spark Dynamic Partition Pruning

May 15

Nethaji Kamalapuram

Adaptive Query execution — spark 3.0

This article explores Apache Spark 3.0’s Adaptive Query Execution (AQE) feature and its benefits for optimizing query performance.

May 13

Adaptive Query execution — spark 3.0

May 13

Nethaji Kamalapuram

Spark Memory allocation & Management

Lets talk about how memory allocation works for spark driver and executors. Understanding driver and executor memory allocation is crucial…

May 13

Spark Memory allocation & Management

May 13

Nethaji Kamalapuram

Spark Jobs, Stages, Tasks & Slots

Key Concepts:

May 12

Spark Jobs, Stages, Tasks & Slots

May 12

Nethaji Kamalapuram

Apache Spark Runtime Architecture

Apache Spark is a distributed computing platform.

May 11

Apache Spark Runtime Architecture

May 11

Nethaji Kamalapuram

Nethaji Kamalapuram

As a Lead Data Engineer, I specialize in cutting-edge data engineering tools and possess the expertise to design highly standardized architectures.

Following

Help
Status
About
Careers
Press
Blog
Privacy
Terms
Text to speech
Teams