Naveen NelamaliWhat is Adaptive Query Execution and How it Improves PySpark PerformanceAdaptive Query Execution (AQE) is one of Spark 3.0's greatest features. It reoptimizes and adjusts query plans based on runtime statistics…4 min read·Mar 25, 2024----
Naveen NelamaliPySpark — SparkAI and Its Mose Used Methods with ExamplesThis is the second article in a series on leveraging the pyspark-ai package in PySpark to use the natural English language. In my first…7 min read·Mar 25, 2024----
Naveen NelamaliinTowards AIUsing English SDK in PySparkHave you ever wondered what PySpark AI English SDK is and how it simplifies using PySpark with out learning complex SQL and DataFrame API·5 min read·Mar 24, 2024----
Naveen NelamaliinSparkByExamplesSpark Executor Memory Overhead: Understanding & Best PracticesSpark executor memory overhead refers to additional memory allocated beyond the user-defined executor memory in Apache Spark·5 min read·Jan 14, 2024----
Naveen NelamaliinSparkByExamplesWhen to use Hive Partitioning and Bucketing?Hive Partition is a way to split a large table into smaller tables based on the values of a column(one partition for each distinct value)4 min read·Jan 12, 2024----
Naveen NelamaliinSparkByExamplesDo you know you can use Variables in Hive?Hive variables are key-value pairs that can be set using the set command and they can be used in scripts and Hive SQL.4 min read·Jan 12, 2024----
Naveen NelamaliinSparkByExamplesHow to Union Pandas DataFrames?In pandas, you can use the concat() function to concatenate or union the Pandas DataFrames along with a particular axis.3 min read·Jan 12, 2024----
Naveen NelamaliinSparkByExamplesWhy Avoid UDFs in Spark & PySpark?User-Defined Functions (UDFs) in Spark can incur performance issues due to serialization overhead, necessitating the conversion of data…4 min read·Jan 12, 2024----
Naveen NelamaliinSparkByExamplesWhat are the Specific Roles of Spark Driver and ExecutorHave you ever wondered what the different roles of Apache Spark Driver and Executor play when running your application in a distributed…3 min read·Jan 12, 2024----
Naveen NelamaliinSparkByExamplesIntroduction to Spark-Submit: A Comprehensive Guide to Submitting Spark ApplicationsEvery thing you need to know about spark-submit5 min read·Jan 11, 2024----
