Naveen NelamaliSpark Executor Memory Overhead: Understanding & Best PracticesSpark executor memory overhead refers to additional memory allocated beyond the user-defined executor memory in Apache SparkJan 14Jan 14
Naveen NelamaliWhen to use Hive Partitioning and Bucketing?Hive Partition is a way to split a large table into smaller tables based on the values of a column(one partition for each distinct value)Jan 12Jan 12
Naveen NelamaliDo you know you can use Variables in Hive?Hive variables are key-value pairs that can be set using the set command and they can be used in scripts and Hive SQL.Jan 12Jan 12
Naveen NelamaliHow to Union Pandas DataFrames?In pandas, you can use the concat() function to concatenate or union the Pandas DataFrames along with a particular axis.Jan 12Jan 12
Naveen NelamaliWhy Avoid UDFs in Spark & PySpark?User-Defined Functions (UDFs) in Spark can incur performance issues due to serialization overhead, necessitating the conversion of data…Jan 12Jan 12
Naveen NelamaliWhat are the Specific Roles of Spark Driver and ExecutorHave you ever wondered what the different roles of Apache Spark Driver and Executor play when running your application in a distributed…Jan 12Jan 12
Naveen NelamaliIntroduction to Spark-Submit: A Comprehensive Guide to Submitting Spark ApplicationsEvery thing you need to know about spark-submitJan 11Jan 11
Naveen NelamaliRemember These when Accessing Matrix Elements in RWhen accessing elements in an R matrix, there are several important things to remember to ensure accurate and effective data manipulation:Nov 11, 2023Nov 11, 2023
Naveen NelamaliRead Snowflake table into Spark DataFrame — Spark by {Examples}In this Snowflake data warehouse article, I will explain how to read a Snowflake table into Spark DataFrame and learn different connection…Feb 28, 2020Feb 28, 2020
Naveen NelamaliWhich Spark HBase Connector to use in 2019? — Spark by {Examples}This tutorial explains different Spark connectors and libraries to interact with HBase Database and provides a Hortonworks connector…Aug 30, 2019Aug 30, 2019