SparkByExamples

SparkByExamples

Naveen Nelamali

Spark Executor Memory Overhead: Understanding & Best Practices

Spark executor memory overhead refers to additional memory allocated beyond the user-defined executor memory in Apache Spark

5 min readJan 14, 2024

--

Spark Executor Memory Overhead: Understanding & Best Practices

--

Naveen Nelamali

When to use Hive Partitioning and Bucketing?

Hive Partition is a way to split a large table into smaller tables based on the values of a column(one partition for each distinct value)

4 min readJan 12, 2024

--

When to use Hive Partitioning and Bucketing?

--

Naveen Nelamali

Do you know you can use Variables in Hive?

Hive variables are key-value pairs that can be set using the set command and they can be used in scripts and Hive SQL.

4 min readJan 12, 2024

--

Do you know you can use Variables in Hive?

--

Naveen Nelamali

How to Union Pandas DataFrames?

In pandas, you can use the concat() function to concatenate or union the Pandas DataFrames along with a particular axis.

3 min readJan 12, 2024

--

How to Union Pandas DataFrames?

--

Naveen Nelamali

Why Avoid UDFs in Spark & PySpark?

User-Defined Functions (UDFs) in Spark can incur performance issues due to serialization overhead, necessitating the conversion of data…

4 min readJan 12, 2024

--

Why Avoid UDFs in Spark & PySpark?

--

Naveen Nelamali

What are the Specific Roles of Spark Driver and Executor

Have you ever wondered what the different roles of Apache Spark Driver and Executor play when running your application in a distributed…

3 min readJan 12, 2024

--

What are the Specific Roles of Spark Driver and Executor

--

Naveen Nelamali

Introduction to Spark-Submit: A Comprehensive Guide to Submitting Spark Applications

Every thing you need to know about spark-submit

5 min readJan 11, 2024

--

Introduction to Spark-Submit: A Comprehensive Guide to Submitting Spark Applications

--

Naveen Nelamali

Remember These when Accessing Matrix Elements in R

When accessing elements in an R matrix, there are several important things to remember to ensure accurate and effective data manipulation:

2 min readNov 11, 2023

--

--

Naveen Nelamali

Read Snowflake table into Spark DataFrame — Spark by {Examples}

In this Snowflake data warehouse article, I will explain how to read a Snowflake table into Spark DataFrame and learn different connection…

3 min readFeb 28, 2020

--

Read Snowflake table into Spark DataFrame — Spark by {Examples}

--

Naveen Nelamali

Which Spark HBase Connector to use in 2019? — Spark by {Examples}

This tutorial explains different Spark connectors and libraries to interact with HBase Database and provides a Hortonworks connector…

5 min readAug 30, 2019

--

Which Spark HBase Connector to use in 2019? — Spark by {Examples}

--

SparkByExamples is a BigData, Data Engineering, Machine Learning, AI, and Cloud platform knowledge-sharing publication platform with the intent to share the knowledge that we come across in real-time projects.
Connect with SparkByExamples
Editors
Naveen Nelamali
I am seasoned Data Engineer Architect with extensive knowledge in Apache Spark, Hive, Kafka, Azure Databricks, AWS and GenAI
Spark Examples

Help
Status
About
Careers
Press
Blog
Privacy
Terms
Text to speech
Teams