Optimize Spark SQL Joins

Joins are one of the fundamental operation when developing a spark job. So, it is worth knowing about the optimizations before working with joins.
In Data Kare Solutions we often found ourselves in situations to joining two big tables (data frames) when dealing with Spark SQL. In this…

Key factors to consider when optimizing Spark Jobs

Compaction in Hive

This article centers around covering how to utilize compaction effectively to counter the small file problem in HDFS.

Small File Problem

HDFS is not suitable to work with small files. In HDFS a file is considered smaller, if it is…

Hive Design Patterns

Incremental Ingestion

Apache Hive

Apache Hive has evolved as one of the most popular interactive and analytical data store in the Hadoop ecosystem, due to this demand, Hive will play a major role in designing a robust…

Structured Streaming: Kafka integration

This article focuses on explaining how to integrate Spark’s new stream processing engine Structured Streaming with Apache Kafka brokers 0.10 and higher along with all necessary configuration details.

Apache Kafka

NIFI — Monitoring Data Flows

Before moving an Data pipeline in production, the key thing is to designing/deciding an monitoring tool…

Spark SQL — Salient functions in a Nutshell

As, Spark DataFrame becomes de-facto standard for data…

Structured Streaming

Introduction

As streaming frameworks are emerging gradually, it encourages the developers to concentrate on business challenges rather than focussing on potential streaming analytics issues. Structured Streaming is a part of the Apache Spark venture, which…