Spark

Introduction to Streaming What is Stream Processing? Stream processing refers to a method of handling and analyzing data in real-time as it is continuously generated or received. It involves the processing of data streams in a sequential and continuous manner, rather than storing and processing data in batches or as static datasets. Stream…

Introduction: Apache Spark is a popular framework for big data processing, but data skew can significantly impact its performance, especially during join operations. Data skew refers to an uneven distribution of data across partitions, leading to imbalanced workloads and reduced efficiency. …

Introduction This article covers Spark UI, and tracking jobs using the Spark UI. …

Automate Spark unit tests using pytest. — This is a spin-off of a previous article where we used the Python unittest framework. In this tutorial, we will do the following. Build a simple ETL using PySpark. Implement unit tests using the Python module pytest. Run automated tests using CI pipeline in Azure DevOps. Publish test results to…

1 . Need for Spark In Hadoop, it is expensive to save to disk for fault tolerance (MapReduce is expensive for iterative and interactive). Invoke a lot of slow disk I/O (read and write) and lack of efficient data sharing. The solution to this is Resilient distributed datasets (RDDs). It is immutable, built through coarse-grained…

Problem Statement: In previous weeks, I have been working with large data processing . I was comparing Relational Database Data vs Data on S3 Bucket. Data size is huge. I am working on AWS environment. Data compare includes counts and Hash calculations for selected column. Data sets contains blob, CSV…

GraphX, a powerful graph processing library in Apache Spark, provides a scalable and efficient platform for analyzing connected data. Combined with the expressive capabilities of Scala, GraphX enables data scientists and analysts to uncover valuable insights from complex graphs. In this article, we will explore graph processing with GraphX in…

Introduction: Apache Spark has revolutionized big data processing, empowering businesses to extract valuable insights from vast datasets. However, harnessing the full potential of Spark requires careful attention to performance optimization. In this article, I will share my personal experiences and insights on how to supercharge your Spark applications and unlock…

Hello, digital adventurers! Welcome to another dispatch from the front lines of the DevOps revolution. As I wade deeper into this fascinating field, my aim is to create a roadmap of my findings for you, my esteemed readers. …