Apache Spark

In our last blog, we tried to understand about using the spark streaming to transform and transport data between Kafka topics. After reading that many of the readers asked us to give a brief description of RDDs in Spark which we used. So, this…

Spark + AI Summit was the world’s largest big data event focused entirely on Apache Spark assembling 4,000 people(best engineers, scientists, analysts, and executives) from over 40 countries attending to share their knowledge and receive expert training on…

In this blog we will learn how to tuning yarn with spark in both mode yarn-client and yarn-cluster,the only requirement to get started is that you must have a hadoop based yarn-spark cluster with you. In case you want to create a cluster you can follow this blog here.

1. Structured Streaming: What is it? &
2. Structured Streaming: How it works?

We got to know 2 major points about Structured Streaming -

In our previous blog post — Structured Streaming: What is it? we got to know that Structured Streaming is a fast, scalable, fault-tolerant, end-to-end, exactly-once stream processing API that helps users in building streaming applications.