Ingesting Raw Data with Kafka-connect and Spark Datasets

Ronald Ángel
Oct 15, 2019 · 6 min read
Photo by Fabio Ha on Unsplash

In this blog post, I will explain how we use Kafka-connect and spark orchestrated by platforms like Kubernetes and airflow to ingest raw data. You will get some insights about the advantages of storing your source data as it comes and how to ease data ingestion when you need early data exploration and a business case is not entirely defined.

Why do you need to…