Kafka to Spark Structured Streaming, with Exactly-Once Semantics

Partitioning and Bucketing in Hive: Which and when?

Spark and Airflow with Kotlin

Recently, I was thinking about something new I could learn, and I ended up with two options. The first was…

Data Quality Dashboards: Is Your Data Doing Ok?

Everyone loves data dashboards, right? Fancy visualisations which provide key insights into otherwise opaque data? Of course you do.

While such dashboards are becoming an increasingly commonplace tool for business…

Portable Pipelines with Apache Beam

There are lots of use cases for data processing and analytics pipelines, and nearly as many frameworks to use. Apache Spark is probably the de facto standard these days, but it is far from the only option. In this article, we'll take a look at Apache Flink — one such…

Working With Data

We don’t have to emphasise on the importance of data anymore, frankly it would be waste of time. We should now put our focus…