When I first started out in data engineering some three years ago, I was so eager to get my first pipeline up and running. I had heard that GCP was in style and I knew that streaming data was the name of the game. So I set out to build…

When I first started learning Scala Spark a couple of weeks back I was overwhelmed with the number of “prerequisites” required to package your code into a working JAR file that could be passed to a cluster. I installed Metals, SBT, ran the Spark shell locally, set up Docker containers…

In this tutorial we will set up an Apache Airflow workflow that spins up an auto terminating Spark cluster on AWS Elastic Map Reduce (EMR) running Scala Spark. The pipeline ingests and performs a number of transformations on a CSV formatted dataset in S3. …