Geek Culture
Published in

Geek Culture

Ingesting Data from Qlik Replicate using Azure Databricks

Real-World Data Engineering Sample Project

Ingesting Data from a CDC Replication tool (Qlik) that streams changes from a source database and land it on a Data Lake as JSON files into a Data Lake Analytics platform.

The goal of this article is to describe a sample real-world data engineering use case for beginner data engineers that would like to know what type of work a data engineer would be doing.

In this case, I describe how my team built a data ingestion pipeline to process…

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store