Ingesting Data from Qlik Replicate using Azure Databricks
Real-World Data Engineering Sample Project
Ingesting Data from a CDC Replication tool (Qlik) that streams changes from a source database and land it on a Data Lake as JSON files into a Data Lake Analytics platform.
The goal of this article is to describe a sample real-world data engineering use case for beginner data engineers that would like to know what type of work a data engineer would be doing.
In this case, I describe how my team built a data ingestion pipeline to process…