End-to-end Azure data engineering project — Part 1: Project Requirement, Solution Architecture and ADF reading data from API
This series comprises 4 articles showcasing the comprehensive data engineering practice on the Azure platform. It encompasses the utilization of Azure Data Lake, DataBricks, Azure Data Factory, Python, Power BI, and Spark technology. In this initial part (part 1), we will acquaint ourselves with the dataset and delve into the solution architecture of Azure and Databricks. Furthermore, we will establish the initial environment setup and execute the loading of data from the source system to Azure Data Lake via the API approach using ADF.
Please review the related articles in the series here:
End-to-end Azure data engineering project — Part 1: Project Requirement, Solution Architecture and ADF reading data from API
End-to-end Azure data engineering project — Part 2: Using DataBricks to ingest and transform data
End-to-end Azure data engineering project — Part 4: Data Analysis and Data Visualization (Power BI)
