Member-only story
The Medallion Architecture
Data is a hot topic in the business world. Everyone wants to talk about the insights and value they can derive from data. There’s a good reason for that; Data is one of the most valuable resources available to today’s companies.
🧞 Who Rules The Data, Rules The World.
With the increased volume of the data, data processing and analysis are becoming more and more time-consuming. Companies are looking beyond the traditional data architectures to meet their on-demand analytical needs.
Databricks tackled the problem using the delta lake framework combined with the Medallion Architecture.
The Delta Lake framework
Delta Lake is an open source project that enables building a Lakehouse architecture on top of data lakes. Delta Lake provides ACID transactions, scalable metadata handling, and unifies streaming and batch data processing on top of existing data lakes, such as S3, ADLS, GCS, and HDFS.
Specifically, Delta Lake offers:
- ACID transactions on Spark with Serializable isolation levels that ensure readers to never see inconsistent data.
- Leverages Spark distributed processing power using the Scalable metadata handling
- A table in Delta Lake is a batch table as well as a streaming source…
