Handling Slowly Changing Dimensions (SCD) using Delta Tables

Handling the challenge of slowly changing dimensions using the Delta Framework

For a long time, the Kimball method has been a standard for dimensional data modeling techniques. As per Kimball “ The notion of time pervades every corner of the data warehouse”. What does this mean in the context of data analytics? At a high level, modern analytics can be seen as the aggregation of constantly changing data with the passage of time. The problem is that the constantly changing data not only includes new additions but also changes to previous datasets.

Overall dimensional data modeling groups data into two major categories:

Facts — This data represents limitless data sets that store measurements of entities. It contains data that is essential for quantitative analysis and decision-making. Fact tables frequently have columns that join to other tables (dimensions) for reference.

Dimensions — This data represents relatively limited data sets that provide descriptive information regarding the measurements performed in the fact tables. In comparison to fact tables, dimensions evolve at a lot slower pace. This is the reason they are commonly referenced as “slowly changing dimensions”.

Kimball’s approach involves the creation of a star schema based on facts and dimensions. Because of the denormalized structure, the star schema is well suited for analytics use cases…no…