Conditional Lineage: Merging Joins, Filters, and Transformations into Recursive Column Lineage

Extending column lineage with joins and filters to reveal how data truly flows through SQL pipelines.

Why Conditional Lineage Matters

Column lineage is often presented as a simple question:

“Which source columns feed this target column?”

But in real SQL pipelines, the answer depends on how data moves — under which joins, filters, and transformations the flow is active.

When migrating, optimizing, or validating pipelines, you don’t just want to know what maps to what — you want to understand which source attributes truly matter for producing a given target table or column.

That’s where conditional lineage comes in: extending recursive column lineage to include query conditions (JOIN and WHERE clauses) alongside traditional column mappings.

💡 Not a Medium member? You can read this article for free using this friend link.

From Static Mappings to Conditional Flows

Traditional lineage captures only explicit column mappings in the SELECT clause:

SELECT
    c.customer_id AS customer_key,
    SUM(o.amount) AS…