Overview

In today’s complex data ecosystems, organizations rarely operate within a single platform. Data flows through multiple systems, from source databases like PostgreSQL and MySQL, through orchestration tools like Apache Airflow and dbt, into Snowflake, and finally out to BI tools like Power BI, Tableau, etc. Understanding this complete data journey, known as data lineage, is critical for impact analysis, debugging, and governance.

Snowflake’s External Lineage extends the platform’s native lineage capabilities to include external data sources and destinations. By leveraging the open-source OpenLineage standard, Snowflake now provides a unified view of how data moves through your entire data pipeline, not just within Snowflake itself.

What is OpenLineage?

OpenLineage is an open standard for capturing and sharing metadata about data workflows. It defines a consistent model for emitting lineage events from tools like Airflow, dbt, and Spark. By standardizing lineage capture, OpenLineage enables interoperability across the modern data stack.

How External Lineage Works in Snowflake

External lineage works by accepting OpenLineage-compatible events through a REST endpoint:

POST https://<account_identifier>.snowflakecomputing.com/api/v2/lineage/external-lineage

External tools send lineage metadata to Snowflake, which incorporates this information into the native lineage graph displayed in Snowsight. This creates a connected view showing how data moves and transforms from source to final use.

Key Use Cases

External Lineage addresses several critical scenarios for data teams:

1. Impact Analysis for Data Engineers

Scenario: dbt to Snowflake, Airflow to Snowflake, S3 to Snowflake

External ETL tools such as dbt and Airflow are popular for building data pipelines across multiple source systems. Data engineers need to understand:

• How changes to Snowflake tables will impact downstream datasets and dashboards outside Snowflake (Power BI, Tableau)
• How changes to non-Snowflake sources (Amazon S3, PostgreSQL) will impact downstream datasets in Snowflake

Example: A data engineer modifying a staging table in Snowflake can instantly see which dbt models, Airflow DAGs, and Power BI dashboards depend on that table.

2. Debugging & Troubleshooting for Analysts

Scenario: Snowflake to Power BI/Tableau

Analysts using Power BI or Tableau connect to Snowflake tables to create dashboards. When data quality issues arise, they can use Snowflake’s lineage feature to:

• Trace data back to its source
• Identify transformation steps that may have introduced errors
• Understand the complete data flow from source to visualization

3. Data Governance & Compliance

Scenario: Cross-platform data flow visibility

Governance teams need visibility into data movement and transformation history to ensure compliance with internal policies. External lineage provides:

• Audit trails showing how data flows between platforms
• Documentation of data transformations for regulatory requirements
• Clear ownership and accountability across the data pipeline

Supported Data Sources by Platform

External Lineage supports integration with major data orchestration platforms. Here’s a comprehensive breakdown:

Apache Airflow

dbt (Data Build Tool)

Apache Spark

Setting Up External Lineage

Prerequisites

Before configuring external lineage, you must grant the necessary privileges:

-- Create a dedicated role for lineage ingestion
CREATE ROLE dbt_lineage_role;

-- Grant the INGEST LINEAGE privilege
GRANT INGEST LINEAGE ON ACCOUNT TO ROLE dbt_lineage_role;

-- Assign the role to your service user
GRANT ROLE dbt_lineage_role TO USER dbt_integration_user;

Configuring OpenLineage with dbt

Step 1) Install the OpenLineage-dbt integration

pip3 install openlineage-dbt

Step 2) Configure the transport variables in your YAML configuration file

transport:
  type: http
  url: https://MYORG-DEV_ACCOUNT.snowflakecomputing.com
  endpoint: /api/v2/lineage/external-lineage
  auth:
    type: api_key
    apiKey: eyJ0eXAiOiJKV1QiL...  # Your security token
  compression: gzip

Step 3) Replace standard dbt commands with dbt-ol

# Instead of: dbt run
dbt-ol run

# Instead of: dbt build
dbt-ol build

The integration uses a wrapper script (dbt-ol) that runs dbt and then reads generated artifacts (manifest.json, run_results.json, and optionally catalog.json) to extract metadata and emit OpenLineage events.

Configuring OpenLineage with Apache Airflow

Step 1) Install the OpenLineage Airflow provider

# For Airflow 2.7+
pip install apache-airflow-providers-openlineage

# For older versions
pip install openlineage-airflow

Step 2) Configure the transport variables

Option A: Via environment variables

export OPENLINEAGE_URL=https://MYORG-DEV_ACCOUNT.snowflakecomputing.com
export OPENLINEAGE_ENDPOINT=/api/v2/lineage/external-lineage
export OPENLINEAGE_API_KEY=eyJ0eXAiOiJKV1QiL...
export OPENLINEAGE_NAMESPACE=my-airflow-instance

Option B: Via YAML configuration (openlineage.yml)

transport:
  type: http
  url: https://MYORG-DEV_ACCOUNT.snowflakecomputing.com
  endpoint: /api/v2/lineage/external-lineage
  auth:
    type: api_key
    apiKey: eyJ0eXAiOiJKV1QiL...
  compression: gzip

Step 3) Run your DAGs normally. Lineage events are emitted automatically during DAG execution by hooking into Airflow’s task lifecycle.

Lineage Flow Architecture

The following diagrams illustrate how data lineage flows through a typical enterprise data stack with Snowflake External Lineage enabled:

End-to-End Lineage Flow:

Data Flow Example:

Preview State & Current Limitations

Important: External Lineage is currently in Preview (Open) state, available to all accounts on Enterprise Edition or higher.

Current Capabilities

• Accept OpenLineage-compatible events via REST endpoint
• Visualize external nodes in Snowsight lineage graph
• Distinguish between native Snowflake objects and external objects
• One-year retention period for external lineage events

Authentication Options

Snowflake supports multiple authentication methods for the external lineage endpoint:

1. Key Pair Authentication (Recommended for automation)
2. OAuth
3. API Keys

Example curl request with key pair JWT:

curl -i -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer eyJ0eXAiOiJKV1QiL..." \
  -H "Accept: application/json" \
  -H "User-Agent: myApplication/1.0" \
  -H "X-Snowflake-Authorization-Token-Type: KEYPAIR_JWT" \
  -d "@request_body.json" \
  "https://MYORG-DEV_ACCOUNT.snowflakecomputing.com/api/v2/lineage/external-lineage"

Removing External Lineage

To remove lineage relationships, send a DELETE request:

DELETE https://<account_identifier>.snowflakecomputing.com/api/v2/lineage/external-lineage?
  sourceNamespace={namespace}&
  sourceName={FQN}&
  sourceDatasetType={type}&
  targetNamespace={namespace}&
  targetName={FQN}&
  targetDatasetType={type}

Note: Users must have the DELETE LINEAGE privilege on the account to remove lineage.

Conclusion

External Lineage represents a significant step forward in Snowflake’s data governance capabilities. By embracing the OpenLineage standard, Snowflake enables organizations to achieve true end-to-end visibility across their entire data ecosystem.

Key Takeaways

Unified Visibility — View lineage from source databases through Snowflake to BI dashboards in a single interface

Open Standards — Built on OpenLineage, ensuring compatibility with major orchestration tools

Easy Integration — Simple configuration for dbt and Airflow with minimal code changes

Enterprise Ready — Available on Enterprise Edition with robust security and access controls

Getting Started

1. Ensure you’re on Snowflake Enterprise Edition or higher
2. Grant INGEST LINEAGE privilege to your service accounts
3. Configure your dbt or Airflow environment to emit OpenLineage events
4. View your complete lineage in Snowsight

As this feature evolves from Preview to General Availability, expect enhanced capabilities including column-level lineage, native BI tool connectors, and custom lineage editing for truly comprehensive data governance.

External Lineage is currently in Preview. For the latest information, visit the Snowflake Documentation.

Achieving Cross Platform Lineage on Snowflake was originally published in Snowflake Builders Blog: Data Engineers, App Developers, AI, & Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.