Ollion’s Notes from Snowflake Summit Part 5: Exploring Dynamic Tables
At the Snowflake Summit, one of my favorite sessions provided insights into the evolving landscape of data orchestration and the new capabilities offered by dynamic tables compared to legacy streams and tasks.
Dynamic Tables vs. Streams & Tasks: A Comparative Analysis
1. Level of Orchestration Control
Streams and tasks in Snowflake have been the go-to for many organizations for years, offering a robust but somewhat manual approach to data management. With streams, you capture change data, and tasks allow you to schedule and execute SQL statements in response to those changes. This approach gives a significant level of control over the orchestration process but requires meticulous management and oversight.
Dynamic tables, on the other hand, represent a declarative approach to data orchestration. Instead of managing every aspect of the data pipeline manually, you define your requirements, and Snowflake takes care of the orchestration. Unfortunately, this automation includes deciding when to perform incremental vs. full refreshes based on the complexity of the operations involved, so be sure to keep your DTs simple and modular because you cannot directly control it.
This abstraction simplifies the orchestration process and reduces the operational overhead.
2. Speed of Refresh
The speed at which data is refreshed is critical in any data pipeline. With streams and tasks, the refresh speed largely depends on how the tasks are scheduled and the frequency of the polling intervals. This can lead to latency in data availability, especially if the orchestration is not optimized.
Dynamic tables optimize this process by allowing Snowflake to handle the refresh logic. If the transformation logic is simple, incremental refreshes are used, ensuring that data is updated swiftly. For more complex transformations, a full refresh might be necessary, but even this is managed efficiently by Snowflake. The speaker highlighted the importance of breaking out complex CTEs (common table expressions) to simplify the transformation logic and leverage incremental refreshes more effectively.
Concept: “Data Activation” and Completing Feedback Loops
The session also introduced the concept of “Data Activation,” which revolves around the idea of making data actionable as soon as it is available. This is a crucial aspect of completing feedback loops within an organization. By ensuring that data is immediately available for analysis and decision-making, businesses can respond faster to changes and opportunities.
Dynamic tables play a pivotal role in this by eliminating the need for batch job scheduling. Instead, you define the desired data freshness (lag relative to the source data), and Snowflake ensures that the data is kept up-to-date accordingly. This seamless integration and activation of data close the feedback loop, providing real-time insights and driving more informed business decisions.
Implementing CDC with Snowpipe and Dynamic Tables
One of the session’s highlights was the discussion on using Snowpipe and dynamic tables for CDC. Snowpipe is Snowflake’s continuous data ingestion service, which works perfectly with dynamic tables to handle change data capture. By combining these technologies, organizations can ensure that their data warehouses are always current, reflecting the latest changes in source systems.
The speaker also mentioned the possibility of replacing dbt (data build tool) orchestration with dynamic tables while still maintaining the models within dbt. This hybrid approach leverages the strengths of both technologies, providing a powerful and flexible data pipeline without the need for traditional job scheduling.
Conclusion
The session on dynamic tables at the Snowflake Summit highlighted a significant shift in data orchestration practices. By moving towards a more declarative approach with dynamic tables, organizations can simplify their data pipelines, reduce operational overhead, and achieve faster data refreshes. This not only enhances data activation but also ensures that feedback loops are completed efficiently, driving better business outcomes.
As we continue to explore the capabilities of dynamic tables and their integration with other tools, it’s clear that Snowflake is paving the way for more agile and responsive data management solutions.
Speaking of Data Management, go back and read the most significant part of the series on Iceberg tables.
Very relevant to Ollion, Snowflake also discussed it’s collaboration with AWS. Read about it in Part 6.
About Ollion
At Ollion, we have been a proud Snowflake Service Partner for almost a decade. Our mission is to connect companies and capabilities worldwide, helping ambitious organizations achieve game-changing breakthroughs without losing sight of the people impacted. We offer a unique point of view as an independent, straightforward partner backed by a global team of client partners, sales, engineering, delivery, and more.
Let me know if you attended and want to talk more about Snowflake Summit 2024!