Ollion’s Notes from Snowflake Summit Part 6: Snowflake and AWS Collaboration

Historically, Snowflake has been renowned for its capabilities as a data warehouse, a consumer of s3 data lakes, and a pivotal data source for machine learning (ML). However, the landscape has significantly evolved, and Snowflake now offers a many more features to enhance collaboration with AWS.

All diagrams from the Snowflake session

From Data Warehouse to Comprehensive Data Solutions

Initially, Snowflake’s strength lay in its robust data warehousing capabilities. It ingested from AWS data lakes (where the data was originally landed), curated the information and acted as an essential data source for BI/ML applications. Today, Snowflake has expanded its offerings to include:

• Data Warehouse and Data Lake: Snowflake now integrates seamlessly with both data warehouses and data lakes, providing a unified platform for diverse data management needs via services like iceberg tables.
• Data Engineering and Pipelines: The integration supports complex data engineering tasks and streamlined pipelines, enhancing the efficiency of data processing workflows.
• Collaboration: Facilitating collaboration across teams and organizations, Snowflake’s platform supports cooperative data initiatives.
• AI/ML: Snowflake’s advanced AI/ML capabilities, powered by integrated tools, enable sophisticated data analysis and predictive modeling.
• Applications: With its growing ecosystem, Snowflake supports a wide range of applications, from traditional data apps to cutting-edge AI-driven solutions.

Key Integrations Enhancing Capabilities

Snowflake’s expanded functionalities are supported by several key integrations and technologies:

• Iceberg Support: Snowflake’s support for Apache Iceberg enhances its data lake capabilities, allowing for efficient and scalable data storage and management.
• Integrated Data Lakes with AWS: Seamless integration with AWS data lakes ensures smooth data flow and accessibility across platforms.
• Hybrid Cloud Architectures: Snowflake’s flexible architecture supports hybrid cloud environments, facilitating data management across on-premises and cloud infrastructures.
• Snowpark: This feature offers a powerful environment for developing complex data processing tasks and ML models (using languages other than SQL). With Snowpark, users can leverage the full potential of AWS SageMaker for ML.
• Snowpark Container Service and Cortex AI: Designed for developing and deploying data applications, these services enhance the creation of intelligent, data-driven applications.
• GenAI and LLM: Generative AI and large language models (LLM) are now integral to Snowflake’s platform, pushing the boundaries of what’s possible with AI and ML.
• Snowpipe Streaming: This feature supports streaming data and low-latency use cases, ensuring real-time data processing and analysis.

Hybrid Cloud and Data Mesh: A Logical Evolution

Integrating Snowflake into hybrid cloud environments and as part of a data mesh architecture is a logical step forward. The hybrid cloud approach allows organizations to leverage the best of both on-premises and cloud-based solutions, optimizing resource use and ensuring data accessibility.

Similarly, a data mesh architecture decentralizes data ownership, promoting better scalability, autonomy, and collaboration across business units:

I love the idea of storing data in S3, virtualizing it in an AWS-deployed Snowflake, and then sharing it cross-cloud with other divisions within the enterprise. In other words, store once in AWS, but still share across the company without “forcing” anyone into a particular format or platform.

Thought this was particularly great concept

Addressing Data Access Considerations

While Snowflake’s integration with AWS offers immense potential, there are important data access considerations to keep in mind:

• Role/Service Trust Relationships: Snowflake accesses AWS S3 based on role/service trust relationships, which might not provide fine-grained access or detailed user identity information.
• External Functions and Access: Similar challenges exist with external functions and external data access, which can complicate governance and security.

However, Snowflake Horizon provides a solution to these challenges, offering seamless governance across the entire data ecosystem. This capability ensures that organizations can maintain robust data security and compliance while leveraging the full spectrum of Snowflake and AWS integrations.

Conclusion

The Snowflake Summit showcased how Snowflake’s collaboration with AWS is revolutionizing data management and analytics. From enhanced data warehousing and data lake capabilities to sophisticated AI/ML integrations and hybrid cloud architectures, Snowflake is paving the way for a new era of data-driven innovation. As organizations continue to explore these possibilities, the potential for transformative growth and efficiency becomes increasingly apparent.

Interested in Data Management within Snowflake? Check out my last post on Dynamic Tables.

About Ollion

At Ollion, we have been a proud Snowflake Service Partner for almost a decade. Our mission is to connect companies and capabilities worldwide, helping ambitious organizations achieve game-changing breakthroughs without losing sight of the people impacted. We offer a unique point of view as an independent, straightforward partner backed by a global team of client partners, sales, engineering, delivery, and more.

Let me know if you attended and want to talk more about Snowflake Summit 2024!