How Snowflake enables its providers to automatically replicate their data across the Data Cloud using Auto fulfillment — Marketplace

Published in

Snowflake Builders Blog: Data Engineers, App Developers, AI/ML, & Data Science

5 min readSep 22, 2022

Opinions expressed in this post are solely my own and do not represent the views or opinions of my employer.

Overview

Snowflake’s most important goal is to deliver on Snowflake’s promise of a cross-region, cross-cloud service. We want to ensure that every Snowflake list is able to automatically route data, code and application to the consumer’s region in a timely and cost effective manner.

Before, providers needed to manage manually the data replication according to the requests they receive from consumers. Means, they need time and people to do it.

What is it ?

Auto Fulfillment automatically manages fulfillment of data across clouds and regions. Sharing data between clouds and regions is as easy as sharing within a single deployment/region. Execution is initiated when there is an actual request in a cloud/region and validated by the provider.

Considerations when using listing Auto-fulfillment

Auto-fulfillment enforces a 10TB limit on the size of the database being replicated. On Public Listing, Auto-fulfillment is currently in Public Preview and it is available for Standard and Paid listings and is not supported for personalized listings

It’s also available for Private Listing. Auto-fulfillment is not currently supported for Gov Cloud.

If you are using a database that has already been manually replicated to other regions/accounts, you have to set up replication groups. Or disable the replication and use the full benefits of aut-fulfillment.

https://docs.snowflake.com/en/sql-reference/sql/create-replication-group.html

Customers who have signed up for Snowflake via AWS Marketplace, GCP Marketplace, or Azure Marketplace are limited to creating accounts and secure shared areas in these clouds only. Replication for regions outside of these areas will fail.

Why does it matter ?

Auto-fulfillment makes cross-region and cross-cloud sharing easy.

The benefits for Providers :

No account silos.
No additional regions to manage.
Near zero management with an automatic replication.
Reduce manual efforts.
Share data securely across clouds and regions.
Can share data with others in any region whatever the cloud provider they choose without creating silos of Snowflake accounts.
Make data available in all regions where desired.
Pay for implementation only when there is demand.

The benefits for Consumers :

Get data immediately without having to wait for the provider to fulfill data manually.
Can request data in any region without worrying about where the provider’s data resides.
No additional pipelines are required and the overhead and operating costs are minimal.

General benefits :

Across all multi-tenant deployments.
Available for sharing from Multi-tenant (MT) into Virtual Private Snowflake (VPS) deployments.
Reduce time to market.

How does it work ?

When a public list is available in multiple regions, providers can enable auto-running. When creating private lists, the system automatically detects cross-region accounts and allows users to enable Auto fulfillment.

Auto fulfillment automatically creates a snowflake secure shared area in the cloud/remote region, enables and manages replication. Data is only replicated when a consumer requests it.

Provider creates listing :

The Type of the listings must be Standard or Paid listings.

Provider sets up auto fulfillment configuration :

Chooses regions according to cloud providers, and frequency.

Remember that if you have already replicated the data manually, all options under “Data Product Fulfilment to Visible Regions” will not be visible.

Provider publishes listing :

Making it available to others (public -all, private -select accounts).

Consumer discovers data and “Gets data” from listing in their local region :

Auto fulfillment creates “Secure Shared Area” in requested region :

Auto fulfillment replicates the requested databases and creates shares :

Customer mounts the share once created :

As customers request other regions the same process repeats for regions where customer demand exists :

What about the pricing ?

You will only pay for storage and egress if and where there is customer demand, and you can build these costs into your pricing for custom lists and free sample shopping lists.

You can easily monitor the cost replication on the listing using Snowsight UI.

Scroll down to the region availability section
Click manage
Click “monitor replication cost”