Understanding Warehouse Cost and Optimization

Hello All! Thanks for reading my earlier Snowflake blog series. This is the second blog in a new series on Snowflake cost optimization. You can read the first blog here if you haven't read earlier — https://medium.com/snowflake/understanding-snowflake-cost-10165aea767f

Snowflake’s compute or query processing layer is one of the layers of its architecture. Compute is referred to as Warehouse. Warehouses are available in various sizes and their consumption is calculated based on the usage in the form of credits. The earlier blog refers to the credit-to-cost computation. This blog refers to computing the warehouse cost and components that are considered as part of compute cost.

Compute costs represented in the form of credits used for:

• Warehouse Computation — Virtual warehouses or compute consume credits as they execute queries, load data and perform other DML operations. These warehouses are user-managed hence users can control the usage of warehouses.
• Compute for Serverless Services — Snowflake’s serverless features like Snowpipe, Materialized views, Query Acceleration service, and Search optimization etc. use compute resources that are managed by Snowflake instead of user-managed warehouses.
• Compute for Cloud Services — The Cloud Services layer is the topmost layer of the Snowflake architecture. This layer performs metadata services that tie together all the different components of Snowflake to process user requests, login, query display, and more. Cloud Services compute resources are managed by Snowflake.

Virtual Warehouse Compute Cost —

Users can create standard or Snowpark-optimized warehouses based on their workloads. The usage of warehouses is calculated based on the uptime of warehouses and the size of warehouses. The below chart shows the warehouse sizes and the credit consumption for warehouse type for an hour.

You can also set warehouses to scale and add a scaling policy to add nodes based on the workloads.

Serverless Compute —

Serverless credit usage is the result of features relying on compute resources provided by Snowflake rather than user-managed virtual warehouses. These compute resources are automatically resized and scaled up or down by Snowflake as required for each workload.

For these serverless features, which usually require continuous and/or maintenance operations, this model is more efficient, allowing Snowflake to charge based on the time spent using the resources. In contrast, user-managed virtual warehouses consume credits while running, regardless of whether they are performing any work, which may cause them to be over utilized or sit idle.

The below table lists the various serverless features of Snowflake and their required compute details —

Cloud Services Compute —

Usage for cloud services is charged only if the daily consumption of cloud services exceeds 10% of the daily usage of virtual warehouses. The charge is calculated daily (in the UTC time zone). This ensures that the 10% adjustment is accurately applied each day, at the credit price for that day.

Keep the following in mind:

• Serverless compute does not factor into the 10% adjustment for cloud services.
• The 10% adjustment for cloud services is calculated daily (in the UTC time zone) by multiplying daily warehouse usage by 10%.
• The adjustment on the monthly usage statement is equal to the sum of these daily calculations.
• If cloud services consumption is less than 10% of warehouse compute credits on a given day, then the adjustment for that day is equal to the cloud services used by your account. The daily adjustment never exceeds actual cloud services usage for that day. Thus, the total monthly adjustment may be significantly less than 10%.

Compute Cost Optimization —

Snowflake’s compute cost can be optimized using native features and properties to optimize the usage. You can use below features and configurations to optimize the cost —

1. Warehouse offers Auto Suspend and Auto Resume properties.
2. Auto Suspend — This enables users to specify the time to suspend the warehouse if it's not in use. Warehouse cost is computed only when the warehouse is up and running for a given time.
3. Auto Resume — This enables users to specify automatic warehouse resumes when there are loads submitted on the warehouse.
4. Warehouse clustering — There are two types of scaling policies — standard and economy. You can use one to ensure the scaling adds resources as and when workloads are there. This takes care of active loads with warehouses.
5. You can save credits with Auto suspend. This shuts down or suspends warehouses as soon as there are no active workloads. Its recommended to use this property to save credits
6. Use query tags to tag your workloads, you can use this to monitor the credit consumption.
7. You can also use the query parameters to ensure you do not have any long-running queries that keep running on the warehouse. use STATEMENT_TIMEOUT_IN_SECONDS and STATEMENT_QUEUED_TIMEOUT_IN_SECONDS to avoid any long-running queries or wait for queries that will keep the warehouse active.
8. Setup appropriate resource_monitors to monitor the credit consumption, send alerts, and take action to suspend warehouses.
9. Setup monitoring metrics, build dashboards using Snowsight.
10. Snowflake also announced budgets private preview. You will learn more about this in upcoming blog.

Please follow my blog series on Snowflake topics to understand various aspects of data design, engineering, data governance, cost optimizations etc.

About Me :

I am one of the Snowflake Data Superheroes 2023. I am also one of the Snowflake SnowPro Core SME- Certification Program. I am a DWBI and Cloud Architect! I am currently working as Senior Data Architect — GCP, Snowflake. I have been working with various Legacy data warehouses, Bigdata Implementations, and Cloud platforms/Migrations. I am SnowPro Core certified Data Architect as well as Google certified Google Professional Cloud Architect. You can reach out to me LinkedIn if you need any further help on certification, Data Solutions, and Implementations!