Cost control using Snowflake Budgets — Part 1
In the fast-paced digital landscape of today, businesses rely heavily on cloud services to fuel their operations and foster innovation. Cloud technology offers unparalleled flexibility, scalability, and accessibility, allowing companies to streamline processes and reach new heights of efficiency. However, amid the convenience lies a challenge: controlling the costs associated with these services. As organizations harness the power of the cloud, managing expenses has become a crucial aspect of their strategies. This is Part 1 of Budget series blog, in this blog, I delve into Snowflake budget (Public preview) and in Part 2 I will cover how it is different than Snowflake Resource Monitor service on credit/cost control, exploring best practices, and real-world examples that will help optimize cloud spending without compromising on performance or innovation.
To note: Snowflake Budget is only available as Public Preview on AWS as of now for all Snowflake editions (except VPS). The budget feature is not available on Azure/GCP including of All cloud Trial accounts. (as on 10/10/23)
As the name suggests, Budget is defined to set control on how your Snowflake costing on resources including of Storage and serverless can be limited with certain credit quota given to each projects/load types/environments however you want to categorize it. Budget is account level object and if you have multiple Snowflake accounts then enabling budget will be different at each account level, I will try to cover all scenarios here as far as budget implementation is concerned.
Budget Types
1. Account Level Budget
2. Custom Budget
In above diagram, Account level budgets is shown to be created for dedicated Snowflake accounts for DEV , QA and PROD environments. Each environment is hosted on dedicated Snowflake Account hence budget for each account. Custom budget is not at account level but customized to environments or projects within single Snowflake account.
Account Level Budget: Setting up this budget works on account level and may not be good use case where Snowflake account is used in big enterprise having multiple enterprise functions using isolated Snowflake resources under one or multiple Snowflake accounts mapped to one organization. However this can be best use case where Account level budgeting can be set and OPEX or FINOPS team informed/altered about daily Snowflake usage.
Note: Following SQL’s are available on Snowflake Documentation and referred from there. https://docs.snowflake.com/user-guide/budgets
-- Courtesy for following SQLs , Snowflake Documentation https://docs.snowflake.com/user-guide/budgets#label-enable-budgets
-- STEP 1 -- Activate Budget on Account level
-- you must use ACCOUNTADMIN Role to actiavte budget on Account
USE ROLE ACCOUNTADMIN;
CALL snowflake.local.account_root_budget!ACTIVATE();-- STEP 2 -- Adding Account budget of 1000 Credits
CALL snowflake.local.account_root_budget!SET_SPENDING_LIMIT(1000);-- STEP 3 -- create email notification integration
-- Note: it is must to verify your email id before creating email notification and using the email
-- please refer to Snowflake documentation on verifying email using snowsight or classic console
CREATE NOTIFICATION INTEGRATION budgets_notification_integration
TYPE=EMAIL
ENABLED=TRUE
ALLOWED_RECIPIENTS=('youremail@domain.com');-- STEP 4 -- Grant the USAGE privilege on the notification integration
GRANT USAGE ON INTEGRATION budgets_notification_integration TO APPLICATION snowflake;-- STEP 5 -- link notification integration to account budget
CALL snowflake.local.account_root_budget!SET_EMAIL_NOTIFICATIONS(
'budgets_notification_integration',
'youremail@domain.com');Checking Account level Budget in Snowsight if it is enabled
- Using ACCOUNTADMIN role
- Go to Admin, Usage
- Go to Budget
- Check spend (indicates usage of credits from 12:00 AM UTC first day of calendar month) Note: for account level the usage is backfilled and shown here
- Check Interval (indicates number of days the counter started for budget even if you create budget in middle of month interval will show as 50%, indicating next rest of budget cycle will be done on 12:00 AM UTC following calendar month
Custom Budget: Setting up custom budget is good use case on allocating budget for each projects or projects within each environment in big enterprise Snowflake accounts where certain budget related roles can be granted to each projects platform or admin team (not Snowflake admin) to manage budgets and costings. For example, we have project 1 in an enterprise and can be related to function owning the project 1, similar to project 1 , enterprise function can own multiple projects and each project may share compute with dedicated databases or separate compute with dedicated databases, the budget setting approach may be different for each use case as per enterprise and different functions within using Snowflake. It is always necessary to avoid noisy neighbor issues in case project 1 is using huge credits and project 2 from different function is asked to share the cost as tracking of credits is not done at that level, with budget, it will be more convenient and easy to allocate credit and track the spend for FINOPS or correctly chargeback to consumers who are using more credits. following consideration while setting up custom budget.
Custom Budget High level approach
Considering above high level approach, lets detail out how custom budget can be created here.
Approach, Consider we create role for each project/enterprise function budget admin so that respective project team can manage all aspects of setting up and monitoring the budgets. I am demonstrating the example using Budget Admin role created for Project 1 owned by enterprise function for e.g. SALES.
-- Create Custom Budget based on Single Multi Account strategy as shown in diagrams
-- In following example, we will create Project1 Budget for PROD, UAT , QA and DEV envrionments considering all environments in Single Snowflake account
-- Incase your Production account is different as shown in diagram then you must create separate Budget for PROD account and other account budget can be shared for nonprod environments
-- STEP 1 -- Check which budgets exists in accounts
USE ROLE ACCOUNTADMIN;
SELECT SYSTEM$SHOW_BUDGETS_IN_ACCOUNT();--STEP 2 -- create custom role for each project managing project
-- considering each project has to manage their own budget and should not see other project budgets
-- we use project 1 database for example DEV in this case and create schema budget_schema
USE ROLE ACCOUNTADMIN;
CREATE ROLE budget_admin_project1;--STEP 3 -- grant role to create budget
GRANT DATABASE ROLE SNOWFLAKE.BUDGET_CREATOR TO ROLE budget_admin_project1;--STEP 4 -- create budget schema isolated from other schemas in projects
-- own database, you may create schema in BUDGET database as well based on approach of centralized vs decentralized budget management
-- Here we are creating budget schema within existing databases of project1 or entirely separate database can be created to manage budget
USE ROLE SYSADMIN;
CREATE SCHEMA DEV.budgets_schema;--STEP 5-- make sure APPLYBUDGET privilege is given to budget admin role
USE ROLE ACCOUNTADMIN;
GRANT APPLYBUDGET on DATABASE DEV to role budget_admin_project1;
GRANT APPLYBUDGET on SCHEMA DEV.SALES_RAW to role budget_admin_project1;
GRANT APPLYBUDGET on WAREHOUSE COMPUTE_WH to role budget_admin_project1;
GRANT CREATE SNOWFLAKE.CORE.BUDGET ON SCHEMA DEV.SALES_RAW TO ROLE budget_admin_project1;-- STEP 6 -- ensuring either accountadmin or other use mapped to admin role else you wont be able to switch role to budget admin user :)
GRANT role budget_admin_project1 to role accountadmin;-- STEP 7 -- Switching context as Budget Admin for SALES function and creating first budget for project 1 in SALES function
USE ROLE budget_admin_project1;
USE SCHEMA DEV.budgets_schema;
CREATE SNOWFLAKE.CORE.BUDGET project1_budget();-- STEP 8-- setting up limit given by SALES enterprise function of 1000 credits
-- Note that this is overall limit for Calendar month and includes project1 in all environments of SALES enterprise function
CALL project1_budget!SET_SPENDING_LIMIT(1000);
-- STEP 9-- setting up email notification to receive alerts on budget exhuastion incase
-- You must create notification integration for email and verify your email id using snowsight email verification to complete Step
CALL project1_budget!SET_EMAIL_NOTIFICATIONS('budgets_notification_integration','youremail@domain.com');
-- STEP 10 -- Adding storage resources here, can be one mentioned from table above "Supported Snowflake objects/services for Custom Budget"
CALL DEV.budgets_schema.project1_budget!ADD_RESOURCE(SYSTEM$REFERENCE('TABLE', 'SALES_RAW.EMP', 'CALL', 'applybudget'));
-- STEP 11 -- Adding compute resources here, can be one mentioned from table above "Supported Snowflake objects/services for Custom Budget"
CALL DEV.budgets_schema.project1_budget!ADD_RESOURCE(SYSTEM$REFERENCE('WAREHOUSE', 'COMPUTE_WH', 'CALL', 'applybudget'));-- STEP 12 -- Grant the ADMIN instance role to your custom budget_admin_project1role:
USE ROLE ACCOUNTADMIN;
GRANT SNOWFLAKE.CORE.BUDGET ROLE DEV.budgets_schema.project1_budget!ADMIN TO ROLE budget_admin_project1;-- STEP 13 -- INCASE budget monitoring needs to be managed by other user/roles then use following SQL
-- here budget_viewer_project1 ROLE is created and assigned to Budget viewer role
-- Enables a role to view budget usage data.
USE ROLE ACCOUNTADMIN;
GRANT APPLICATION ROLE SNOWFLAKE.BUDGET_VIEWER TO ROLE budget_viewer_project1;
-- STEP 14 -- INCASE budget administration needs to be managed by other user/roles then use following SQL
-- here budget_admin_project_all ROLE is created and assigned to Budget Admin role
-- Enables a role to modify the properties of a budget including spending limit, list of email addresses to receive notifications, mute notifications setting, and the objects in the group.
GRANT APPLICATION ROLE SNOWFLAKE.BUDGET_ADMIN TO ROLE budget_admin_project_all;
Viewing Custom Budget in Snowsight
Understanding Custom Budget Costings
Nothing comes free :) , in order for Snowflake to monitor daily Budget spend and track usage against budget, some services are running background and charged on usage like REPLICATION service for example.
When you enable budget on account, Snowflake runs serverless background tasks (_MEASUREMENT_TASK and _BACKFILL_TASK) that collect credit usage data for the account budget and custom budgets in your account. These tasks are billed to your account. Please check SERVERLESS_TASK_HISTORY view for understanding cost incurred by these serverless budget processes.
Resources used
- Snowflake Documentation https://docs.snowflake.com/user-guide/budgets
- Snowflake Access Control and Privileges https://docs.snowflake.com/en/user-guide/security-access-control-privileges
- Own Snowflake experience
- Conceptual PoC’s done using Snowflake BUDGET and RESOURCE GROUP documentation, eventually Snowflake decided not to use RESOURCE GROUP with BUDGET instead add RESOURCES directly to budget.
