Keeping Track of Snowflake Costs

In the data community, by now everybody has heard of Snowflake — the cloud-based data platform. Snowflake’s popularity can be attributed to its scalability, performance and zero management overhead, but it’s not uncommon for users to be met with a large invoice at the end of the month if they are not familiar with how Snowflake works or best practices for their particular data use-case.

The problem

As developers of our own data platform at Matatika, we have had our fair-share of experience with Snowflake, as we test each of our supported extractors against Snowflake on a regular basis. When we set this process up initially, we were running each sync every day, which ended up costing us over £200 for the first month alone— not ideal for just tests! With some small tweaks and a bit of understanding of our usage, we were able to reduce the cost significantly. As our test workload increases with each new connector we add, we can see that increased spend is inevitable. This got us thinking — what else are we missing that could help to reduce our Snowflake operating costs for running these tests?

Ideally, we want to be able to run our extractor tests against Snowflake both as frequently and with as low a cost as possible. There will always be a trade-off here, but our goal is to identify and reduce unnecessary Snowflake compute resource utilisation. This would enable us to either flat-out reduce our spend or potentially ramp-up our testing frequency for the same outgoing cost.

The solution

We came up with a solution help us monitor our Snowflake spend, leveraging our own infrastructure and some dbt models written by SELECT that pull out billing information. We packaged these up in a file bundle (analyze-snowflake-monitoring) containing resources deployable to a Matatika workspace — including a store to connect to Snowflake with, a dbt pipeline to run the models, and some datasets as to visualise the data as insights. Now have a way to monitor our Snowflake spend from our own app using some nice charts and graphs!

Try it yourself

You can monitor your own Snowflake spend in your Matatika workspace in the same way we did here:

1. Open your workspace
2. In the Lab, install the Snowflake Cost Explorer file bundle plugin from the Available tab in the Plugins view
3. Wait for the workspace to update its configuration
4. Follow the new task to provide your Snowflake credentials and save changes
5. Go to the Pipelines view and run the Snowflake Costs pipeline
6. Switch over to the app to view your data visualised as insights