Monitoring Consumption with Operational Reporting in Snowflake
In the age of business strategy fueled by big data, there has been a huge growth of companies, small and large enterprises, moving towards cloud-based data storage solutions, with business leaders seeking greater flexibility, control, and security of their data. This can include data lake as well as cloud-based data warehousing solutions. Consequently, it is essential for companies to be given the ability to track the activity of their databases and warehouses, as billing is determined as a function of this.
Snowflake, a leader in Cloud data warehousing solution, provides the functionality for clients to achieve this via in-built information schema functions that track account metadata — from login and query histories, to credit usage by warehouse and storage information by database. However, the output of each function is only given for a specified time parameter, and the information is not persisted beyond a year. At Cervello, as part of our innovation efforts, we have used Snowflake’s metadata repository to create a methodology for automated updating and persisting of this metadata to create a rich persisted metadata repository in Snowflake. This persisted metadata repository provides information to create operational reporting tables, and a set of dashboards to monitor credit consumption by teams and workloads. These operational dashboards provide information that allow a team to strategize around tactical time and credit usage, thus enabling a more streamlined and efficient approach to data warehousing and analytic solutions.
We created a set of scripts to persist the data from Snowflake’s metadata repository in the shared database, then created a set of views that feed operational monitoring dashboards created in the BI tool of your choice. What did you team spend last month on ELT? How much Snowflake credit was spent in creating your BI reports? How quickly is the usage of the sandbox environment growing? How many new users are in Snowflake and how has their usage profile changed? We have that answer for you!
The cloud storage market is expected to double in value by 2022, and our operational reporting suite can ensure that our clients utilise this exciting technology in the most effective way possible.
If you’re interested in learning more about Snowflake, and how Cervello can optimize your cloud data warehouse and reporting efforts, please reach out!
