How Fetch integrated Plotly Dash with Snowflake to Power A Comprehensive Self-Service Analytics Platform
TL;DR: The story of how the Fetch analytics team created more than 25 complex, self-serve data applications for their Analytics Hub platform with Plotly Dash and Snowflake, deployed and hosted on AWS.
Author: Alex Kalish, Tech Lead, Fetch
About Fetch
Fetch is redefining the landscape of loyalty and rewards programs through its innovative approach to integrating technology and shopping. At the heart of Fetch’s success is a platform that caters to millions of consumers, bridging the gap between them and their favorite brands.
The daily interactions of our users and brands on the Fetch mobile application create an enormous flow of data — several terabytes — which our partners and teammates need to operate and execute programs every day. Access to this data is made possible by cross-functional teams, who play a pivotal role in shaping the analytical narrative both internally and externally with Fetch data.
One of the outcomes of this collaboration is the Fetch Client Analytics Hub. A simple UI development framework using Dash and Snowflake, providing a seamless, intuitive data delivery tool for our partners and stakeholders that parses terabytes of data into hundreds of different slices in a matter of minutes.
Just as important, we wanted any developer to be able to develop a data application quickly and easily, in a declarative way. Our applications use custom components (similar to Dash’s all-in-one components) to abstract much of the complexity of app development, with minimal code to move to a final product. All-in-all, our developers can make a highly complex, self-service, data application in less than 500 lines of code and have it deployed in 24 hours. This article will describe in some detail how we did that.
Simplifying UI development with Dash and leveraging Snowflake’s warehouse capabilities
Fetch’s Client Analytics Hub provides a suite of tools that streamline the data analysis process. Our platform combines Dash’s intuitive user interface development, Snowflake’s powerful data warehouse, and an in-house custom Python package to facilitate strategic data exploration without the complexity of traditional SQL query construction or the need for exponentially growing pre-defined aggregations.
Our platform offers analysts and engineers a declarative method of creating BI tools, defining the parameterized SQL query with complex joins and logic, Dash Core Components (e.g., drop-downs, radial dials, and date slides), Plotly visuals, and other typical BI functions.
The platform utilizes Dash’s routing to create a home for all reports, making navigation and discovery easy for end users. There are dozens of independent applications that, in aggregate, get queried thousands of times per day by end users. Utilizing the features of Dash, development is simplified using the following pattern:
The Pillars of Our Analytics Platform: A custom Python library seamlessly integrating Dash, Plotly, and Snowflake
Simplified Creation of Filter Objects: A custom class transforms user inputs into dynamic Dash filters that generate the required predicates tailoring data retrieval to individual needs while remaining in scope of each application’s purpose. The developer simply chooses a Snowflake table and the fields from that table in a Python list and the filters are dynamically generated by button push, in order of which they are placed within the application’s core code. This allows end users to focus on only the fields they want while allowing access to any field they need. When instantiated, these class variables are then passed to the next class. On the front end, we utilize refresh buttons to avoid pre-downloading inputs for the drop downs, buttons, etc. which re-query Snowflake based upon the current selections, allowing for fast but dynamic retrieval of inputs for selection in the UI.
Seamlessly Integration of Inputs into SQL Queries: An additional custom class takes the inputs from the previous class based on the user’s manipulation of the UI and integrates them into logic to easily create the user’s desired outcome in SQL. This query gets executed against Snowflake and returns the desired output. We utilize Snowflake caching to re-query the results for multiple cuts without rescanning large tables by storing the query id in a Dash object.
Easy Visual Creation and Integration: A final custom class uses the outputs to declaratively re-query the cache with additional stored queries allowing users to aggregate the data quickly based on the analytical use case. The developer declaratively selects a visual type (line graph, bar chart etc.), and whether they want a download button or accompanying table. The tables, visuals, and assets are all pre-formatted to match Fetch colors and standards. They will automatically be ordered and fitted to the app based on the order declared and executed by the analyst.
The engine behind the scenes: Snowflake operations
A simple connector class manages the apps pushdown to Snowflake, tagging, and any other necessary features that are available in Snowflake today:
Connection Management: This feature seamlessly handles connections to the Snowflake data warehouse, ensuring secure and efficient data transactions.
Query Execution: The feature parses and runs the SQL statements generated from user inputs, retrieving data and then passing it to Dash. It allows for dynamic returns based on the user’s need, either returning a query id to the output for access to the Snowflake cache or the data itself.
Below is an example of an app created in 500 lines or less:
An example: Generating audiences based on user criteria (500 lines of Python code)
A user can dynamically add drop downs as needed. After hitting refresh, Dash callbacks will retrieve the fields from Snowflake as needed to avoid pre-loading drop down parameters
This specific report allows users a second set of options to exclude data based on the same criteria, in essence, creating anti-joins without the need for understanding SQL. Once the user is ready with their inputs, they can run the report which outputs the desired visuals as well as the query ID and text for reference and discussion with analysts.
Summary
We have over 25 different applications tied to our Analytics Hub home page. These applications have enabled our teams to track offer performance, diagnose application errors, provide meaningful audiences to partners, develop the perfect sales pitch, deliver shopper insights, and summarize KPIs quickly and easily, in a visually appealing way.
These data apps provide a way for anyone, at any technical level, to gather insights and understand their business in a few minutes without writing tickets or slacking an analyst. The simplicity and power of Dash combined with Snowflake, enables our developers to build high quality, bespoke, web applications in no time.
With a little magic, we have made Dash and Snowflake a highly effective combination for our analytics team!
The team consists of Alex Kalish — Tech Lead, Tom Vanichachiva — Development Lead, Rebecca Reicherter — Product Lead, and major contributors Kajal Amarnani, Christian Carmellini, and Saketh Kumar Kappala.
