Flare: A Lightweight Data Monitoring Tool

Easy to set up monitoring alerts through Slack, Zapier, or any other webhook target. TL;DR? Skip ahead to the code.

In most of our work, as in life, the only constant is change. That’s especially true when creating models, where global events or changes in customer behavior, broken data engineering pipelines, and other unforeseen circumstances can quickly degrade the quality of your model’s predictions. In the simplest case, a null column may break your model entirely, or in a potentially more harmful scenario, the distribution of inputs may change relative to your training data. Early detection of these changes in incoming data is key to ensuring the integrity of your models.

While there are a few open source tools for detecting data drift, they require you to host a streaming aggregation pipeline to capture inputs and outputs, store this sensitive production data somewhere secure, and maybe even build your own analysis and alerting systems. Universally, these tools are complicated to maintain and require a significant amount of time to set up.

Introducing Flare

The Domino R&D team is launching Flare, a lightweight data monitoring system that makes it easy to set up monitoring alerts through Slack, Zapier, or any other webhook target. It works with your preferred tools for hosting and executing your model, and doesn’t capture (or require you to capture) any of your data, so you don’t have to worry about data security or storage.

We built this for anyone involved in deploying or managing model operations, including data scientists, making it easy to get started without having to build out custom pipelines to alert on inference traffic for each of your models.

Flare works by running two types of checks on incoming data: one to make sure the features are in the correct format, and a second to determine if they fall within statistical bounds calibrated using your training data. Keep an eye out for our upcoming release, which will incorporate additional data drift tests.

Try It Out!

The only requirements for running Flare are that your data can be formatted as a Pandas DataFrame and you have access to an alerting system such as Slack, Zapier, or a webhook. Navigate to our open-sourced Github project for detailed instructions on setup or follow along with the quickstart below. We also have an example model in the repo if you want to get Flare up and running in just a few minutes!

Get Started

You first need to install Flare, and ensure that you’re working in Python version 3.8 or higher.

pip install domino-flare

You’ll then use the Flare baseline function on the Pandas DataFrame containing your training data to automatically create JSON files with the constraints and statistical properties that will be used to test your inference data.

from flare.generators import baseline as flare_baseline
flare_baseline(<your data frame>)

Next, you’ll configure the target for your alerting system. We have detailed instructions on setting this up for Slack and Zapier, though you can use a webhook to send alerts to any URL that accepts POST requests.

#example for a Slack alert
from flare.alerting import SlackAlertTarget
alert_target = SlackAlertTarget("<web hook URL for your slack workspace>")

After you’ve set up this alerting target, you’ll add the Flare context to your inference code. This will run the checks, while also capturing unexpected inference exceptions and sending these as alerts.

# This is the Flare context. It loads the Constraints/Statistics 
with Flare(“<your model name>”, <input features>, alert_target):
# Your normal inference code goes here
     output = model.predict(<input features>)

That’s it for the setup! Now each time your model is executed, Flare will send you an alert if the incoming data features violate any of the constraints or contain statistical outliers.

Here’s an example of what that alert looks like in Slack:

You can now rest easy knowing that you’ll be alerted right away if there are issues with your inference data that could affect your model’s performance. From here you may choose to re-train your model, or determine if your pipeline for working with incoming data needs additional modification.

Keep in Touch!

If you have comments or feature requests, feel free to submit them to our GitHub repository, or contact us @domino_research on Twitter. We’d love to hear from you!