Monitor Models & Beat Drift with Open Tools

You’ve done the hard work with your ML models — that’s the main course. Now it’s time for the best part: the dessert. Our blog on monitoring is the sweet, essential knowledge you can’t afford to miss

8 min readJun 14, 2025

Introduction: The Silent Killer of ML Models — Why Monitoring Matters

Hey guys, Tapan here! Ever deployed a machine learning model that worked perfectly in testing but mysteriously degraded in production? You’re not alone. I’ve seen countless models silently fail because no one was watching for the invisible enemy: drift.

Today, I’m excited to share how you can set up robust monitoring systems using open-source tools to catch drift before it kills your models!

Let’s dive into the world of ML monitoring and learn how to keep your models performing at their best, even as the world around them changes.

Understanding Drift: Data Drift vs. Concept Drift Explained

Before we jump into tools and implementation, let’s clear up what we’re actually looking for.

Data Drift is like changing the rules of a game while playing. It occurs when the statistical properties of your input features change between training and production.

For example:

You trained a loan approval model on applicants with average incomes of $60K, but suddenly your user base shifts to predominantly $120K earners
A retail prediction system trained on normal shopping patterns faces a holiday season surge

# Simple visual check for data drift using histograms
import matplotlib.pyplot as plt
import numpy as np

# Compare feature distributions
plt.figure(figsize=(12, 5))
plt.subplot(1, 2, 1)
plt.hist(training_data['income'], bins=30, alpha=0.7)
plt.title("Training Income Distribution")
plt.subplot(1, 2, 2)
plt.hist(production_data['income'], bins=30, alpha=0.7, color='orange')
plt.title("Production Income Distribution")
plt.tight_layout()

Concept Drift, on the other hand, is more insidious. The input data looks similar, but the relationship between inputs and outputs has changed.

Real-world example:

Consumer preferences change after a global pandemic (same demographic features, different buying behavior)
A fraud detection system faces new types of attacks that don’t match historical patterns

The impact? Your model’s performance metrics plummet, often without any obvious reason why. A model with 90% accuracy can drop to 60% — costing you money, trust, and potentially regulatory compliance.

The MLOps Monitoring Stack: Key Open-Source Tools

Let’s look at our open-source toolkit for fighting drift:

Prometheus: The time-series database that collects and stores your model’s metrics.

Pros: Scalable, reliable, excellent query language
Perfect for: Storing prediction counts, latency, and basic performance metrics

Grafana: The visualization layer that transforms Prometheus data into actionable dashboards.

Pros: Highly customizable, beautiful visualizations, alerting capabilities
Perfect for: Creating ML-specific dashboards and triggering alerts

Evidently AI: The ML-specific monitoring library that specializes in drift detection.

Pros: Purpose-built for ML monitoring, excellent drift detection, interactive reports
Perfect for: Detecting various types of drift with minimal setup

WhyLabs: Open-source observability for ML (with a managed option).

Pros: Built for data scientists, profile-based monitoring
Perfect for: Teams who need a more comprehensive solution

The ideal stack combines these tools: Prometheus collects raw metrics, Evidently AI analyzes for drift patterns, Grafana visualizes everything, and alerts trigger when thresholds are exceeded.

Hands-on: Setting Up Your Monitoring Pipeline

Let’s build a practical monitoring pipeline with these tools!

Step 1: Instrument Your Model

First, we need our model to expose metrics:

from prometheus_client import start_http_server, Summary, Counter
import time
import numpy as np

# Create metrics
PREDICTION_TIME = Summary('prediction_time_seconds', 'Time spent processing prediction')
PREDICTIONS = Counter('predictions_total', 'Total number of predictions', ['model', 'outcome'])
FEATURE_MEAN = Summary('feature_mean', 'Mean value for model features', ['feature'])
# Decorate your prediction function
@PREDICTION_TIME.time()
def predict(features):
    # Record feature stats for drift monitoring
    for feature_name, value in features.items():
        FEATURE_MEAN.labels(feature=feature_name).observe(value)
    # Make prediction (your model code here)
    result = model.predict([list(features.values())])[0]
    # Record prediction outcome
    PREDICTIONS.labels(model='fraud_detector', outcome=str(result)).inc()
    return result
# Start Prometheus HTTP server to expose metrics
start_http_server(8000)

Step 2: Configure Prometheus

Create a prometheus.yml configuration:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'ml-model'
    static_configs:
      - targets: ['localhost:8000']

Launch Prometheus:

docker run -p 9090:9090 -v /path/to/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus

Step 3: Set Up Grafana Dashboard

Create a Grafana dashboard with panels for:

Model prediction throughput
Feature distribution over time
Accuracy metrics (if ground truth is available)
Drift indicators

Here’s a sample query for visualizing prediction outcomes:

sum(rate(predictions_total{model="fraud_detector"}[5m])) by (outcome)

Detecting Drift in Practice: Statistical Methods & Specialized Tools

Now for the real magic — automatically detecting drift!

Statistical Methods

The simplest approach uses statistical distance metrics:

Kolmogorov-Smirnov (KS) Test: Measures maximum difference between cumulative distributions
Population Stability Index (PSI): Quantifies distribution shifts
Jensen-Shannon Distance: Measures similarity between probability distributions

Let’s implement drift detection with Evidently AI:

from evidently.dashboard import Dashboard
from evidently.dashboard.tabs import DataDriftTab
from evidently.model_profile import Profile
from evidently.model_profile.sections import DataDriftProfileSection

# Create a dataset reference (baseline)
reference = model_training_data.sample(5000)
# Set up the dashboard
drift_dashboard = Dashboard(tabs=[DataDriftTab()])
# Function to check for drift in new data
def check_drift(new_data):
    drift_dashboard.calculate(reference, new_data, column_mapping=None)
    # Get drift metrics as dictionary
    drift_profile = Profile(sections=[DataDriftProfileSection()])
    drift_profile.calculate(reference, new_data, column_mapping=None)
    report = drift_profile.json()
    # Extract drift metrics for alerting
    metrics = report['data_drift']['data']['metrics']
    drifted_features = [f for f in metrics if metrics[f]['drift_detected']]
    if len(drifted_features) > 0:
        print(f"ALERT: Drift detected in features: {', '.join(drifted_features)}")
        # Trigger your alert/retraining system here
        return True
    return False
# Check new production data every hour
while True:
    new_data = fetch_latest_production_data()  # your function to get new data
    drift_detected = check_drift(new_data)
    time.sleep(3600)  # wait an hour

For more advanced detection, NannyML can estimate model performance without ground truth:

import nannyml as nml

# Load reference and analysis data
reference_data = nml.load_credit_card_data()[0]
analysis_data = nml.load_credit_card_data()[1]
# Initialize the drift calculator
calculator = nml.DriftCalculator(
    feature_column_names=feature_columns,
    timestamp_column_name='timestamp'
)
# Fit on reference data
calculator.fit(reference_data)
# Calculate drift on analysis data
results = calculator.calculate(analysis_data)
# Plot univariate drift for a feature
nml.plots.univariate_drift(results, 'income', kind='feature')

From Detection to Action: Alerting and Automated Retraining Strategies

Detecting drift is only half the battle. When drift happens, you need an action plan:

1. Alerting Configuration

Set up Grafana alerts to notify your team when drift is detected:

# Alert rule in Grafana
# Name: Data Drift Detected
# Query: max(data_drift_score) > 0.7
# Evaluate: Every 1h For 0m
# Notification message: Data drift detected in model $modelName with score $driftScore

2. Automated Retraining Pipeline

Create an ML pipeline that automatically retrains when drift is detected:

from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.sensors.external_task import ExternalTaskSensor
from datetime import datetime, timedelta

default_args = {
    'owner': 'mlops',
    'depends_on_past': False,
    'email_on_failure': True,
    'email': ['mlops-team@company.com'],
    'retries': 1,
    'retry_delay': timedelta(minutes=5),
}
# DAG that runs when triggered by drift detection
dag = DAG(
    'model_retraining',
    default_args=default_args,
    description='Retrains ML model when drift is detected',
    schedule_interval=None,  # Triggered by drift detector
    start_date=datetime(2023, 1, 1),
    catchup=False,
)
def fetch_new_training_data(**kwargs):
    # Code to fetch recent data for retraining
    return "path_to_new_data"
def retrain_model(**kwargs):
    ti = kwargs['ti']
    data_path = ti.xcom_pull(task_ids='fetch_new_training_data')
    # Retrain model with new data
    # Your model training code here
    return "path_to_new_model"
def evaluate_and_deploy(**kwargs):
    ti = kwargs['ti']
    model_path = ti.xcom_pull(task_ids='retrain_model')
    # Evaluate model quality
    # If good enough, deploy it
    # Your deployment code here
    fetch_data = PythonOperator(
        task_id='fetch_new_training_data',
        python_callable=fetch_new_training_data,
        dag=dag,
    )
    retrain = PythonOperator(
        task_id='retrain_model',
        python_callable=retrain_model,
        dag=dag,
    )
    deploy = PythonOperator(
        task_id='evaluate_and_deploy',
        python_callable=evaluate_and_deploy,
        dag=dag,
    )
    fetch_data >> retrain >> deploy

3. Decision Framework

Not all drift requires immediate action. Use this decision matrix:

Drift TypeSeverityActionMinor data driftLowMonitor closelySignificant data driftMediumRetrain model with recent dataConcept driftHighRetrain model, possibly revisit featuresFeature corruptionCriticalAlert data engineering team, rollback model

Case Study: Monitoring a Deployed Classification Model

Let’s see this in action with a fraud detection model:

Challenge: A fintech company’s fraud detection model slowly degraded from 92% to 78% accuracy over three months.

Monitoring Setup:

Prometheus collecting prediction metrics and feature statistics
Evidently AI checking for data drift weekly
Grafana dashboard showing key metrics

What the Monitoring Revealed:

Feature “transaction_amount” showed significant drift (KS test p-value < 0.01)
Average transaction values were steadily increasing
New transaction patterns emerged on weekends

Action Taken:

Automated drift detection triggered retraining pipeline
New features were added to capture time-based patterns
Model was retrained on recent data including the new patterns

Results:

Accuracy recovered to 91%
Automated monitoring caught subsequent drift events early
Business prevented estimated $300K in fraud losses

Conclusion & Key Takeaways: Maintaining Peak Model Performance

Building monitoring into your ML systems isn’t just a nice-to-have — it’s essential for production models. Remember these key points:

Monitor continuously: Set up automated checks for both data and concept drift
Choose appropriate metrics: Know which statistical tests work for your data types
Create automated responses: Don’t just detect problems — solve them automatically
Use visualization: Make your metrics understandable to both technical and non-technical stakeholders

As ML systems become more critical to business operations, robust monitoring separates successful deployments from costly failures. The open-source tools we’ve covered today provide everything you need to build production-grade monitoring without breaking the bank.

🔍 Interview Questions on ML Model Monitoring

Q: How would you detect concept drift in a model where ground truth is delayed by 30 days?
A: Implement performance monitoring with a 30-day lag window, use proxy metrics for early detection (like prediction distribution shifts), and implement sequential analysis techniques that can detect changing patterns even without immediate ground truth.
Q: What’s the difference between model monitoring and model observability?
A: Model monitoring focuses on tracking predefined metrics over time, while observability provides deeper insights into model behavior, enabling you to investigate unexpected issues and understand why they occur — not just that they occurred.
Q: How would you handle drift in a multimodal model with both text and image inputs?
A: I’d implement separate, specialized drift detection for each modality — statistical tests for numerical features derived from images, embedding distance metrics for text, and techniques like KL divergence to measure changes in the joint distribution of outputs.

⚠️ Common Pitfalls in Model Monitoring

Overlooking feature-level monitoring: Many teams only monitor overall accuracy, missing early warning signs at the feature level.
Setting thresholds without baselines: Establish normal variation patterns before setting alert thresholds to avoid alert fatigue from false positives.
Ignoring seasonality: Make sure your drift detection accounts for normal seasonal patterns to avoid retraining on expected variations.
Missing data quality issues: Often what looks like model drift is actually data pipeline problems. Monitor for schema changes, missing values, and encoding issues.

📚 Further Reading

Evidently AI Documentation — Comprehensive guide to the ML monitoring library
Prometheus for ML Monitoring — Setting up metrics collection
WhyLabs AI Observatory — Open-source data logging for ML
Monitoring Machine Learning Models in Production — In-depth article on monitoring best practices
ML Observability with Grafana and ClearML — Advanced monitoring configurations

The battle against model drift is ongoing, but with these tools and practices, your models can maintain peak performance even as the world changes around them. What monitoring challenges are you facing with your ML models? Share in the comments below!