Integrating Evidently AI with MLflow for ML Model Monitoring

In this article, we will explore how to integrate Evidently AI with MLflow to monitor machine learning models.

Step 1: Installation

!pip install evidently==0.4.16 jupyter_contrib_nbextensions==0.7.0 pandas==2.2.1 scikit-learn==1.4.1.post1 jupyterlab==4.1.2 mlflow==2.12.1

Step 2: Import Necessary Libraries

import pandas as pd
from sklearn.model_selection import train_test_split
from evidently.report import Report
from evidently.metric_preset import ClassificationPreset
import json
from sklearn.preprocessing import LabelEncoder
from sklearn.linear_model import LogisticRegression

import mlflow
import mlflow.sklearn
from mlflow.tracking import MlflowClient

Step 3: Load and Prepare the Dataset

df = pd.read_csv("datasets/student-dataset-v1.csv")

le = LabelEncoder()
df['Result'] = le.fit_transform(df['Result'])

X = df.drop(columns=['Result'])  
y = df['Result']  

# Encode categorical variables 
X = pd.get_dummies(X)

# Split data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

Step 4: Train the Model and Generate Evidently AI Report

# Logistic Regression model
lr_model = LogisticRegression()
lr_model.fit(X_train, y_train)

X_train['prediction'] = lr_model.predict_proba(X_train)[:, 1]
X_test['prediction'] = lr_model.predict_proba(X_test)[:, 1]

X_train['target'] = y_train
X_test['target'] = y_test

lr_class_report = Report(metrics=[ClassificationPreset()])
lr_class_report.run(reference_data=X_train, current_data=X_test)

lr_class_report.save("json_reports/lr_report_v1.json")

# Load JSON data
with open('json_reports/lr_report_v1.json', 'r') as file:
    data = json.load(file)

Step 5: Access Metrics from the Evidently AI Report

lr_accuracy_v1 = None
metric_results = data['suite']['metric_results']

for result in metric_results:
    if 'current' in result:
        lr_accuracy_v1 = result['current'].get('accuracy')
        if lr_accuracy_v1 is not None:
            break

if lr_accuracy_v1 is not None:
    print("Logistic Regression v1 Accuracy:", lr_accuracy_v1)
else:
    print("Accuracy not found in the JSON data.")

Step 6: Log Metrics and Model in MLflow

# Log into MLflow
client = MlflowClient()

# Set experiment
mlflow.set_experiment('Monitoring with EvidentlyAI')

# Start new run
with mlflow.start_run() as run: 
    # Log metrics
    mlflow.log_metric("accuracy", lr_accuracy_v1)
    
    # Log the model
    mlflow.sklearn.log_model(lr_model, "logistic_regression_model")

    # Print run info
    print(run.info)

Step 7: Visualize in MLflow UI

To visualize the experiments and models logged in MLflow, run the MLflow UI in your terminal:

mlflow ui

By logging metrics and models in MLflow, users can track model versions, compare performance over time, and make informed decisions about model maintenance and updates.

GitHub: https://github.com/Sanjjushri/evidently-ai-poc

Explore more articles on Evidently AI:

1. Monitoring Machine Learning Model Using Evidently AI

2. How to Access Data in an Evidently AI Report