AI INDUSTRIALIZATION: DIAMOND-CUT-DIAMOND APPROACH
Co-authored by siddhant khare & Kumarnav
Automating detection & remediation of Data/ML Model drift through Machine Learning itself, even before the availability of Ground Truth!
An Integral tenet of Industrializing AI is to automate the process of Model Monitoring by embedding intelligence into the process of model maintenance, before & after the availability of ground truth. Classical machine learning and data mining methods rely on the assumption that data distribution stays the same in production as it was during model training and operation. This is often not the case!
Dealing with data where its nature changes through time are one of the core problems in Time Series Forecasting & Machine Learning predictions, especially before the availability of ground truth, & for such scenarios, we present a data-driven analytical approach to:
· Identify When the ML model’s health has degraded (when the model needs to be re-trained)
· How can the model be adaptive to the change in production.
It is extremely critical for businesses to make decisions based on non-decayed healthy ML Models especially when the availability of ground truth is delayed & therefore, for such scenarios, we have devised a mechanism to detect these Model/data drifts and adapt to the change through ML framework itself for the following 2 strategic pillars of AI-driven solutions:
· Time Series Forecasting Models &,
· Machine Learning Classifier Estimator
TIME SERIES FORECASTING MODEL
When is it time to retrain your ML model?
We detect the change (drift) with an advanced statistical technique: ADaptive WINdowing (ADWIN) which captures a statistical description of the windows in a Time Series, in an adaptive manner and compares if these windows come from the same distribution or not. Drift detected with ADWIN indicates when the model should be retrained & where should the intervention be made in the Time Series.
How do we strategize the retraining process?
After drift has been detected, we implemented two methods: first, dropping all the data points before the drift detection point from the dataset. Second, dropping the oldest window from the dataset. This leads to the generation of two new datasets based on the original dataset & this process of selective sampling keeps the continuous temporal structure intact which is a pre-requisite for the TSF model.
PANEL CLASSIFICATION MODEL.
When is it time to retrain your ML model?
We estimate & predict model performance in the future even before the availability of its ground truth i.e., the performance of the ML model on the new production data which the model has not learned on. We do so by building stacked hybrid ML models based on how new production data distribution deviates from training dataset distribution.
How do we strategize the retraining process?
Once we identify the performance curve in future time steps, we append the new labeled data using different sampling techniques by providing calculated weights to the prediction samples which lie around the threshold margins of the classifier estimator & accordingly re-training iterations are executed.
