What is Model Drift

‘Change is the only constant in life’ a famous quote by a Greek philosopher Heraclitus. People, culture, societies and notion are constantly changing as the years go by, what was once the state of the art has now become obsolete. Understanding changes and the ability to quickly adapt, is a vital aspect of business. Mobile phones designed 15 years ago are now considered obsolete in value.

Model drift refers to the degradation of a model’s prediction power due to changes in the environment. This arises essentially when the relationship between the target variable and independent variable changes with time. This drift makes model prediction to become unstable and erroneous with time

In technical terms, using linear regression concept, we map the independent variable x_i to predict the target variable y:

y=ax+b

where, a is the intercept and b_i corresponds to the coefficients for the variable x_i. The assumption is that the mapping of the independent variable x_i to predict the target variable y is constant, that the coefficients B_i and the intercept a do not change with time and that the relationship governing the prediction of the target variable y will be valid for future data as well. This assumption which might be true at the time the model was developed, may no longer remain true due to changes in underlying condition, so when this scenario occurs, the prediction of the model becomes faulty and thus poses a threat to the business or organization using the model

Why do Model’s drift

Every deployed model has the possibility of degrading over time due to the following reasons

1) Variance in deployed data: This is when at deployment, we provide the model with a set of data that is not as clean as the training and testing data used for designing the model

2) Changes in data integrity: changes in data such as formats, renamed fields, new categories can adversely affect models’ performance.

3) Data drift: This occurs when the statistical properties of the predictors change. Changes in demographic, market shift, unexpected world epidemic, changes in data pattern due to seasonality etc, could make the training data less relevant to the current situation thus reducing the precision of the model’s result

4) Concept Drift: When the statistical properties of the target variable changes, the precision of a model prediction can become irrelevant due to changes in the expectation of the end user.

Solution

To successfully address model drift simply retrain the model in a scheduled manner , using recent data, as often as you are able to ascertain the model is likely to decay.

Another way to address model drift is by making the model learn in real time, this is achieved by allowing the model to access data in sequential order as soon as it becomes available rather than training the model with batch data

References

[1] David Weedmark. https://www.dominodatalab.com/blog/machine-learning-model-deployment [published 03 November 2021].

[2] Christophergs https://christophergs.com/machine%20learning/how-to-deploy-machine-learning-models/ [published 17 March 2019].

[3] Thuwarakesh Murallie https://towardsdatascience.com/3-ways-to-deploy-machine-learning-models-in-production-cdba15b00e [published 09 Nov 2021].

[4] Sushrut Shendre https://towardsdatascience.com/ model-drift-in-machine-learning-models-8f7e7413b5633 [published 13 May 2020].

[5] Datatron https://datatron.com/what-is-model-drift/ copyright 2022

What is Model Deployment?

Machine learning model deployment is the process of placing a finished machine learning model into a live environment where it can be used for its intended purpose. Models can be deployed in a wide range of environments, and they are often integrated with apps through an API so they can be accessed by end users.

The process requires diverse range of skill sets, collaboration between different people on different teams and access to tools that would facilitate efficiency. A data science team develops the model, another team validates it, and engineers are responsible for deploying it into its production environment

Deployment is the third stage of the data science lifecycle (manage, develop, deploy and monitor), This step is crucial as every aspect of a model’s creation is performed with deployment in mind. Models are usually developed in an environment with carefully prepared data sets, where they are trained and tested

Tools for Model Deployment

Python language is typically used in the research and production environment because of the rich data science and data processing ecosystem. The scikit-learn pipelines and Sparks Pipelines with MLeap are often used for modelling.

“Shadow Mode” and “Canary deployment” are two separate ways to deployment. The shadow mode allows you to capture inputs and prediction of your model before releasing, this allows you to analyse the result in house and detect bug without consequences. Whilst the Canary approach allows for gradual release of model to small fraction of end users for testing to minimise mistakes. A web service such as a Flask web framework can also be used to deploy a Model

Platform as a Service (PaaS) or Infrastructure as a Service (IaaS) as well as (AWS, Google, Microsoft) can be used for deployment

Steps for deploying a Machine Learning Model into Production

1) The data science team develops the model by selecting algorithm, setting its parameters, and training it on prepared cleaned data

2) The model is then tested, reviewed, and validated by another team usually the business unit,1)

a) They test the model on a set of fresh data to ensure that, result received are comparable with those received initially when the data was being trained.

b) They review the training documentation to ensure the methodology satisfy the required standard specified by the organisation and that data used to design and test the model corresponds to the requirement of the end users.

c) Validation is then completed usually by the compliance team who would dictate what data can be used, how the data would be processed, stored, and documented

3) The model is then passed on to Engineers for deployment into its production environment where there is access to hardware resources and data sources it requires

a) They test for the type of application that can access the model when completed,

b) They review required resources for functionality eg (GPU/CPU resources and memory) and how data would be fed to it

c) They facilitate its accessibility to laptop via API or integrating it to software for end users

4) The last stage after a successful deployment of a model is Model Monitoring.

a) The System analyst team is tasked with ensuring model functions as its expected

b) That supporting software and resources are performing as required

c) That end users are adequately trained to use the model

d) The process of monitoring can then be automated to evaluate its performance

References

[1] David Weedmark. https://www.dominodatalab.com/blog/machine-learning-model-deployment [published 03 November 2021].

[2] Christophergs https://christophergs.com/machine%20learning/how-to-deploy-machine-learning-models/ [published 17 March 2019].

[3] Thuwarakesh Murallie https://towardsdatascience.com/3-ways-to-deploy-machine-learning-models-in-production-cdba15b00e [published 09 Nov 2021].