Continuous AI: Maintain Value Creation of AI Models
Life is a series of repairs. Every day, something new is broken or not functioning as it is supposed to, requiring attention. Whether it is a health issue or a problem at work, we seek ways to maintain or improve our living standards. But how about Artificial Intelligence (AI) models? Do they need attention as well? The answer is yes. This blog post will focus on the Continuous AI concept, which includes the maintenance of the AI models during the production phase.
Why do AI models need maintenance?
AI models learn from their mistakes in the past using their training data just like children. As the models grow under their parents’ supervision, they start to make fewer mistakes. Once a model reaches maturity, it is introduced to the real world to predict the best solutions for the problems it encounters in the future. Typically, models tend to perform similarly during the initial deployment phase as they did with the test data. However, as the models get older, accuracy results start to decline due to various reasons. When that is the case, AI models need the support of developers, just like a child needs the support of her/his parents.
Performance loss in AI models could be caused by many different reasons. These problems can be grouped under three categories:
- Concept Drift, changes in the characteristics of the dependent variable
- Data Drift, changes in the characteristics of independent variables.
- Algorithm Drift, changes in business needs, algorithms are not aligned with the business requirements anymore.
In this post, we do not go into the details of drifting. However, we highly recommend curious readers to check The Drifting Effects on AI Models: Why Continuous AI is a Must? to further explore drifting effects on AI models.
How to detect drifting in the model?
The first step of the Continuous AI is detecting the anomalies in the model before they lead to more significant issues. The general approach to detect drifting in the model is monitoring the KPIs or metrics, such as mean square error, mean absolute error, F1-Score.
Data quality issues are among the most common problems in AI models. The fundamental principle is “Garbage in, garbage out”. Before going into details of the model’s independent variables, we have to ensure that there is no problem in the data flow by integrating KPIs on data quality.
Besides monitoring, creating an intelligent alarm system is essential to notify the user about the source of the problem.
The model is underperforming. Now what?
The drifting concept is a real challenge when dealing with an AI model. At that point, the model needs a human intervention to stay accurate and reliable. Most of the time, models need to be retrained to overcome such drifting issues.
The straightforward approach is, retraining the model with both drifted and non-drifted data. This method would include the effects of the past data and new information coming from the drifted data. However, this might not be enough for drastic data drifts. In that case, it would be wise to assign higher weights to new data so that the model would prioritize the drifted information.
In a dynamic business environment where customer patterns change periodically, regular retraining of the model might be necessary to adapt to the evolving datasets. The model should be retrained systematically (usually monthly or quarterly) to keep the accuracy at the same level. The retraining frequency depends on the business sector and the business needs.
Continuous AI is a necessary step in every AI models’ lifecycle. Most AI models become outdated and eventually obsolete due to the lack of maintenance. It is essential to grease the model to get the most out of AI.
Berk Baris | Senior Analytics Consultant, KoçDigital
