Model Improvement in Production

In this article, we explore the steps that one should follow for monitoring and continues improvement of the model

We expect models to degrade over time. This phenomenon is called model drift. Model drift happens due to following reason

• We validated on limited data. Real data might be different. In B2B, performance in UAT environment will be differ from performance in production.
• Structure of the Input might change. For example, values of feature Team changed from Sales, Presales etc to, Sales_Asia, Sales_Europe, Presales_Asia etc.
• Just like the people’s interest, the nature of the data and their relationship changes over a period
  - Concept Drift: Concept or pattern that the model have learnt has become irrelevant. Example: For Assignment team classification, location was an important feature. Later, the team decided that the Assignment team should be based on the category of problem, not location. This results in wrong output.
  - Data Drift: Data and properties might change. Example: Canned responses might be added and deleted. Some class might be used more or less compared to validation data.

The following steps will help us in improving the model in production.

Capture the ground truth

• Create a system to capture ground truth. In CRS and FC cases, user will explicitly fill in the correct output.
• Just like testing team, have a separate annotation team that would annotate the actual values. Ground truth of SA can be captured this way
• For the use case like CCP, we have to wait for alteast few weeks to get the actual truth

Ask the Customer for feedback

• Is the Client using the feature
  - Sometimes, some setup might be required to show the prediction. In just cases, send a group mail with instruction or even demo would make it faster
  - Some clients might not be interested. One of the highly profile customers was not ready to use FC. Reason was that they felt that it might reduce the workload and might result in job loss. Another startup was happy to use it, since they are high load and it was pretty useful for them.
• Address the Concern
  - Expectations might be high. It might be mainly because it was oversold by the PreSales team :). In CRS cases, the model should automatically predict the new canned response. We suggested that we require 50 examples and at least two weeks to rebuild the model.
  - it is better to capture before the customer complaints. Here is where the implementation team comes to resume.

Give powers to the Implementation Team:

• Implementation team is the team that is closely working with the client. See the implementation as the hero and data scientist team as the team which builds cool gadgets for the hero. If it is the first client, data scientist has to do implementation work at least for few months. Later it can be taken by the implementation team. Supporting tools is required to build for the implementation team to purse.
  - Monitoring tools : Tools for performing accuracy, precision etc
  - Data Monitoring tools: Tools for monitoring class and feature distribution
  - Alert System: Email Alert system if the prediction is poor and distribution is changed drastically
  - Model Rebuilding Tool: Tool to Retrain the model with additional data. If the new model does not return the expected result. Data scientist support is required.
  - Data Scientist Analysis: Data scientists have to see where the problem is. Problems might be new features, might need to be included or features need to be normalized, etc. This requires extensive work similar to previous articles

This is the high level overview of Model improvement in production. Next article talks about the Deploying ML Pipelines. Please click here for next article

Please click here to go to the first article

Mlearning.ai Submission Suggestions