Keeping Your Machine Learning Models Up-To-Date
Continuous learning with IBM Watson Machine Learning (part 1)
If you scour the web you will find many tutorials that show you how to train and score machine learning models. You may even find tutorials that teach you how to deploy those models and run predictions from your end-user applications.
It’s more difficult, however, to find resources that show you how to maintain machine learning systems. How do you ensure your predictions continue to be accurate? How do you keep your models up-to-date with new training data?
Shameless plug! You can read my previous tutorials on training and scoring a simple ML model with the IBM Watson Data Platform here:
As I have stated in previous posts, the data is the most crucial component of a successful ML system. If you’ve found a data set that provides you with accurate predictions that’s a great start, but how long will that data continue to provide accurate predictions?
Let’s take the example of predicting house prices. House prices change all the time. The data you used to train a machine learning model that predicts house prices six months ago could provide terrible predictions today. For house prices, it’s imperative that you have up-to-date information to train your models.
When designing a machine learning system it is important to understand how your data is going to change over time. A well-architected system should take this into account, and a plan should be put in place for keeping your models updated.
Manual retraining
One way to maintain models with fresh data is to train and deploy your models using the same process you used to build your models in the first place. As you can imagine this process can be time-consuming. How often do you retrain your models? Weekly? Daily?
On the other hand, as you are manually retraining your models you may discover a new algorithm or a different set of features that provide improved accuracy.
Continuous learning
Another way to keep your models up-to-date is to have an automated system to continuously evaluate and retrain your models. This type of system is often referred to as continuous learning, and may look something like this:
- Save new training data as you receive it. For example, if you are receiving updated prices of houses on the market, save that information to a database.
- When you have enough new data, test its accuracy against your machine learning model.
- If you see the accuracy of your model degrading over time, use the new data, or a combination of the new data and old training data to build and deploy a new model.
The benefit to a continuous learning system is that it can be completely automated.
It’s probably a still a good idea to review your process on a regular basis. As I mentioned before, you may find a different algorithm or a new set of features that improves your predictions, and this isn’t necessarily something a continuous learning system is good at.
Continuous learning with Watson ML
If you are using the IBM Watson Data Platform or the Watson Machine Learning service, you will be happy to know that there is built-in support for continuous learning.
You can use the Model Builder to train your model and enable continuous learning without writing a single line of code, but if you want more control over the life cycle of your machine learning models (like I do) you can use the Watson Machine Learning libraries and APIs.
To run continuous learning with Watson ML you will need to have instances of the following services:
- Watson Machine Learning manages your models and the continuous learning process.
- Db2 Warehouse on Cloud stores your training and feedback data.
- Apache Spark is used by Watson ML to test and train your models.
The following diagram shows how the continuous learning process works when you build your models in Jupyter notebooks:
Let’s break this down:
1. You start by storing your training data in a table in Db2 Warehouse on Cloud. When you are ready to train your model, pull your training data into a Jupyter Notebook.
2. In your notebook, build your model using Spark ML. Spark ML is currently the only machine learning library that Watson ML supports for continuous learning.
3. Deploy your model to Watson ML. When deploying your model tell Watson ML where to find the training data (a table in the data warehouse) as well as where it will find feedback data (another table in the warehouse) later for evaluation.
It’s also important to tell Watson ML how to measure model accuracy and what value represents the current model’s accuracy.
Finally, instruct Watson ML at what accuracy threshold a model should be retrained. This means if your model is performing below a certain threshold (i.e., it is not performing well) then Watson ML will retrain your model and determine if it can improve predictions.
4. For continuous learning to be effective you need to have some type of automated process for consuming new data. This could be a REST API, a script that downloads data nightly, or any other process that gathers new data. This is referred to as feedback data. When new feedback data is received, send it to Watson ML, and Watson ML will store it in the warehouse.
5. When you determine you have amassed enough feedback data to test, instruct Watson ML, via a REST API, to start feedback evaluation.
6. Watson ML pulls any new feedback data and runs predictions against the current model. (Watson ML will not include previously evaluated feedback data.)
7. After feedback evaluation completes, the accuracy of the data is measured against the accuracy threshold. If the accuracy is below that threshold then retraining is triggered.
8. Watson ML then pulls in all training data and all feedback data to build a new model and measures its accuracy.
9. If the new model’s accuracy exceeds the original model’s accuracy, then the new model is automatically deployed.
That’s the typical flow one might follow with the Watson ML continuous learning system. As you design your own continuous learning system, you will find that many of these steps are configurable, and enable you to implement variations of this flow. For example, you can choose to redeploy your model every time, whether or not accuracy is improved. Alternatively, you could choose to never automatically redeploy your model.
The learnings continue
Stay tuned for an upcoming blog post where I will walk through an end-to-end continuous learning example. We’ll start with a notebook where we take a data set, clean it, and then upload it to the warehouse. In a second notebook, we’ll pull the data from the warehouse, train our model, and deploy it to Watson ML. Finally, we’ll upload feedback data to the warehouse, kick off feedback evaluation, and watch the continuous learning process unfold.
