Refining Watson AutoAI Output Pipelines
This post is a follow-up of “Building COVID-19 models with Watson AutoAI SDK”.
Training a regression model using Watson AutoAI and deploying that using Watson Machine Learning is covered in “Building COVID-19 models with Watson AutoAI SDK”. However, data science is an iterative process. Users may want to understand and analyze the machine learning pipelines output by the automation. They may want to make a few tweaks to the pipelines and compare the performance. Watson AutoAI gives a lot of options to the user while setting up an experiment, but it also acknowledges the need for further modifications and analysis. This post discusses steps to programmatically refine the output of Watson AutoAI based on user preferences and re-run automation. A self-contained notebook with all these steps is available here.
If you have followed the linked post, you have a regression experiment that has been created using Watson AutoAI. In order to refine the machine learning pipelines associated with that experiment, we would need to (1) retrieve that experiment run from the history of runs, (2) modify one of the output pipelines, (3) re-run optimization to perform algorithm selection and/or hyper-parameter tuning and evaluate. The next section discusses how to retrieve it from the history of runs and access its output pipelines.
Retrieve a previous run of a Watson AutoAI experiment
To programmatically work with AutoAI experiments, we need to install thewatson-machine-learning-client-V4 package. Installation of autoai-libs is needed to be able to train and test locally.
You may need to restart your notebook kernel for the installation to take effect.
Next, define Watson Machine Learning credentials as below. You already have these credentials if you have followed the previous post.
The next step is to create an AutoAI experiment object associated with the WML instance.
We want to access the history of runs associated with this experiment and pick one of the runs to get the pipeline and dataset. The next few lines of code do that:
Here, we retrieved Pipeline_4, which is the pipeline created during the fourth stage of AutoAI. Visualization of this pipeline shows that AutoAI does some preprocessing and also adds new features that are derived from features in the original data. In a notebook, users can hover over each of the nodes in the pipeline to find out the name of the transformer/estimator and its hyperparameters. The names of the nodes are hyperlinks to the documentation. For example, hovering over the first feature engineering transformer TA1 shows that it applied a square function to the column Elapsed_days :
and clicking on TA1 takes us to its documentation.
We can also access the training as well as holdout test datasets associated with that AutoAI run as follows:
We can re-create the original dataset from these splits so that it can be used for the refinement steps.
The dataset was sorted by Elapsed_days because we are going to split it into a train and holdout test to make sure the test split follows the train split in terms of time. We create those splits in the next cell as follows:
Refine the retrieved pipeline
We now have a pipeline that was created by AutoAI and the dataset used. The pipeline by default is a Lale pipeline, which is a scikit-learn like pipeline with some enhancements for visualization and support for automation.
For demonstration of refinement, we consider a scenario as follows: The pipeline created by AutoAI was meant to perform well on the given dataset. Out of all possible regressor algorithms it considered, AutoAI chose ExtraTreesRegressor since the performance of such a pipeline was the best (r_squared was used as performance metric for this experiment). Say the user wants to check how an interpretable regression technique would perform compared to ExtraTreesRegressor .
We will use Lale for pipeline manipulation and Combined Algorithm Selection and Hyper-parameter tuning (CASH) for this experiment.
In the next steps, we modify the pipeline to remove the regressor. Preprocessing and feature engineering are important aspects of AutoAI output, so while it is possible to remove those, we chose to keep them.
We can then create a new pipeline which keeps a choice of regressors limited to DecisionTreeRegressor and LinearRegression which are easy to understand models. We use the algorithmic choice combinator | from Lale to indicate that. This is also manifested in the visualization.
Re-run optimization
To automatically select the regressor and tune its hyperparameters, we
create an instance of Hyperopt from Lale which internally uses hyperopt as an optimizer. During optimization, we use sklearn.model_selection.TimeSeriesSplit for obtaining cross validation splits. More options to be used with Hyperopt are documented here. After running the optimization for 50 trials, we can see that the best pipeline found has LinearRegression as regressor. Please note that unlike scikit-learn, Lale does not mutate objects, so it is important to assign the output of fit to a new object.
We can call predict on the pipeline found above to get predictions on the holdout test dataset and calculate the performance metric. The plot below compares these new predictions with ground truth as well as predictions from the original pipeline.
It appears that prediction quality has decreased a bit since we chose to use a simpler model compared to an ensemble. However, it may be the cost to pay for interpretability and the choice is up to the user. Such refinement can be an iterative process where users may choose to do more analysis and comparison. Once the new pipeline satisfies performance and any other evaluation criteria, it can be exported as a scikit-learn pipeline by calling export_to_sklearn_pipelineon the pipeline object and deployed to WML as a webservice. Instructions for such a deployment are included in the previous post.