Governing AI Model Lifecycle with IBM Watson OpenScale and IBM Open Pages
(Co-authored by Manish Bhide and Sampada Basarkar)
One of the big challenges in enterprise adoption of AI is the lack of tools and techniques to track and govern the entire end to end AI model lifecycle. AI Governance is a pretty broad area which covers multiple things such as data and model lineage, policies around model quality and its usage, model version management and tracking, etc. One of the critical things for AI Governance is that of tracking the end to end lifecycle of AI models and storing the associated metadata. In this blog we outline how this can be achieved using IBM Open Pages and IBM Watson OpenScale.
AI Model Lifecycle
A simplistic view of the AI model lifecycle is given in Figure 1.
As shown in the figure, the different steps in the lifecycle are:
- Model Creation Request: The first step is that of generating the model creation request. This outlines the business need for building the model, its constraints, likely usage, downstream impact, etc.
- Model Development: The next step is that of the data scientist building the model.
- Model Validation: Once the model is built, it is validated by a team of model validators.
- Model Approval: The validation report generated by the model validators is then analysed by model approvers who decide if the model should be approved or sent back to the data scientist for additional changes.
- Model Deployment: Once the model has been approved, it is deployed into the production environment.
- Model Monitoring: The production model needs to be continuously monitored for things such as fairness, drift, quality, etc. If any issues are detected in the model, a new version of the model is created and the entire validation/approval process is repeated on the new model version.
- Model Retirement: If the model is no longer required, it is finally retired.
IBM Open Pages provides capability to define custom workflows which can be used to track and govern the different stages of the model lifecycle. Using Open Pages users can create a model creation request. Once the request is created, based on the defined workflow, it will notify the specified data scientists about the request so that they can start building the model. Once the model is built, the data scientist can document the model metadata in Open Pages and mark the model creation step as complete. Open Pages will then notify the model validators to validate the model.
One of the challenges faced by model validators especially in the banking industry is that although they are well conversant with validating statistical models, they do not have the skills needed to validate AI models. This is where IBM Watson OpenScale helps. As mentioned in our previous blog post, OpenScale allows model validators to validate AI models without writing a single line of code! On top of that, OpenScale has deep integration with Open Pages. Hence we can link a model in OpenScale with its counterpart in Open Pages (see Figure 2). Once this link is established, the model validator can push the metrics computed using OpenScale to Open Pages. Additionally the model validator can also push the model validation report generated using OpenScale to Open Pages. Thus these capabilities allow model validators to easily validate an AI model and also push the results of the validation to Open Pages.
Once the model validation is finished, the model validator marks the step as complete in Open Pages. Open Pages then notifies the model approvers to approve/reject the model. The model approver will look at the metrics computed by OpenScale to decide if the model needs to be approved or rejected. As the metrics are available within Open Pages (See Figure 3), the approvers can do their entire work in Open Pages.
After the model has been approved, it is deployed to production. OpenScale again plays a critical role to continuously monitor the model in production. It not only generates alerts whenever any of the metrics exceed their thresholds but also pushes these threshold violations data to Open Pages. One can define workflows in Open Pages to act on these threshold violations such as starting a new workflow which will notify the data scientist to re-evaluate the model and/or build a new version of the model.
Thus IBM Watson OpenScale and Open Pages have a deep integration and provide a comprehensive capability to track and govern the end to end AI Model lifecycle.
