AI Model Validation without writing a single line of code!
(Co-authored by Manish Bhide and Ravi Chamarthy)
Model development using automated tools is a well-known concept. But is it really possible to validate an AI model without writing a single line of code? The answer is Yes! IBM Watson OpenScale provides Model Risk Management capability which allows model validators to validate AI models without writing any code. In this blog post, we provide an overview of this capability and how enterprises can leverage this to speed up their adoption of AI.
Challenges with AI Model Validation
Ask any software exec what processes they have in place before any software is deployed to their production environment and they would tell you about their time tested and robust methodology. If you were to flip the question and ask what process they employ for validating AI models before they are deployed to production, chances are that the answer will be far less convincing.
There are many reasons for this. One of the big challenges in AI Model validation is that just like model development, model validation is complex. It involves multiple things such as checking for bias, understanding model quality (precision, recall, false positive rate, RMSE, etc.), understanding model drift, comparing with a challenger model, etc. Such validation requires specialised resources who can understand and apply these concepts — resources which, unfortunately, are hard to find.
The second problem with AI model validation is the lack of a standardised way to validate and document the results of model validation. Different validators might use different techniques to validate and document the results of model validation. This leads to a lack of uniformity across the enterprise.
Model Risk Management in OpenScale
If a model validator has to validate an AI model, he/she needs to understand how to score the model, how to understand and interpret the output to figure out if the model is exhibiting bias or drift and find out about the model quality. The Model Risk Management capability in OpenScale allows model validator to validate an AI model by simply uploading some test data. OpenScale takes care of performing different kinds of tests for bias, quality, drift as well as model explainability. Thus the model validator can validate the AI model without writing a single line of code!
Let us now get into the details of how things work in Model Risk Management. AI model validation is typically done in a pre-production environment. One of the first capabilities that OpenScale provides is a way to tag a model serve environment as either production or pre-production.
When an environment is tagged as a pre-production environment OpenScale will assume that models deployed in this environment would be validated by a model validator. For such models, OpenScale provides an easy to use GUI to validate these models. Once the deployment has been configured, the model validator can simply upload some test data. OpenScale will then validate the model along different dimensions such as fairness, drift, quality as well as explainability. The model validator is provided a summarised dashboard (see Figure 2) which provides an easy to understand summary of the tests that were run and how many of them passed.
The validator can also choose to download the results as a PDF document so that they have a record of all the tests that were done and its results.
One very important capability that is provided by OpenScale is that of comparing the model under validation with a challenger model (see Figure 3). The challenger model could be a model which is already deployed in production or could be a model which the model validator builds using tools such as Auto AI. This provides confidence to the model validator that the model under validation is good/better than the one deployed in production.
Thus the use of Model Risk Management allows model validators to validate AI models without writing a single line of code. It also brings in uniformity across the validations done by all the validators thereby ensuring that the right standards are followed while validating AI models. Finally, the documentation capability of OpenScale helps validators to store the results of their validation.
