Bias Detection in IBM Watson OpenScale
Fairness or Bias is one of the big issues that plagues the adoption of AI in enterprises. Consider the case of an insurance company which is planning to use an AI model to make a decision on whether an insurance claim should be approved or denied. LOB owners need to be absolutely sure that the model will not make biased decisions. Even if a model is known to act in an unbiased manner during training, it does not necessarily mean that it will continue to exhibit similar behaviour once it is deployed to production. IBM Watson OpenScale helps address this problem by monitoring the behaviour of AI Models in production and checking if they are acting in a biased manner.
What is Bias?
Before we understand what is meant by bias, let us clarify a few key concepts.
Fairness Attribute: Bias or Fairness is typically measured using some Fairness attribute such as Gender, Ethnicity, Age, etc. It can also be measured using non-societal attributes such as Policy Age (to ensure that the model is not biased against new customers), etc.
Monitored/Reference Group: Monitored group are those values of Fairness attribute for which we want to measure bias. The rest of the values of the fairness attributes are called as reference group. In case of Fairness Attribute=Gender, if we are trying to measure bias against females, then Monitored group is “Female” and Reference group is “Male”.
Favourable/Unfavourable outcome: An important concept in bias detection is that of favourable and unfavourable outcome of the model. E.g., Claim approved can be considered as a favourable outcome and Claim denied can be considered as an unfavourable outcome.
Disparate Impact: This is used to measure bias and is computed as the ratio of percentage of favourable outcome for the monitored group to the percentage of favourable outcome for the reference group. Bias is said to exist if the disparate impact value is below some threshold.
E.g., if 80% of claims made by males are approved whereas only 60% of claims made by females are approved, then the disparate impact will be: 60/80 = 0.75. Typically the threshold value for bias is 0.8 and as the disparate impact ratio is less than 0.8, the model is said to be biased.
Bias Threshold: If the bias threshold is 1, then it would mean that we expect Females to get the same or better claim approval rates as compared to males. However, in some scenarios we might be okay if the monitored group gets a little bit lesser favourable outcome as compared to the reference group. In order to handle this, the customer can set the threshold value for each model to a value less than 1. Typically the value is set to 0.8.
Bias Detection in Watson OpenScale
IBM Watson OpenScale helps in detection of Bias at run time. It monitors the data which has been sent to the model as well as the model prediction (this data is called as Payload data). It then identifies bias issues and notifies them to the user. The dashboard of bias looks like follows:
The fairness attribute in the above example is Age and it shows that the model is acting in a biased manner against people in the age group 18–24 (monitored group). The percentage of the favourable outcome for people in age 18–24 is 52% whereas the percentage of favourable outcome for people in the age 25–75 (reference group) is 70%. Thus the disparate impact ratio is 0.52/0.7=0.74 which is below the threshold of 0.8 and hence the model is flagged as acting in a biased manner.
Data Perturbation
Consider a scenario where the payload data consists of 100 insurance claims (i.e., the model has been used to make decisions for 100 claims). In these claims, lets say 60 claims were made by people in the age group 18–24 and 40 claims were made by people in the age group > 24. All the people in the age group 18–24 had very high claim frequency and were know to have made fraudulent claims in the past. Hence the model rejected all the 60 claims made by customers belonging to the age group 18–24. On the other hand, the 40 claims made by people with age > 24 were all genuine claims and hence were approved by the model. If we measure the disparate impact for this it would be 0/1 = 0. Hence we would flag the model behaviour as being biased. However, the model is not really acting in a biased manner as it is making an accurate decision based on the data provided to it. Thus if we report such a model as being biased, it is not something that the business owner would want to fix as they would obviously not want to approve a fraudulent claim just be fair.
In order to address this issue, we do data perturbation while computing fairness. The way data perturbation works is that if there is a claim by a person with say age 20 and it was rejected by the model, we perturb the record by flipping the age from 20 to a random value in the reference group (say 40) and then send this perturbed record to the model. All the other features of the record are kept the same. Now if the model predicts that the claim is approved, then it means that it is acting in a biased manner by making decisions just based on the age of the customer. Thus while computing bias, we do this data perturbation and compute the disparate impact using the payload + the perturbed data. This will ensure that we report bias only if there are sufficient number of data points where the model has exhibited biased behaviour across the original and the perturbed data. Thus we are able to detect if there is genuine bias in the model and not get impacted by the kind of data being received by the model.
In the screenshot mentioned above, the bar chart shows the statistics across the payload + the perturbed data. The percentage of favourable outcome (52%, 70%, etc.) is also computed using combination of payload + perturbed data.
Summary
IBM Watson OpenScale provides a mechanism for enterprises to monitor if their models are acting in a biased manner at runtime. It computes bias in such a manner that it will not lead to false positives. In other words, if IBM Watson OpenScale reports a bias, it will be something that enterprises would want to fix and it will be practical for them to do so. This is a key ingredient to help build trust in AI.
