Responsible AI Series Part I: Model Selection & Training
by Manav Mehra, Senior Engineer at Eightfold.ai
As part of our series of blog posts on Responsible Practices in AI, we would like to deep dive on the following aspects for Part Ⅰ,
Model Selection And Training
Eightfold’s web-based platform utilizes an underlying mathematical model built using machine learning techniques. The underlying model predicts the match between a candidate profile and a job position, and displays candidates for a given job position in a rank-list manner as a use case, and supports other use cases such as candidates viewing jobs on career pages. The model operates on a candidate-position pair. It is not a stand-alone score for a candidate. Rather, it is the match of the candidate to job requirements as specified by the calibration of the job position.
At Eightfold, we believe in employing the use of algorithms that support a high degree of explainability. The mindful choice of the algorithms becomes another guard rail towards fairness in the decision making process powered through our AI. Explainable algorithms allow users of the product to understand the reasoning behind algorithms’ scoring.
In addition to the right algorithms, data and features the training process for these models/algorithms act as an added layer of protection against unwanted bias from creeping in during inference. During training, we divide all the data that we have into train and test sets. This distribution is done keeping in mind that there is no data leakage between the two sets, to ensure that the model is tested on samples that it has not seen during training.
We also incorporate early stopping based on classification results made by the model across input datasets of different groups and protected categories. Additionally, keeping up with the recent literature we’re exploring different ways of integrating anti-bias and fairness efforts as part of the loss function on which the model optimizes.
Model Evaluation
Post training, we conduct rigorous evaluations on the model so that it meets our standards before it’s launched to our customers. On passing the initial check, a myriad of metrics are computed to evaluate the model performance and fairness to compare it with the previous iteration of the model. We highlight below certain metrics we use to measure fairness. In addition to the standard accuracy metrics like AUC, Precision, Recall and F1 we compute metrics specifically to measure bias.
Fairness criteria on which these metrics are based upon can be of two types,
- Group Fairness: These metrics essentially compare the outcome of a classification algorithm for two or more groups that are defined on the basis of a protected category.
- Individual Fairness: In this set of metrics we ensure that the outcome of a classification algorithm is the same for two similar inputs. Two inputs are considered to be similar on the basis of a predetermined threshold on a similarity (distance) metric.
We are including a summary of certain commonly used metrics. This is an evolving field, and we continuously aim to improve, evaluate and update our methodology periodically.
Parity-based Metrics
These metrics only take into account predicted positive rates. The metrics computed here are discussed as follows,
We first start the discussion on the metric of Demographic Parity, This metric examines fairness as an equal probability of being classified as a positive. Ideally, each group should have the same probability of being labeled as a positive outcome.
Formula,
Secondly, we discuss the metric of Impact Ratio, similar to parity, except the fact that the ratio is calculated between unprivileged and privileged groups. A model is said to have failed Impact Ratio if it’s value falls below 0 .8 or above 1 .25
Formula,
Another Interpretation discusses Impact Ratio as a ratio of positivity rate of a lesser-represented group to the positivity rate of a more represented group.
These metrics may not take into account potential qualitative differences between the predictions of the groups. More metrics can help in that regard. Algorithmic fairness is an important and heavily researched area. To get a more holistic perspective on fairness, in addition to above, we examine the algorithm’s prediction quality using the following metrics.
Confusion Matrix-based Metrics
These metrics take into account True Negative Rate (TNR), True Positive Rate (TPR), False Negative Rate (FNR), and False Positive Rate (FPR). The advantage of these metrics is that they take into account the underlying qualitative differences between groups that are otherwise not included in the parity-based metrics.
The first metric that we would like to talk about here is the Equality of Opportunity. It’s defined as the probability of a person in a positive class assigned to a positive outcome of the model’s classification. The goal of it is to have very close ratios for all the members of a protected category (such as female and male). In the formula below, s means a particular group; Y is the ground-truth label, and Y_pred is the predicted outcome.
Another metric would be Equalized Odds which is the probability of a person in the positive class being correctly assigned a positive outcome of the model’s classification, and the probability of a person in the negative class being incorrectly assigned a positive outcome of the model’s classification. The goal is to have very close ratios for all the members of a protected category .
Finally we also track whether the ratio of false negatives and false positives is close for all categories in a protected group. In literature, this is called Treatment Equality.
