What? Fairness in regression?
Bringing fairness to regression problems
Why it was a problem?
For classification problems, we have well-defined fairness measures. There is a lot of research and discussion on the topic of bias, bias mitigation, new metrics, when and where to use them, etc. (for example during conferences like https://facctconference.org/). However, the regression field was left without the tools for measuring unfairness for a while. There was a way to measure demographic parity for ML models. The metric checked if R (numerical, continuous output of a model) was similar in all subgroups. This however was not enough. In classification models, we can measure a lot of fairness metrics with the use of a confusion matrix and metrics derived from it. Due to the regression model’s continuous output, we cannot do it like that.
There is some basic knowledge needed to understand some concepts of fairness and plots. To do it check out my earlier blog about introduction to fairness module in dalex.
What is the solution?
The solution is an approximation. As described in the paper Steinberg, D., et al. (2020) we can measure fairness metrics by making classification models! According to Barocas, et al. (2019) most fairness metrics are checking either relaxations or equivalents of three non-discrimination criteria.
The idea is to measure the independence of those variables. It is done by taking particular variables and fitting them to a logistic regression algorithm. If models are good at predicting the A from R and Y there is discrimination occurring. This is only a general idea, therefore for a more detailed description of the algorithm I encourage you to check out Steinberg, D., et al. (2020).
How to do it in python package dalex?
The code here is taken from a more detailed tutorial that can be found here. The reader however should note that this approach and implementation is experimental, and there is still a lot of feedback needed.
We will be using the popular dataset Communities and Crime which we will use for building Decision Tree Regressor. Firstly let’s load and split the data.
Then we make a simple model and make Explainer with the help of dalex.
Last but not least, we create an object which is assigned to fobject and with it, we can plot the fairness_check .
As we can see there is a lot of discrimination within the metrics. But how to interpret this plot? It was nicely described in the previously referenced Steinberg, D., et al. (2020). (The variable labels were adjusted to fit this blog post)
These have an intuitive interpretation; the separation and sufficiency approximations work by determining how much more predictive power the joint distribution of Y and R has in determining A over just considering the marginals, Y or R respectively. Similarly for independence, how much more predictive of A the score is, R, over base rate distribution P(A).
Below we show an example of how does the metrics behave if there is no discrimination in the model towards a particular subgroup.
Key takeouts
As we can see this tool might be really helpful when handling regression models. It is easy to use and does all the work for the user. If you are planning to use it, please note that it is still in the experimental phase. We would love to hear your feedback!
Be sure to check it out. You can install dalex with:
pip install dalex -UIf you want to learn more about fairness I really recommend:
- Tutorial about the usage of dalex fairness module in classification.
- Tutorial about the usage of dalex fairness module in regression.
- Ebook covering various topics of fairness
