Privacy vs. Protected Attributes: The Interconnected Opposite Forces that Underpin Fairness
In this blog, Dr Tao Zhang, a Data Scientist in Artificial Intelligence & Platform Services (AIPS) at SEEK, Melbourne, writes about the crucial role of protected attributes in fair machine learning and the dilemma between privacy and fairness in utilising protected attributes.
1. Introduction
Machine learning (ML) is now in wide use as a decision-making tool in many areas, such as job employment, risk assessment and many other essential precursors to equity. However, the popularity of ML has raised concerns about whether the decisions algorithms make are fair to all individuals, and many facts show that ML could lead to discriminatory model outputs [1]. Recently, fairness in ML is an established field of ML that studies how to evaluate and remove bias in data and algorithms and not lead to models that disadvantage individuals based on protected attributes. Protected attributes are personal attributes protected from discrimination by law, such as age, sex, gender identity, race, disability, etc.
In practice, protected attributes are mostly private information and, therefore, unavailable, which challenges the use of protected attributes for fair purposes.
Most research assumes that the protected attributes are always available when building fairness-aware models and evaluating model fairness. However, this assumption sometimes does not hold in practice. Protected attributes might be unavailable for many reasons — users may choose to withhold this information (for privacy or other personal preferences), or the dataset may have been constructed for use in a setting where collecting the sensitive information of the dataset’s subjects was unnecessary, undesirable, or even illegal.
For example, in credit decisions in the financial services industry, laws and policies define protected categories, such as race and gender, and prohibit discrimination based on a customer’s membership in those protected attributes [6]. However, financial service providers are not allowed to ask applicants what race they are when they apply for credit.
In this blog, we first explain the key role protected attributes play in fair ML and then discuss the dilemma and solutions when protected attributes are unavailable.
1.1 When to use protected attributes
Firstly, let’s consider where protected attributes are allowed to be used in an ML task. Usually, there are three datasets in an ML task — training set, validation set and test set, which correspond to the training phase, validation phase and test phase.
Protected attributes cannot be utilised as features in the training set.
This is because the model may learn correlations between protected attributes and labels, and we don’t want a model to make decisions based on protected attributes. Research showed that even if the protected attributes are not used for model training, the training results may still be biased. This is because sometimes it is impossible to hide protected attributes in real-world scenarios to make decisions because non-protected attributes may have correlations with protected attributes. For example, zip code strongly correlates with the protected attribute, race. ML communities are aware of this problem and have proposed research to build fair ML models.
Protected attributes can be used during the training phase (not utilised as features) to guide or assist the training towards a fairer direction. In the validation phase, protected attributes might be used to select hyperparameters to help achieve the ideal balance between accuracy and fairness. In the testing phase, we make use of protected attributes to check whether model outputs are fair. In the following, we will discuss how to use protected attributes in building fair models and evaluating model fairness.
1.2 How to use protected attributes
1.2.1 Building fair models
Protected attributes are significant information when building fairness-aware models. Methods for fair supervised learning mainly include pre-processing, in-processing and post-processing methods, and all of these methods need protected attributes.
Pre-processing methods, such as Reweighing and Uniform Sampling, need information on protected attributes [2]. For example, in Reweighing, different weights will be attached to data examples in the protected and unprotected groups. In-processing methods, such as fairness constraints [3] and fairness regularisers [4], require information on protected attributes when building fair models. For example, fairness constraints are built based on the covariance between the users’ protected attributes and the signed distance from the users’ feature vectors to the decision boundary [3]. Post-processing methods [5] also need protected attributes to adjust model outputs in the protected and unprotected groups to meet the fairness metrics.
During the validation, protected attributes might be used to select hyperparameters to help enhance the trade-off between accuracy and fairness. For example, early stopping is an optimisation technique used to reduce overfitting without compromising model accuracy. When considering model fairness, we want to stop training based on early stopping criteria, not only considering model accuracy, AUC, etc. but also fairness metrics. The training epoch selected in this way helps to achieve a better trade-off between accuracy and fairness.
Note that when building fairness-aware models, protected attributes are usually allowed to be used because the intent of using protected attributes is to reduce discrimination. Most fair ML research uses protected attributes to reduce bias [1].
In practice, protected attributes help improve fairness. For example, affirmative action establishes fair access to employment opportunities to create a workforce that accurately reflects the demographics of the qualified available workforce in the relevant job market. To achieve fairness, affirmative action needs demographic information (similar to protected attribution information).
Another example is that some projects, such as Women in Engineering or Women in STEM, make use of gender information to reduce gender bias in engineering and STEM areas. This is a very similar situation when we use the information of protected attributes for ML.
As far as we know, there are no clear policies or guidelines to explain the use of protected attributes in building fair models. We are calling for more work on ensuring the correct use of protected attributes in ML.
1.2.2 Assessing model fairness
Protected attributes are also crucial information in fairness evaluation. This is because protected attributes are the required information when calculating most fairness metrics, such as demographic parity, equal opportunity, and counterfactual fairness metrics.
We need the information on protected attributes to see if the model outputs satisfy a specific fairness metric. For example, demographic parity requires an equal proportion of positive predictions in each group. Therefore, protected attributes are needed to know the group information (an individual is in a protected or unprotected group). However, some fairness metrics, such as individual fairness metrics, do not require information on protected attributes because individual fairness metrics are calculated based on similarity. So far, the implementation of individual fairness is rarely in practice due to the challenging definition of similarity between individuals.
In summary, protected attributes are necessary information in building fair models as well as evaluating model fairness. However, protected attributes are usually unavailable due to privacy concerns, thus the challenge would be how to ensure fair ML without protected attributes.
In the following, we will discuss one of the promising solutions to this challenge — proxy methods. We’ll analyse proxy methods from legal, ethical and technical perspectives.
2. Proxy Methods
A few papers have worked on how to build and evaluate model fairness without protected attributes.
We can classify them into two categories:
- use proxy attributes to infer protected attributes [7, 8, 9, 10];
- use multi-party computation methods to access protected attributes [11, 12, 13].
Overall, proxy methods might involve ethical and legal risks. It is crucial to formulate more comprehensive and clear regulations for the collection and use of (inferred) protected attributes in practice. Multi-party computation methods need protected attributes in scale. Sometimes, it is easier, more ethical and maybe lawful to ask users for additional information for research purposes.
In this article, we focus on interring methods, and we will discuss the second method in the future.
In proxy methods, we usually have two datasets, D1 and D2, where D1 has attributes X and labels Y, and D2 has attributes X and protected attributes A (as shown in Figure 2). Proxy attributes are the attributes relevant to the protected attributes — therefore, proxy attributes can be used to infer protected attributes.
A common approach is to train a protected attribute classifier f2 on a separate dataset D2 to infer protected attributes, and then use it in an ML pipeline to evaluate the bias of a given model f1. Proxy methods have been implemented in some industries in the USA.
For instance, in a credit decision task, proxy attributes are used to train a gender classifier. Gender information is then used to evaluate the bias of a classifier that decides if the credit decision is yes or no.
Another example is that using proxies for race or ethnicity is standard practice in finance in which protected attributes are often unavailable, but fairness is a concern. To assess racial discrimination in credit decisions, regulators such as the Consumer Financial Protection Bureau use proxy methods to impute unknown racial labels of customers [14]. They use the customer’s surname and address of residence as proxy attributes to infer race [7].
The advantage of this approach is that it decouples the data requirement, and in particular, the attribute classifier can be trained on a separate dataset without any label information. On the other hand, such an approach may produce poor estimates. A few things that we can consider to produce an accurate bias estimation with proxy methods.
2.1 What are we allowed to do?
As far as we know, there are no clear regulations on whether we can use, or how to use, inferred protected attributes. For example, GDPR is a comprehensive regulation to protect personal data and privacy, but inferred protected attributes appear to receive the least protection of the personal data types prescribed by GDPR.
It is unclear that if inferred personal attributes are considered personal data, should the data protection rights enshrined in GDPR also equally apply? Inferred protected attributes may pose significant risks regarding privacy and discrimination.
This is because inferred protected attributes cannot be verified at the time of decision-making, and therefore data subjects are often unable to understand and examine these inferences. It seems reasonable to suggest that, data owners should be informed at the data collection point that protected attributes in the data might be inferred for the purpose of ensuring fairness in ML models, and a strict validation process is necessary to justify the use and the consequence of the inferred protected attribute.
More legal consideration should be given to address the dilemma of inferring protected attributes in practice for the fairness intent.
2.2 What should we do?
In some situations, industries are not allowed to collect information on protected attributes because of privacy and fairness concerns. Proxy methods don’t seem to address these concerns. Regarding fairness, the Consumer Financial Protection Bureau has published assessment methodologies that describe the use of proxy models to impute race labels [14].
However, the result overestimates the disparate impact and has sparked some controversy. Regarding privacy, some protected attributes are also private information, such as sexual orientation. Customers may not feel comfortable when their private information is inferred via proxy methods.
Regarding transparency, data owners should be informed at the data collection point that protected attributes in data might be inferred for the purpose of ensuring fairness in ML models.
Regarding accountability, proxy methods might not accurately estimate bias across groups, which can mislead the use of protected attributes in bias removal methods or bias evaluation.
So far, how to use proxy methods in practice is still in progress and needs more consideration in fairness, privacy, accountability and transparency from ML communities.
2.3 What can we do?
Some technical issues may occur when using proxy methods, and considering data distributions, error rates across groups, and the threshold will contribute to a more accurate bias estimation.
2.3.1 Data distribution
Data distributions in D1 and D2 will decide how well the proxy methods can perform. If the two datasets are generated from dissimilar distributions, the correlations between X and Y (and X and A) may be arbitrarily different for the two distributions.
For example, we have a task to make credit decisions with D1 collecting from Singapore and D2 collecting from Malaysia. If the attribute classifier is trained on D2 to predict gender, this attribute classifier probably will not work well to predict gender in D1. This is because the individuals’ attributes in Singapore are different from the individual’s attributes in Malaysia. Instead, it would be better to use a gender classifier trained on Malaysian data since Malaysia and Indonesia have more similarities.
2.3.2 Error rates across groups
The test accuracy of an attribute classifier is not always related to its effectiveness in estimating the bias of downstream models. Research [9, 10] found that using the Bayes optimal attribute classifier does not lead to the most accurate bias estimation.
The performance of a proxy attribute classifier depends not only on the overall errors of the classifier but also on how these errors are distributed. Therefore, when training a protected attribute classifier, at least two metrics are important: overall accuracy and error rates across groups.
2.3.3 The choice of the threshold
Work in [15] expresses caution when choosing thresholds in practice. The threshold is the variable that classifies individuals into a protected group or an unprotected group.
Take gender as an example. When the threshold is set to be 0.5, then a gender-proxy model output over 0.5 is considered a female and below 0.5 is considered a male. Obviously, different thresholds will produce different bias estimations.
Such hard classification rules inevitably misclassify some people, given the inherent uncertainty of the classification of protected classes. Empirical evidence shows that the bias of threshold-based imputation is generally upward. Therefore, the value of the threshold should be carefully chosen. The threshold is a hyperparameter that can be fine-tuned via a validation set. Chen also et al. proposed a weighted estimator to propagate the uncertainty resulting from the probabilistic proxy onto the final estimation [15].
3 Summary
Protected attributes are key information for learning fair models or checking whether a given model is fair. When protected attributes are unavailable and inaccessible, proxy methods can help infer. Inferring should be processed carefully, as inferring protected attributes might be subject to technical, ethical and legal issues. As discussed, there is a tension between fairness and privacy when using protected attributes in practice, and we are looking for more research on fairness and privacy, law and policy, to address this tension.
References
[1] Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR), 54(6):1–35, 2021.
[2] Faisal Kamiran and Toon Calders. Data preprocessing techniques for classification without discrimination. Knowledge and information systems, 33(1):1–33, 2012.
[3] Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rogriguez, and Krishna P Gummadi. Fairness constraints: Mechanisms for fair classification. In Artificial intelligence and statistics, pages 962–970. PMLR, 2017.
[4] Toshihiro Kamishima, Shotaro Akaho, Hideki Asoh, and Jun Sakuma. Fairness-aware classifier with prejudice remover regulariser. In Joint European conference on machine learning and knowledge discovery in databases, pages 35–50. Springer, 2012.
[5] Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. Advances in neural information processing systems, 29, 2016.
[6] Jiahao Chen. Fair lending needs explainable models for responsible recommendation. arXiv preprint arXiv:1809.04684, 2018.
[7] Marc N Elliott, Peter A Morrison, Allen Fremont, Daniel F McCaffrey, Philip Pantoja, and Nicole Lurie. Using the census bureau’s surname list to improve estimates of race/ethnicity and associated disparities. Health Services and Outcomes Research Methodology, 9(2):69–83, 2009.
[8] Yan Zhang. Assessing fair lending risks using race/ethnicity proxies. Management Science, 64(1):178–197, 2018.
[9] Pranjal Awasthi, Alex Beutel, Matthäus Kleindessner, Jamie Morgenstern, and Xuezhi Wang. Evaluating fairness of machine learning models under uncertain and incomplete information. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 206–214, 2021.
[10] Vincent Grari, Sylvain Lamprier, and Marcin Detyniecki. Fairness without the sensitive attribute via causal variational autoencoder. arXiv preprint arXiv:2109.04999, 2021.
[11] Hui Hu, Yijun Liu, Zhen Wang, and Chao Lan. A distributed fair machine learning framework with private demographic data protection. In 2019 IEEE International Conference on Data Mining (ICDM), pages 1102–1107. IEEE, 2019.
[12] Niki Kilbertus, Adrià Gascón, Matt Kusner, Michael Veale, Krishna Gummadi, and Adrian Weller. Blind justice: Fairness with encrypted sensitive attributes. In International Conference on Machine Learning, pages 2630–2639. PMLR, 2018.
[13] Michael Veale and Reuben Binns. Fairer machine learning in the real world: Mitigating discrimination without collecting sensitive data. Big Data & Society, 4(2):2053951717743530, 2017.
[14] Using publicly available information to proxy for unidentified race and ethnicity. https://www.consumerfinance.gov/data-research/research-reports/using-publicly-available-information-to-proxy-for-unidentified-race-and-ethnicity/.
[15] Jiahao Chen, Nathan Kallus, Xiaojie Mao, Geoffry Svacha, and Madeleine Udell. Fairness under unawareness: Assessing disparity when protected class is unobserved. In Proceedings of the conference on fairness, accountability, and transparency, pages 339–348, 2019.
