New Way to Measure Crowdsourcing Bias in Machine Learning
An overview of how to use counterfactual fairness to quantify the social bias of crowd workers
Crowdsourcing is widely used in machine learning as an efficient form of annotating datasets. Platforms like Amazon Mechanical Turk allow researchers to collect data or outsource the task of labelling training…