Signal 5: What is That “Black Box” Algorithm Thinking

Expanding on Signal 4 regarding risk assessment tools — what are ways we can understand and audit “black box” algorithms? This is critically important with increasing reliance on AI and unconsciously baking our own implicit biases into these tools which could have huge impacts on people’s lives.

S. Tan et.al. (2017) proposed using a “Distill-and-Compare” approach to “look” inside these black-boxes — meaning we treat the opaque models as teachers training student models which are transparent to mimic the same results. Then compare the student model results to a another transparent model trained on ground-truth outcomes. Essentially, we want to know if the risk scores and the ground-truth outcomes align/closely-relate. If the answer is yes, then the black-box model is doing its job. If not, then using these transparent models could help elucidate if features which were excluded from the model such as race, in reality, had been influencing the outcomes unintentionally because there was bias in the training data-sets.

Understanding these approaches to auditing is key to our training as data scientists; it’s a tool in our toolbox. With increasing reliance on AI, one innovation would be to scale up these types of audits and being able to execute audits in a more timely manner.

Citation:

Snow, J. (2018). We are starting to peer inside “black box” AI algorithms. [online] MIT Technology Review. Available at: https://www.technologyreview.com/s/609338/new-research-aims-to-solve-the-problem-of-ai-bias-in-black-box-algorithms/ [Accessed 13 Nov. 2018].

S. Tan et.al. 2017. Detecting Bias in Black-box Models Using Transparent Model Distillation. In ArXiv Report. https://arxiv.org/abs/1710.06169