Agata Foryciarz

…tisfied, the business in question will have quite a lot of documentation for each model they build. For any model in production, companies will need to know and explain (among other things) who trained it, using which methodology (and why this method was chosen over others), where the data came from, how it was balanced for demographic fairness, how the outcomes are tracked to ward against discriminatory practices, what the accuracy metrics were, how those are updated over time, how the testing and validation took place, how the model’s security and robustness are addressed (including its resistance to adversarial attacks), and to provide an impact explanation that seems to require enterprises to study the broader societal impact of the use of their model in practical terms. And, for each explanation type, the guidelines also stipulate that companies must document their ap…

This is great news for the public, because the spirit of the right to explanation is to allow consumers to challenge decisions that directly impact their lives. Ask anyone who’s trained a model using modern AI techniques and they’ll tell you that weird things …

Given the complexity of ML decisioning, model-level explanations are absolutely necessary to describe the intended use, context, and the impact of these complex systems. But, given the reality of the behavior of these models and the presence of outliers for every observable rule, global explanations could in no way ever be enough to allow users to effectively challenge automated decisions made by “black box” algorithms.