Building on the Blueprint

We need everyone around the table to move AI principles into practice.

By Maroussia Lévesque, Tom Zick, Dennis Redeker, You Jeen Ha, Florian Martin-Bariteau, and Mason Kortz — inspired by discussions within the Rebooting AI Working Group at the Berkman Klein Center for Internet & Society

The White House recently released a Blueprint for an AI Bill of Rights. It consists of five high level principles and supplementary information detailing its relevance, expectations, and examples of good practices.

We propose to build on the Blueprint, opening up the conversation between and across people working on AI governance from various fields.

The legal disclaimer is careful to say that this is ultimately a non-binding white paper, so any expectation of binding legislation is off the table for now. The Blueprint aligns with a history of aspirational, yet mostly non-binding documents that articulate people’s rights and technology governance principles in the digital age. Such Internet bills of rights have been proposed by various actors including tech firms, civil society, and states. Even more closely related, documents outlining AI principles have proliferated around the world, mostly non-binding and mostly as high-level principles.

Yet the Blueprint for an AI Bill of Rights stands out from other such documents in several respects. First, the document was drafted in a relatively inclusive one-year process, which has been well documented. Second, it adopts a civil rights framing. Civil rights come with a body of clarifying case law, enforcing institutions and mechanisms, and a shared vocabulary to articulate, contest, and redress harm. Third, concrete executive/administrative actions accompany the Blueprint. This signals a comprehensive governmental approach to AI policy that is often missing and may allow the Blueprint to percolate across the various nooks and crannies of government. Fourth, the Blueprint attempts to complement high-level guidance with cues for implementation. In our opinion, this crucial piece is often missing to bridge ambitious policy ideas to the nitty-gritty task of implementation.

In this post, we explore how to further specify the Blueprint’s principles into workable guidance for engineers and others on the ground. We propose to build on the Blueprint, opening up the conversation between and across people working on AI governance from various fields.

Now what?

The Blueprint provides an opportunity to further translate general guidance into technical specifications for industry adoption. While “plug and play” recipes are challenging in light of the contextual variability across deployments, academic and civil society organizations can play a key role in mediating between theory and practice. The Blueprint’s breath is vast, but two areas conducive to such specification are discrimination and explanation. In what follows, we propose concrete ways for multi-disciplinary groups to narrow the gap between high-level statements and implementation.

Discrimination

The technical companion to the Blueprint purports to provide concrete steps to move the principles into practice. Its section on disparity assessment suggests three types of metrics: demographic performance measures, overall and subgroup parity assessment, as well as calibration. While this is a step in the direction of more explicit guidance, it still leaves important questions unanswered for the technical community at the frontline of implementing protections against discrimination. To name but one obstacle, some fairness metrics are mutually exclusive, in the sense that satisfying them simultaneously is impossible when demographic subgroups fare differently. As the controversy about risk assessment tools in the criminal justice system illustrates, equalizing error rates entails a loss of accuracy when the base rate differs between demographic groups. The devil is, so to speak, in the metrics.

There’s no one-size-fits-all answer as to what fairness metrics should prevail. But developing a common language to bridge high-level equality principles with the actual task of measuring a model’s performance is a good starting point. A shared vocabulary will foster transdisciplinary debates and avoid the proverbial two ships passing in the night. This is a two-way street: those with legal and policy training should strive to get a more granular understanding of the statistical theory undergirding fairness metrics, while technical experts should grasp the contours of legal and ethical implications. As examples of this sociotechnical approach, questions for further research include:

• What is missing from the proposed metrics in the Blueprint?
• Which fairness metric is most compatible with legal obligations in a given context?
• Who should make judgment calls about which fairness metric to constrain a model with?
• Are the existing siloed institutions (e.g. developers conceiving models ex-ante; courts reviewing harm post-facto) capable of making such calls?
• How should we (re)think the pipeline between the technical front-line implementors and external oversight?

Explanation

Much like discrimination, the notion of explanation in a colloquial and legal sense differs from the technical conception. Moreover, explainability is subject to technical limitations that inhibit its efficacy in a multitude of ways. The implementation guidelines for the Blueprint bundle the notion of ex-ante and ex-post explanations.

On the ex-ante side, the guidelines prescribe documentation reporting “how the system works and how any automated component is used to determine an action or decision,” as well as stakeholder notification when the system has changed. Ex-post explanations are more like justifications. The guidelines state this “need not be a plain-language statement about causality but could consist of any mechanism that allows the recipient to build the necessary understanding and intuitions to achieve the stated purpose.” The guidelines emphasize the need for such explanations to be tailored to their use and validated by user experience research. That’s great. But how that plays out in real world systems can get complicated.

The guidelines correctly point out that explainability is an active area of AI research. This means that ‘explanations’ from AI can take a variety of forms, some of which may not comport with intuitive definitions of explainability. Some explanations can draw on the data used by an AI to reach a decision. Others are completely ex-post, meaning that explanations can be provided by an entirely separate AI system that has been trained to output explanations for the original system’s recommendations. In other words, one AI system produces recommendations, another explains them. Some initial questions that come to mind are:

• What level of sophistication do AI explanations need to reach before they can fill the role the Blueprint envisions? Which types of explanatory AI have cleared this bar (if any)?
• What should be contemplated in ex-ante documentation given the type of AI system?
• How much of the system should be described ex-ante versus how much can ex-post rationales be relied on?
• What does it mean for models to be fully transparent?
• How should techniques for ex-post explainability vary with risk levels?

Next steps for building on the Blueprint

Neither policy, law, or technical approaches alone can build on the Blueprint. Just like carpenters, plumbers, and electricians contribute to building a house, we need different expertise to come together. In terms of AI governance, this requires technical systems to surface facts for human oversight, rather than baking in opaque editorial choices that resist external review. Thankfully, we have references in adjacent areas on how to improve the interoperability between technical implementation and external oversight. For instance, data nutrition labels provide qualitative assessments to inform external evaluations of data provenance, model cards foster transdisciplinary discourse on fairness by detailing salient model parameters, and reward reports do the same for continually learning AI systems.

The Blueprint provides a foundation for aligning AI systems with democratic values and civil rights and liberties, but it is ultimately up to all of us to build on that foundation.