California Regulates Automated Decision-Making Technology
Preface: In March of 2023, I submitted a public comment* on California’s proposed data privacy rule-making around so-called “automated decision-making” (ADM) technologies, which include artificial intelligence (AI)/machine learning (ML) systems. I believe we are at a critical inflection point in the development and deployment of these technologies. California state regulation is an important first step towards broader federal regulation of increasingly powerful AI systems of the future.
In my comment, I urge the new California Privacy Protection Agency (CPPA) to adopt a broad definition of ADM technology that would encompass a range of decision-making technologies and AI systems. The regulations should require businesses to conduct privacy impact and risk assessments before deployment to ensure safety, accuracy, fairness, and human oversight. This is particularly important for AI technologies like large language transformer models (LLMs), such as ChatGPT, which have been trained on massive, permissionless data scraping of personal data and can produce inaccurate outputs.
ADM and AI technology raises risks to important human values. Regulations should ensure transparency, accountability, and human oversight. As these technologies becomes more pervasive and powerful, we need to understand and mitigate their risks.
(*For ease of reading, I’ve deleted the footnotes that were submitted with the original text.)
#privacy #ai #artificial intelligence #regulation #automation
March 27, 2023
California Privacy Protection Agency
Re: PR 02–2023; Comment on CA Regulations Governing Automated Decisionmaking Technology Dear California Privacy Protection Agency,
I appreciate the opportunity to comment on California Privacy Protection Agency’s (“CPPA”) regulation of automated decisionmaking (“ADM”) technology as mandated by the California Consumer Privacy Act (“CCPA”), as amended by the California Privacy Rights Act (“CPRA”).1 These comments reflect my personal opinions, but they are informed by more than two decades of professional experience in data privacy and automation technologies as a registered inhouse and general counsel for California-based and other technology companies. I offer these comments, because I believe meaningful regulation of ADM technology at this critical juncture is essential for the privacy and the well-being of Californians.
My comments focus primarily on the definition of automated decisionmaking (“ADM”) technology and its regulatory scope. As detailed below, I urge the CPPA to define ADM technology broadly as a process for making decisions using automated means without significant human involvement and with significant potential effects on a California resident or household.
Justifying the necessity for such a broad definition of ADM technology requires us to step back briefly for a broader perspective. Vast increases in data availability and advances in ADM technology, including rapid developments in Artificial Intelligence (AI) and Machine Learning (ML), are significantly changing organizations’ decisionmaking processes. ADM technology and consumer profiling impact California residents by influencing or determining high stakes decisions, such as who gets a job interview, a loan approval, or a gig worker assignment. Other, seemingly less impactful uses of ADM technology and profiling processes, such as auto freezing an active account, or algorithmically distributing news or social media content, may also produce potentially significant effects, particularly if evaluating the potential or cumulative impact.
Broadly defining ADM so that significant reliance on ADM technology for decisionmaking triggers minimal regulatory protection, such as rights of notice, explanation, opt-out and human appeal, will help mitigate potential harms to California residents and households. ADM technology has no common sense. ADM processes can ignore important intangible ethical, moral, and other human considerations that should guide high stakes decisions about people’s lives. Promoting transparency and significant human input into ADM processes and requiring explanations in understandable terms is crucial for perceived legitimacy
Mandating risk assessments, and ongoing audits for ADM technology prior to deployment also supports ethical use of algorithm-assisted decisions. Compliance teams require these measures to lift the cover on opaque and complicated ADM processes, allowing them to assess risks and tradeoffs and implement mitigation strategies before deployment. Risk assessments must specifically document human involvement in decision-making, assessing ADM technology usage at various stages, and establish regular governance structures to assess algorithmic accuracy, safety, fairness, transparency, and accountability within organizations.
As ADM technology grows increasingly complex and integrated into organizational decisionmaking at all levels, human input, oversight and redress help protect core ethical human values, including privacy, accountability, fairness, and agency. ADM technology regulations should protect the contextual privacy rights of all California residents, ensuring fairness and transparency in processing of personal information, including the use of such data in training large language models (LLMs). Clearly defined and enforced ADM technology regulations will help protect Californians’ privacy rights and guarantee essential human accountability for ADM technology and processes.
Automated Decisionmaking:
1. What laws requiring access and/or opt-out rights in the context of automated decisionmaking currently apply to businesses or organizations (individually or as members of specific sectors)?
Before the EU’s General Data Protection Regulation (GDPR) was implemented in 2018, EU data protection laws already governed automated decision-making systems. Considering this extensive history and the CCPA’s adoption of GDPR’s ADM regulatory principles, it is essential to thoroughly examine the application of these principles by EU courts and data protection authorities (DPAs).
Article 22 of the GDPR restricts the use of automated decision-making systems where they are 1) “solely automated” and 2) have “legal” or “similarly significant” effects. On the first threshold, the following factors determine whether an ADM process is “solely automated”: 1) whether the decision is supported by a written assessment made by a human; 2) whether the decision is reviewed by a human supervisor; 3) whether the company’s employees have been specifically trained and given detailed guidance on decision-making considerations; and 4) whether the decision was an interim one that is still subject to final human review.
In determining whether an ADM process has “legal or significant effects” on the data subject, the European Data Protection Board (“EDPB”) has proclaimed that a “legal effect” must affect someone’s legal rights, such as the freedom to associate with others, vote in an election, or take legal action under a contract. A decision has a “significant effect’’ if it has the potential to significantly affect the circumstances, behavior or choices of the individuals concerned; have a prolonged or permanent impact; or lead to exclusion or discrimination.
EU based courts and DPAs have construed significant effects broadly including impacts to individual circumstances, behavior or choices where there is a prolonged or permanent impact, including when 1) the decisionmaking significantly affects a resident’s rights and freedoms or legitimate interests; 2) the decisionmaking significantly affects a resident’s economic situation, social situation, health, personal development, reputation, or other important interests; or 3) the decisionmaking significantly affects a resident’s physical or mental health.
Under the GDPR, organizations applying automated decision-making tools must implement “suitable measures to safeguard the data subjects’ rights and freedoms and legitimate interests.” Such measures include the right to obtain human intervention by the controller, a right to contest the decision and “the right to explanation” (i.e., “meaningful information about the logic involved” in the ADM process).
Additional protections require regularly checking datasets used for bias and introducing safeguards to prevent errors and inaccuracies. The United Kingdom’s Information Commissioner’s Office (“UK ICO”) provides practical guidance for organizations implementing automated decision making and profiling that is centered on conducting data protection information assessments or “DPIAs” to consider and address the risks before starting any new automated decision-making or profiling.3
The guidelines to automated decision-making and profiling issued by the Article 29 Data Protection Working Party, now the ECPB (“Guidelines”), note that “complexity is no excuse for failing to provide information.” Organizations should provide “factors taken into account for the decision making process,” “their respective weight at an aggregate level,” as well as information on: 1) the categories of data that have been or will be used provided to individuals, 2) why these categories are pertinent, 3) how any profile using the automatic decision making process is built including any statistics used in the analysis, 4) why the profile is relevant to the automated decision making process, and 5) how it is used for a decision concerning the individual. The Guidelines advise that organizations need not provide a complex mathematical explanation about how the algorithms work or disclose the algorithm, but the explanation must be sufficiently for the individuals to act upon it to contest decisions or to correct inaccuracies or request erasure.
Despite high threshold triggers to qualify for Article 22 protection, EU based courts and DPAs examine the underlying lawfulness of the data processing for the ADM process, thus going beyond the scope of strict Article 22 construction under the “solely automated” and “significant effect” triggers. EU based courts and DPAs strictly scrutinize ADM use to ensure lawful data processing and broad accountability. As the Future of Privacy Forum’s detailed case research report concluded, EU based courts and DPAs frequently go beyond Article 22 in an ADM inquiry to require transparency measures, fairness and non- discrimination documentation, and strict conditions for valid consent.4 These include specific transparency and access requirements for ADM processes under Articles 13, 14 and 15, and mandates to conduct DPIAs for ADM processes under Article 35.
(3) With respect to the laws and other requirements, frameworks, and/or best practices identified in response to questions 1 and 2:
a. How is “automated decisionmaking technology” defined? Should the Agency adopt any of these definitions? Why, or why not?
b. To what degree are these laws, other requirements, frameworks, or best practices aligned with the requirements, processes, and goals articulated in Civil Code
§ 1798.185(a)(16)?
c. What processes have businesses or organizations implemented to comply with these laws, other requirements, frameworks, and/or best practices that could also assist with compliance with CCPA’s automated decisionmaking technology requirements?
d. What gaps or weaknesses exist in these laws, other requirements, frameworks, and/or best practices for automated decisionmaking? What is the impact of these gaps or weaknesses on consumers?
e. What gaps or weaknesses exist in businesses or organizations’ compliance processes with these laws, other requirements, frameworks, and/or best practices for automated decisionmaking? What is the impact of these gaps or weaknesses on consumers?
f. Would you recommend that the Agency consider these laws, other requirements, frameworks, or best practices when drafting its regulations? Why, or why not? If so, how?
The Agency should adopt a definition of “automated decision-making technology” that encompasses a wide range of automated decision-making technology applications and processes, including automated processes, algorithms, artificial intelligence, and machine learning systems that use personal data and are used in decisionmaking. Unlike Article 22 of the GDPR, the CPPA’s statutory mandate to issue ADM regulations is not limited to “solely” automated decisions and those with “legal” or similarly significant effects. The CPPA should take notice that the higher thresholds under the GDPR Article 22 requiring “solely” automated decisions that have “legal” or “similarly significant effect” are mitigated by application of other robust GDPR safeguards, which are broadly applied to EU ADM cases. As referenced above, ADM inquiries by EU based courts and DPAs have frequently scrutinized lawful data processing requirements under the GDPR far beyond the scope of narrowly constructed Article 22 triggers.5
Because the CPPA does not provide similar far-reaching protections and remedies, the definition of ADM technology under the CCPA needs to be sufficiently broad and flexible without application of rigid trigger thresholds such as “solely automated” or “legal or similarly significant effect.”
For the reasons detailed below, CPPA should define automated decisionmaking technology broadly as making decisions using automated means 1) without significant human involvement; and 2) where the decisionmaking has a significant potential effect on a California resident or household.
This definition is broad enough to encompass a wide range of automation technologies that are used to make decisions, from simple algorithms to advanced machine learning models. It clarifies the scope of
5According to the Future of Privacy Forum’s well-researched report on Article 22: [T]he GDPR’s protections for individuals against forms of automated decision-making (ADM) and profiling go significantly beyond Article 22. In this respect, there are several safeguards that apply to such data processing activities, notably the ones stemming from the general data processing principles in Article 5, the legal grounds for processing in Article 6, the rules on processing special categories of data (such as biometric data) under Article 9, specific transparency and access requirements regarding ADM under Articles 13 to 15, and the duty to carry out data protection impact assessments in certain cases under Article 35.” FPF Report: Automated Decision-making under the GDPR — A Comprehensive Case Law Analysis.
ADM regulation and ensures that all relevant technologies are covered. The nuances of this definition can be further parsed around three key questions: 1) What is “decisionmaking”; 2) What is “significant human involvement” in decisionmaking, and 3) What is a “potentially significant effect” of decisionmaking?
1. “Decisionmaking”: A decision can be defined as a choice made or action taken from a range of options, including the act of selecting one course of action from several possibilities. Decisionmaking encompasses the entire process of making decisions, including the steps of identifying a problem, inputting information, and evaluating options and additional inputs and data processing to produce an output or render a choice or action. It’s clear that decisionmaking can be a complex process with multiple stages and steps.6 In the real world, ADM technology and processes interact upstream and downstream with profiling technology and processes. For example, an automated decision-making system might use profiling to make predictions about which consumers are likely to be approved for a loan; or a profiling system might integrate with an ADM technology to make decisions about which individuals to target with a special promotionally priced offer.
The regulations should make clear that any inquiry into whether a decisionmaking process is automated and has significant effect should focus on the entire decision-making process, not just a final stage of a decision. ADM technology regulations need to apply to the entire decisionmaking process and protections should apply to circumstances where automated processing has foreclosed downstream consideration, despite human input in the so-called final decision. Under the CPRA, the CPPA is free to define decision- making to encompass the entire process through which a covered business or organization evaluates, considers, or renders a decision, including upstream processing of personal information and profiling. An automated decision can include decisions that are claimed to be temporary or interim if they have a significant effect on the California resident or household. For example, a decision to freeze a user’s account based on suspected fraud can have a significant effect even if it is claimed by the company to be “interim” and not final. A broad definition of decisionmaking will ensure that all types of automated decision-making processes are held accountable for their potential impacts on privacy and consumer rights.
2. “Significant Human Involvement”: The Proposed Regulations should clarify that an organization must demonstrate “significant human involvement” in the decision-making process by providing adequate documentation or support of human input. While this is similar to the EU’s application of the “meaningful human involvement standard”, the inquiry should be interpreted more broadly to examine the entire decision-making process, not just isolating the final stage of a decision. The regulations should avoid adopting an overly narrow or mechanistic approach that focuses only on whether there was adequate human input in the final step.7 This means applying the “significant human involvement” inquiry to the entire decisionmaking processes, including where an upstream automated processing has foreclosed downstream consideration despite human input. This broad consideration of the entire “decisionmaking process,” including its interaction with profiling, is discussed further below in in subpart 2 “significant effect” and subpart 3 “decisionmaking.”
Determining whether there’s “significant human involvement” in the decisionmaking requires a broad inquiry into the overall organizational environment underlying the decisionmaking, including an organization’s structure, reporting lines, chains of approval, staff training, and internal policies. The burden should be on businesses using ADM technology to demonstrate adequate human involvement in the the ADM process. To incentivize businesses to review their entire decisionmaking process for human involvement, the regulations should emphasize the importance of conducting data protection risk assessments or DPIAs and documenting mitigation measures to reduce risks from ADM technology processes. The CPPA should take note of the EU’s strict scrutiny of complex ADM technology, such as LLMs, that defy interpretability and can result in “automation bias.”8 Captured by “intelligent” technology that appears to be our ally, and incapable of explaining ADM processes in human understandable terms, employees can develop a deference to ADM technology that sometimes resembles mysticism.
Automation bias combined with the increasing pervasiveness and scale of ADM technology necessitate a broad scope of regulation where there’s no significant human involvement in the decisionmaking.
3. “Potentially Significant Effect”: Determining whether an automated decision has the requisite “potentially significant effect” on a California resident or household should not be restricted to specific rigid types or domains of decisionmaking, nor should it be narrowly focused on only fully realized effects. Under the CPRA, in contrast to other pending state ADM regulation, there is no statutory language suggesting that a decision’s impact be limited to defined areas of impact, such as financial services, housing, insurance, education, employment opportunities, healthcare services or access to basic necessities. Rather, it is appropriate to interpret “potentially significant effect” with reference to the CPRA’s explicit and broader definition of profiling as “performance at work, economic situation, health, personal preferences, interests, reliability, behavior, location or movements.”
The ADM technology definition also should not be limited to actual realized harms from a so-called final decision, but potential harms such as the potential for an upstream automation step in the decisionmaking process to foreclose a downstream outcome despite human input in the later stages of the process. For example, the automated ranking or filtering of a job applicant’s materials may foreclose practical consideration to interview candidates by a human reviewer. The automated freezing of a gig driver’s account forecloses tangible income opportunities even if the account suspension is deemed temporary and subject to ultimate review by a human. In such cases, the ADM has a significant effect. ADM technology use also often includes and relies upon upstream profiling that is expressly included in the definition of automated decisionmaking under the language of the CPRA. The larger point is that the express use of the word “decisionmaking” and “profiling” in the statutory language suggests regulation of a larger ADM process that often includes upstream profiling, rather than trying to isolate the location of a distinct and final “decision” in a multi-stage process that includes automated and human components.
The regulations should expressly clarify there are significant effects when a “decision” produces immediate and non-temporary consequences for individuals, including affecting an individuals’ income- making opportunities.
ADM Technology Regulation & Large Language Transformer Models (LLMs): A “decision” can be an action, choice or output based on factual data, including the automated generation of content based on inferred data and profiling as discussed below. The regulations should protect the privacy of individuals whose data is used to train automated decision-making models. The regulations should also require businesses to conduct a privacy impact assessment (PIA) before deploying automated decision- making technology. ADM technology employing large language transformer models (“LLMs”) has been trained on massively scraped data sets to produce outputs that can be inaccurate and are difficult to interpret, predict or explain. They can produce material factual errors that can harm personal privacy and reputational integrity, as well as decision outputs that are biased and harmful.
LLMs have been trained based on permissionless data scraping that violates personal privacy. Even if the personal information scraped by LLMs were to be “publicly available” within the specific context in which it was posted, the data scraping, processing, and use of personal information violates “contextual integrity,” a core privacy principle. There’s simply no express or implied reasonable expectation or permission for using personal data in this manner.9 The ADM technology regulations should protect the contextual privacy rights10 of all California residents and ensure that all types of ADM technology are held accountable for their potential impacts on privacy rights, which require fairness and transparency in the processing of personal information, including using personal data to train LLMs.
Conclusion: ADM technology use raises risks to data privacy, as well as harmful bias concerns from limited or discriminatory data that can reinforce social inequities. ADM technology and processes ignore important intangible human factors that go into real-life decision-making — the ethical, moral, and other human considerations that appropriately influence decisions in the real world. ADM technology should not be allowed to erode privacy, fairness, and human agency in the name of greater efficiency. The regulations must define ADM technology broadly and ensure minimal transparency and accountability, requiring covered businesses to conduct risk assessments before deployment of ADM technology to ensure accuracy, fairness, and human oversight. As ADM profiling technology becomes more complex, human oversight is crucial to ensure privacy, safety, fairness, accountability and respect for human autonomy and agency. Thank you for your attention to this critical issue. I look forward to seeing the adoption of meaningful ADM technology regulations to protect the privacy and well-being of Californians.
Respectfully submitted,
David Phillips
