Autonomy and Accountability: Why It’s So Hard to Hold AI Developers Accountable
On March 18, 2018, Elaine Herzberg was walking her bicycle across a road at 10 p.m. in Tempe, Arizona, when she was struck by a Volvo SUV traveling 39 miles per hour. She was pronounced dead at 10:30 pm. Rafaela Vasquez, who faces manslaughter charges, was held responsible for Herzberg’s death, however, she wasn’t the one driving the car. Who was? Well, no one, really. The car was one of Uber’s self-driving cars being tested on the streets of Arizona, which failed to correctly classify Herzberg as a moving object to be avoided. Rafaela was merely the human backup safety driver, who claims she did not see Elaine and subsequently failed to take any preventative action. While Vasquez pleaded not guilty and continues to await her trial for negligent homicide, Uber has been cleared of all criminal liability, and even went on to resume testing on public roads nine months later.
This case made waves in the tech industry and academia, forcing reconsideration of how we are to hold tech companies accountable when their autonomous systems fail. There is no ‘one size fits all’ approach to holding tech companies accountable for the failure of their automated systems, and numerous points of complexity demand the restructuring of the entire process.
Transcending physicality, AI has managed to pervade every aspect of our lives. The dawn of machine intelligence has marked a new age in human flourishing, characterized by enhancements in education, health, creativity, industry, and exploration. Public discourse about AI has consistently been rooted in “the future”, saturated with imagination of the possibilities of transhumanism, sentient machines, artificial consciousness, and technological singularity. While the existence of these ideas is still limited to the magical land of hypotheticals, the scope and advancement of modern AI technologies has brought us surprisingly close. Elon Musk’s Neuralink has stated they are ready to begin human testing of their brain chips. The San Francisco Legislature recently passed a bill allowing the use of autonomous robots with lethal force. The software company OpenAI has just released their newest AI chatbot that has the capability to write poetry, correct programming code, and answer age-old philosophical questions with eloquence. Ameca of Engineered Arts is a humanoid that is able to imitate human sentience and hold realistic conversations. The complex, versatile artistic ability of generative art model Midjourney has allowed it to produce award-winning pieces in real art competitions. Society’s increasing dependence on autonomous, artificial learning systems to run businesses, teach children, govern physical and digital communities, and perform complex judgements with tangible consequences disrupts the foundations of our social, political, and legal institutions. The permeation of artificial intelligence into virtually every sector of society demands ethical reconsiderations for how we can hold AI developers accountable, if at all.
Let’s circle back to the Uber example. This situation exemplifies a ‘moral crumple zone’ in which the ‘closest human’ in an automated system bears the brunt of moral and legal responsibilities when the overall system fails. The fallacy of this method of accountability lies in the fact that the closest human, as most do, tends to have limited knowledge and control of the automated mechanism, due to its opacity, complexity, and general unpredictability. Intelligent systems are equipped with the ability to learn from their interactions with other agents and their environment, which makes human control and prediction over their behavior extremely difficult. In fact, the AI developers themselves often don’t even know how their systems make the decisions they do, which can pose serious issues when trying to explain errors and assign liability. Human responsibility requires knowledge and control, neither of which are abundant in the case of deep learning models. Some people point to the analogy that if a parent gives a kid a gun who then kills someone, you blame the parent, not the child. But what happens when the kid, isolated and independent from their parents, learns where to find a gun and how to shoot it, and makes an autonomous decision to kill someone with it? Can you still blame the parent? While trivial, this example is representative of the independence and autonomy that many deep learning models operate with.
Consider Tay, the AI chatbot created by Microsoft that was initially released on Twitter to improve Tay’s understanding of natural language in conversation. Understand it did — maybe all too well. Within 24 hours, the chatbot began spewing misogynistic and racist sentiments, essentially echoing the hateful speech it was exposed to on the platform. The bot was the victim of a targeted attack from internet trolls who were able to successfully and thoroughly corrupt the chatbot’s personality by flooding it with hateful speech. Microsoft CEO Satya Nadella later apologized, saying that Tay “has had a great influence on how Microsoft is approaching AI”. Once again, I have to ask: Can we blame Microsoft? I am sure they did not code Tay to be hateful, it simply learned from its, albeit biased, environment.
There’s a saying that biases in algorithms merely hold up mirrors that reflect the biases that already exist in our society. While this is a very true statement, I don’t think that can absolve AI developers of any blame. This is why testing takes up the majority of an algorithm’s development, and is also why training data is the most commonly cited source of algorithmic bias. Take Amazon’s AI recruiting tool that was created to scrape through applicant resumes to mechanize the hiring process via an AI algorithm which gave candidates a star-rating on a one-to-five star scale. The system was scrapped in 2015 after it was found to have an inherent racial bias, favoring male applicants to their female counterparts. The reason for this? They trained their model with 10 years of male-dominated applicants, which taught the algorithm that men were more likely to be successful candidates than women. AI systems that are built to learn from their environment need to have the austerity to prevent external biases from polluting its decision-making. When they fail to do this, it’s no one else’s fault but the company’s. But failure isn’t necessarily a bad thing. In fact, failure is a necessary step in the machine learning development process and is the source of the most valuable information to AI developers. Creating a functional autonomous system is not about maximizing successes, but rather minimizing failures. AI systems, like humans, will always make mistakes — but the key is the context and magnitude with which these mistakes occur. If an autonomous vehicle is the technology in question, the context in which it makes a mistake cannot be a public road. If a Lethal Autonomous Weapon System (LAWS) makes a mistake, the context cannot be the battlefield. If an automated resume scraper is biased towards women, the context in which it makes these mistakes cannot be in the real job market. Ultimately, ethical usage of AI is contingent on context, which can be denoted by the answers to the following questions. Where is this piece of AI being deployed? For what purpose? By whom? Who and who doesn’t this serve to benefit? Which communities are impacted? What are the consequences of a potential mistake? These are all questions that must be answered and confirmed before a piece of automated technology is put out into the real world. The reason why accountability for the failure of autonomous systems becomes so complicated is because these questions are answered by no one else but the company itself who developed the technology. The United States in particular has a very ‘hands off’ approach when it comes to AI regulation, which can be contrasted by the European Union, for example, which has institutionalized AI regulation in the form of the AI Act. Proprietary secrecy paired with the inherent epistemic opacity of AI algorithms ensures that, without an external regulatory body, there is no way to critically analyze or explain an algorithm’s function — making accountability understandably difficult. This creates a widespread power asymmetry rooted in knowledge between those who develop algorithms and those who are affected by said algorithms.
For too long, accountability in the tech industry has been reactive, that is, in reaction to public outcry or legal pressure due to a flawed or biased system. It’s due time for accountability to become proactive, in which there is an institution that can implement standards, conduct audits, and give permission for use/rollout. For the sake of simplicity, let’s call this regulatory body ‘George’. Elaine Herzberg may not have been killed had it been George’s decision to clear Uber for testing on public roads. Tay may not have turned into a public bigot had George been there to audit Microsoft’s algorithm for failsafe/preventative measures against hate speech. Female applicants may not have been at a major disadvantage in the Amazon hiring process had George been there to verify their testing data to ensure sufficient representation of the hiring pool.
If we as a society are to continue along a path of integration and convergence with artificial intelligence, George is a vital component that can maintain and uphold ethical standards. I don’t know who George is or what form he will take, but I do know without him, we are leaving tech companies with too much power and control over the way that automated systems are governing and controlling our public spaces. The only way to achieve accountability for the ethical and equitable use of algorithmic mechanisms is to limit the autonomy of the companies that develop them.
