When Humans Attack
Re-thinking safety, security, and AI
By Data & Society Research Lead Madeleine Clare Elish and Research Analyst Elizabeth Anne Watkins
This is the first blog post in our series on AI & security. As AI becomes increasingly integrated into our everyday lives, we need to start reconceptualizing our notions of security. Read the other post in the series here.
The most pressing safety risks of artificial intelligence (AI) will be found in mundane applications, not in the hands of killer robots. In 2017, a news report—played on a home TV—accidentally triggered someone’s Amazon Alexa to purchase a dollhouse. Later that year, a Burger King commercial intentionally hijacked Google Home Assistants as part of a marketing campaign. Commands hidden in white noise, played over loudspeakers, or embedded directly in music recordings, can direct Amazon Echos to unlock owners’ doors, even if a human is only hearing a lilting orchestra. The potential to produce, and be fooled by, fake audio and visual material, termed “deepfakes,” continues to increase.
Around the world, AI and machine learning (ML) systems are increasingly producing new and confounding concerns about safety, security, and wellbeing. Malicious actors have proven adept at gaming social media platforms like Facebook and Twitter, search engines like Google, and the news media in order to influence political discourse in the service of financial, political, and ideological agendas.
But the manipulation of media is not the only set of risks taking shape. More and more, fundamental applications are being supported by machine intelligence, from stock trading and financial-fraud detection to garbage collection. While dynamic, data-driven algorithms can introduce tremendous benefits into computing systems, they also generate new vulnerabilities. In order to adequately plan for and address these risks, we need to reconceptualize our notions of safety and security.
Attackers leverage the intelligence of a system by redirecting and manipulating the capacity to learn or to act on what has been learned.
The widely disseminated 2018 report Malicious Uses of Artificial Intelligence is a useful overview of the ways in which AI can expand the scope of existing security threats, introduce new types of threats, and change how attacks are conducted. Several other recent reports have focused on specific domains of activity, including military and national defense, threats to critical infrastructure, and threats to private enterprise.
In this growing body of literature we see three main areas of concern taking shape:
1) AI augments the capacities of traditional cyber-attackers, particularly to scale and target attacks. For instance, the tools at the disposal of a scammer are cheaper, more accessible, and easier to scale than ever before.
2) AI will augment the capacity to defend systems from abuse. At the same time that attackers’ tools are getting cheaper and more sophisticated, so too are the tools of defenders. This dynamic has led many to worry about an escalating cyber arms race.
3) AI, specifically machine learning systems, engenders novel kinds of risk.
How does AI create new kinds of risks? Most traditional models of cybersecurity are contingent on securing perimeters against unauthorized access — the logic of locks, keys, and online passwords. These fall short in the context of ML systems; the vulnerabilities of AI and ML aren’t just touch-points where an attacker may gain entry; the vulnerabilities exist in the interactions within and between the social, cultural, political, and technical elements of a system. The unique vulnerabilities of “intelligent” systems are the very mechanisms through which they become “intelligent” and interact with the world. That is, attackers leverage the intelligence of a system by redirecting and manipulating the capacity to learn or to act on what has been learned, undeterred by security practices focused solely on access. An attack on an image-recognition system, for example, can leverage the capacity of the system to recognize an image — like, a “STOP” sign — but distort that recognition — from “STOP” sign to “YIELD” sign — to achieve a malicious end, i.e. creating the conditions for a car to crash.
It is imperative to leverage a socio-technical frame to conceptualize safe and secure AI.
Thus far, conceptualizing this expanded set of vulnerabilities and developing appropriate defenses has been taken on most directly by the technical computer science community. Even before its widespread use, researchers demonstrated the potential for the integrity of machine learning systems to be undermined through maliciously chosen errors in training data. For example, in the context of spam filtering or malware detection. More recent technical work, in the emerging field of “adversarial machine learning,” has investigated how and under what circumstances attacks might occur. This work involves designing and testing “adversarial examples,” referring to inputs that are intended to cause a machine learning model to make a mistake, like generating “optical illusions for machines.”
Research has demonstrated that machine learning systems can be undermined not only through tainted training data, but also by carefully disguised physical objects or phenomena. In such cases, modifications made in the physical world may remain so slight that they are imperceptible to humans. Such was the case when researchers demonstrated that a computer vision system could be tricked into seeing a “STOP” sign as a speed limit sign reading “45 MPH.” The authors of that paper described how they altered a “STOP” sign in a way that would fool the system but also be dismissed as graffiti by a human observer.
So where do we go from here?
Articulating how AI creates new and unique kinds of risks is often about calling attention to the ways in which AI and ML systems are part of larger social contexts or ecosystems. AI and ML must be understood as socio-technical systems, in which the “technology” can never be understood apart from the social actors and social processes that make up the system. Consequently, solutions that focus exclusively on technology will fail. Effective security requires addressing the technical and social points of vulnerability within AI systems.
As digital systems become more interconnected across sectors, and algorithmic tools shape more decision-making, it is imperative to leverage a socio-technical frame to conceptualize safe and secure AI. To achieve this, the traditional concerns of safety and security research need to expand, and the expertise and perspectives of diverse communities need to be a prominent voice in the conversation.
In addition to technical research in the lab, there needs to be more sociological and ethnographically oriented research into the vulnerabilities that emerge despite designers best intentions. Traditional research reports are only one way to conduct such research. Additional methods could include “abusability testing,” White Hat hacker or “bug bounty” programs, “red teaming” scenarios, or even employing science-fiction writers to flesh out potential future vulnerabilities.
Perhaps the best place to start is by asking those who are or work with vulnerable communities. Civil society and advocacy communities have longstanding traditions of examining how systems and programs may or may not be safe and secure for different kinds of users. Their expertise and perspectives will be invaluable in assessing and mitigating the harms and risks at stake in AI.
Madeleine Clare Elish is the research lead of the AI on the Ground Initiative at Data & Society.
Elizabeth Anne Watkins is a research analyst at Data & Society supporting the AI on the Ground Initiative. She is currently pursuing a doctorate in Communications at Columbia University.
