Law and Adversarial Machine Learning
On Friday, December 7, 2018, Salome Viljoen presented our paper, “Law and Adversarial Machine Learning” at the Security and Machine Learning Workshop at the Neural Information Processing Systems (NeurIPS) conference. The paper, co-authored by Salome, David R. O’Brien, Kendra Albert, and me, provides a survey of some existing legal remedies for attacks that have been demonstrated on machine learning systems, and suggests some potential areas of exploration for machine learning researchers given the existing state of legal uncertainty.
Why look at machine learning and the law?
Machine learning (ML) is a subset of artificial intelligence that is being widely deployed in many settings, from chatbots to disease diagnosis. But despite the enthusiasm for ML and its increasing integration into critical infrastructure, it is often insecure. With little effort, adversaries can compromise the integrity and availability of ML systems through targeted or blanket attacks demonstrated by a number of researchers in adversarial machine learning since 2004.
The assault on ML systems have real-world implications: in 2016, researchers from Cornell reverse-engineered the underlying algorithm behind several popular machine learning systems. One of the affected companies responded: “Said another way, even if stealing software were easy, there is still an important disincentive to do so in that it violates intellectual property law.”
Our paper argues that it is unclear exactly whether such statements are true — some commentators and many organizations have assumed, without further analysis, that current legal regimes account for adversarial machine learning attacks, but for at least some attacks, such protection may be uncertain.
What attacks on Machine learning systems did we analyze?
We analyzed the following set of attacks on ML systems, which can be informally defined as:
- Model Inversion — Attackers recover the training data used by the ML system.
- Poisoning attacks — Attackers corrupt the input to the ML systems during its learning phase, so that it learns incorrectly, and therefore performs the task incorrectly.
- Perturbation attacks — Instead of corrupting the input to the learning phase, attackers corrupt the query to the machine learning systems during the evaluation phase.
- Model Stealing — Attackers reverse engineer the underlying algorithm
Takeaways from the paper:
- The Computer Fraud and Abuse Act plausibly covers some supply chain attacks, perturbation and poisoning attacks if its statutory language is interpreted in certain ways.
- Although model stealing and model inversion attacks may seem like good use cases for intellectual property law, copyright law may provide little protection because of the protection of source code as a literary work.
- It may be difficult for end users to invoke products liability law against companies for failing to protect against adversarial machine learning attacks because of the lack of consistent standards for secure ML development. There is no established standard or industry wide practice for protecting against adversarial examples
What’s next?
As new technologies permit new potential harms, judges, legislatures, and regulators are on the spot for rationalizing law with technology. For adversarial machine learning, this process is just beginning — judges have not had much cause to determine the applicability of existing law to attacks on machine learning, nor have specific laws been passed to regulate machine learning systems.
As we mentioned in our Call to Action section in the paper, we are particularly thinking about the dual use of adversarial examples. Adversarial examples can facilitate censorship of political dissidents: ML researchers should anticipate that oppressive governments could seek backdoors in consumer ML systems. On the same note, there are also purported benefits: dissidents in a totalitarian state may be able to evade facial detection using 3D printed glasses as shown in recent research
The NeurIPS paper was just a starting point — there are many other types of attacks we did not provide legal commentary — for instance, reward hacking. The next step is a more detailed study on the different types of intentional and non-intentional ML failure modes.
If you are interested in this intersection or want to discuss further, please reach out to Ram.Shankar@microsoft.com or @ram_ssk on twitter!
