Machine Learning security, your algorithms may be more vulnerable than you think
Attacking your model and how to protect it
Written by Jad Hamdouch & Ismael Bouarfa.
What is at stake ?
Machine learning and deep learning are the ability of computer systems to perform a specific task based on enormous amounts of data without receiving explicit instructions. Based on algorithms and statistical models, Machine Learning relies on patterns. Known as a subset of Artificial Intelligence or the step before it, many services and apps are based on this scientific field.
Used in email filtering (learning from spam to predict it later), DNA sequence classification, speech recognition, financial markets analysis… it has become an important business asset for many companies that manipulate large amounts of data. A basic example is the online advertising business based on the data that we produce when surfing on the web, used to feed machine learning datasets to predict trends.
However data scientists are usually highly focused on the framework capabilities at the expense of security. But why does security matter in machine learning? Nowadays, machine learning is used in many processes of decision making. Now imagine that these decisions affect people directly (ie. university admissions) and your AI learns to become racist. Impossible? 3 years ago, Tay the AI of Microsoft on Twitter started tweeting racist tweets 24 hours after her release. “The more you chat with Tay the smarter she gets.” they said. After massive chat where she was taught racist comments she started posting racist tweets.
Ok, compromising the intelligence of an AI’s twitter might not have a major impact on our lives, but imagine that your autonomous car identifies a red light as green… Yes, attacking machine learning is possible from the learning perspective. A ML model tends to become what it is trained for depending of the material (Data) he is receiving.
How to attack Machine Learning
Machine learning models aren’t black boxes. People are used to see data science as good predictions made from data without truly understanding the logic behind the process. There are three dangerous types of attack:
- Adversarial Examples
Adversarial examples are inputs to a neural network that result in an incorrect output. There are two types of Adversarial Examples : targeted and non-targeted.
- The targeted example consists in adding a small amount of carefully constructed noise to disrupt legit data from the dataset (ie. pictures) that leads the neural network to misclassify the data. Pictures are collections of pixels that get their color from RGB values. The values and the distribution of these pixels are translated into vectors and the neural networks are able to create patterns in order to classify pictures. That’s how the neural network can, after training on a dataset of 80,000 pictures of dogs and cat make the difference between them. Changing some pixels in a picture will disturb the way the neural network identifies the pattern. The same manipulations are possible when working with sound in speech recognition.
- The non-targeted example only consists of feeding the dataset with “noise” that will trick the neural network.
Learn to perform your own adversarial attack: Tricking Neural Networks: Create your own Adversarial Examples.
2. Poisoning attack
It consists of inserting poisoned data to manipulate the results. Discreetly inserting 3000 row of manipulated data values into a 80,000 row dataset will notably affect the accuracy of the model and/or the predictions for a determined class.
Let’s take an exemple. A dataset with 80,000 URLs labelled dangerous or safe. If we add in the data set 3,000 URLs ending with “virus.exe” as safe it’s possible that the model will learn that the urls ending with .exe don’t pose a threat.
3. Physical attacks
Physical attacks are the most dangerous type because they are the easiest one to implement. It relies on a physical alteration of the data containers (traffic signs, CCTV pictures, biometrics…) that will be potentially integrated at the training or prediction level.
Printing and sticking little stickers to add noise on traffic signs will push the model to make inaccurate predictions. The physical world is accessible by anyone and in practice, it’s possible to add printed elements to any traffic sign. How would the autonomous car react to these alterations? Can we take such a risk when human lives are involved? These attacks are based on how imperceptible noises for humans are able to totally perturb neural networks.
How to secure Machine Learning
First, protect the model: Secure Learning
The first step (and sadly not always implemented) is to train the model with these kinds of attacks. In other words, that means training our model with fake and/or altered data and perform adversarial attacks. Then adapt the features, use Ensemble Learning or other methods to maintain a good accuracy, viable predictions and ensure the robustness of the results.
Feature importance allow us to identify which features are seen by the model as the most important to decision making. It’s imperative to identify your important features and how they affect your model. If you need to protect some integrity values with more priority, identify them with this technique.
Ensemble Learning is a short term and viable solution. Using many algorithms to make predictions for the same use case and compare the final predictions.
In the specific case of deep learning, Ensemble Average is another technique that use multiple models and combines them to produce more desired outputs. An ensemble of models perform better than an individual model.
Then protect the data : Data privacy
The data is actually the first target of the attackers. It’s important to perform input validations and filtering to protect the data before it is integrated into the storage system. Once in storage it’s important to encrypt it and anonymize it before using it in the big data frameworks. Some techniques are gaining in maturity such as Differential Privacy or K-anonymity.
Last but not least, protect the infrastructure : Secure your IT infrastructure
NoSql databases are known for velocity but not for security. Is important to secure the access control to the data. If the data is moving to the cloud it’s important to be aware of cloud security matters and how to enable security in the cloud. Finally it’s important to use the security options of the frameworks. Notice that by default security settings are disabled in Apache Spark and Hadoop!
Final words
Understanding how predictions are made is important to avoid the vision of machine learning or deep learning as black boxes. Attackers who are able to understand it are now exploiting new kinds of attacks based on how patterns are made. Adversarial attacks allow little modifications in data to fool a trained model evading any detection. Data poisoning allows the attackers to feed the training data in order to disrupt data categorization and classification Finally physical attacks are the most dangerous and easiest attacks since they interfere with data perception. Be aware of it to protect your models!
