{"id":10442,"date":"2020-08-19T16:00:00","date_gmt":"2020-08-19T14:00:00","guid":{"rendered":"https:\/\/blog.mi.hdm-stuttgart.de\/?p=10442"},"modified":"2023-08-06T21:45:42","modified_gmt":"2023-08-06T19:45:42","slug":"adversarial-attacks","status":"publish","type":"post","link":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/2020\/08\/19\/adversarial-attacks\/","title":{"rendered":"The Dark Side of AI – Part 2: Adversarial Attacks"},"content":{"rendered":"\n
\"\"\/<\/figure>\n\n\n\n

Part 1<\/strong><\/a> of this series discussed how AI technology can be used for good or evil. But what if the AI itself becomes an attack vector? Could an attacker use my models against me? Also, what\u2019s the worst that could happen? Welcome to the domain of adversarial AI!<\/p>\n\n\n\n\n\n\n\n

Introduction<\/h1>\n\n\n\n

If you\u2019re reading technology-focused magazines or blogs, you might have stumbled upon the term adversarial examples<\/em> before. In recent years (mostly since 2013), there has been a large number of publications on them and their possible impact on AI technology. According to the OpenAI blog<\/a>:<\/p>\n\n\n\n

\n

Adversarial examples are inputs to machine learning models that an attacker has intentionally designed to cause the model to make a mistake; they\u2019re like optical illusions for machines.<\/p>\n<\/blockquote>\n\n\n\n

Some Fun Examples<\/h2>\n\n\n\n
\"\"
An untargeted attack using the Fast Gradient Sign Method<\/em> (FGSM)
Source: Explaining and Harnessing Adversarial Examples, Goodfellow et al., 2015<\/a><\/figcaption><\/figure>\n\n\n\n

The example above shows one of the earlier attacks. In short, an attacker generates some very specific noise, which turns a regular image into one that is classified incorrectly. This noise is so small that it is invisible to the human eye.<\/p>\n\n\n\n

\"\"
Source: Adversarial Patch, Brown et al., 2018<\/a><\/figcaption><\/figure>\n\n\n\n

Adversarial attacks can also be applied in the physical world. In this example, a so-called adversarial patch<\/em> is created. If the patch is present, the attacked classifier detects the image as a toaster, instead of a banana. Such a patch is visible to the human eye, but there are other ways to hide or disguise it. For example, it might be shaped like a pair of glasses. Patches on traffic signs may likewise be designed to look similar to graffiti. <\/p>\n\n\n\n

Some Not so Fun Examples<\/h2>\n\n\n\n
\"\"
Source: Robust Physical-World Attacks on Deep Learning Visual Classification, Eykholt et al., 2018<\/a><\/figcaption><\/figure>\n\n\n\n

Adversarial attacks are most prevalent in the domain of computer vision<\/em>, the majority being aimed at misclassification. There are many attack scenarios, from spoofing facial recognition systems to fooling autonomous vehicles. In this example, adversarial examples cause a classifier to mistake a \u201cStop\u201d sign for a \u201cSpeed limit 45\u201d sign. Notably, this attack is robust against changes in scale, rotation and shift of the adversarial patch, which means that it can fool a driving car.<\/p>\n\n\n\n

\"\"
Source: Experimental Security Research of Tesla Autopilot, Tencent Keen Security Lab, 2019<\/a><\/figcaption><\/figure>\n\n\n\n

A security report on \u201cTesla Autopilot\u201d (HW 2.5) from March 2019 found that the platform can be attacked with adversarial examples. The researchers placed adversarial patches on the road to cause the vehicle to detect and follow lanes which weren\u2019t there. The fact that this flaw is (or was) present in a commercial product, firmly places this example in the \u201cnot so fun\u201d section.<\/p>\n\n\n\n

\"\"
Performance of the BERT language model on various text classification tasks, before and after the adversarial attack. 
Source: BERT-ATTACK: Adversarial Attack Against BERT Using BERT, Li et al., 2020<\/a><\/figcaption><\/figure>\n\n\n\n

Adversarial attacks are by no means restricted to the domain of computer vision. Here is an example from the domain of Natural Language Processing<\/em>. It shows a successful adversarial attack on the powerful BERT language model, using \u201cBERT-Attack\u201d. In this example, one original task of the model was to classify Yelp reviews as \u201cpositive\u201d or \u201cnegative\u201d – which BERT did with an accuracy of 95.6 percent. After the attack, BERT\u2019s accuracy was only 5.1 percent – much worse than random. To achieve this, BERT-Attack was able to find the exact words that needed to be replaced with synonyms, to fool the language model.<\/p>\n\n\n\n

Not Fun at All<\/h2>\n\n\n\n

Among adversarial attacks, physical attacks against computer vision are arguably one of the more immediate threats. This is especially true in areas where AI is used in systems with safety- or security relevance, such as autonomous driving or others:<\/p>\n\n\n\n

\n

What you don\u2019t want is your enemy putting an adversarial image on top of a hospital so that you strike that hospital.<\/p>\nJeff Clune, University of Wyoming, on The Verge<\/a><\/cite><\/blockquote>\n\n\n\n

Purely digital attacks might not have the same immediate impact, but they also have the potential to cause significant trouble as well. There are many more domains that I didn\u2019t touch on, such as cybersecurity, sentiment analysis, movement prediction etc., and there are some quite uncomfortable scenarios imaginable if one is only creative enough. Adversarial attacks are clearly more than just a theoretical threat. While the exploration of adversarial attack mechanisms has been a rather academic exercise in the past, there is now a growing awareness of them and their potential impact. <\/p>\n\n\n\n

\n\n\n\n

Threat Model<\/h1>\n\n\n\n

Technically, adversarial attacks have been around for almost two decades now, since long before the recent success of deep learning. Early adversarial attacks often involved manual input. For example, a spammer might try to design their spam emails to appear innocuous to spam filters. To that end they would guess \u201cbad words\u201d that are likely to trigger a filter rule and replace them with others. Modern attacks are largely automated and employ machine learning methods themselves.
Most adversarial attacks are designed against some classification mechanism, for example machine learning models. Such classifiers receive some input and react in a certain way, e.g. by returning an output class. An attacker tries to change this behavior in some way the mechanism was not designed.
While adversarial attacks can target any artificial intelligence and machine learning model, today most research into adversarial attacks concerns (deep) neural networks.
There are several factors to consider when modeling an adversarial attack threat:  The attacker\u2019s assumed control over the target mechanism, their knowledge about it and finally their goal.<\/p>\n\n\n\n

Attack Strategy<\/h2>\n\n\n\n

Adversarial attacks come in two categories: Poisoning and Evasion attacks. The key difference lies in the attacker\u2019s influence on the target mechanism:<\/p>\n\n\n\n

Poisoning Attacks<\/h3>\n\n\n\n

In a poisoning attack, an attacker has access to the creation process itself. In machine learning models this may happen if an attacker can somehow manipulate the training data. Take our spam example from earlier: An \u201cintelligent\u201d spam filter learns which emails are spam and which are not over time. If an attacker can insert spam emails labeled as \u201cnot spam\u201d during this training process, the spam filter will learn to ignore them and won\u2019t filter such spam emails in the future.
Of course, this scenario only works if the attacker has at some point access to the training process. Due to this restriction, poisoning attacks are somewhat less prevalent than evasion attacks.<\/p>\n\n\n\n

Evasion Attacks<\/h3>\n\n\n\n

Evasion attacks on the other hand assume that the target mechanism is already fully developed and immutable. It only assumes that an attacker can send some input to it.
The basic idea is as follows: An attacker causes some minimal change in the input data that they send to the target. This change is either imperceptible to humans, or seems so innocuous that it is simply ignored. The machine learning model on the other hand is affected by this change and produces a different result. Such manipulated inputs are called adversarial examples<\/em>. An Attacker now has two options: Either the adversarial examples simply throw off the AI model enough as to make it unsuitable for its task (untargeted attack), or they force the model to behave in a specific way designed by the attacker (targeted attack).<\/p>\n\n\n\n

In the field of deep learning and especially computer vision, evasion attacks are much more prevalent than poisoning attacks, due not in small part to their applicability in the real world. For that reason, this article will focus on evasion attacks.<\/p>\n\n\n\n

Adversarial Example Crafting<\/h2>\n\n\n\n
\"\"
The basic crafting process for adversarial examples<\/figcaption><\/figure>\n\n\n\n

The process of crafting adversarial examples usually requires knowledge about the target\u2019s internal decision boundaries<\/em>. This information is necessary, since an attacker will try to change the inputs in such a way that the target comes to a different decision, e.g. to make it predict another class than it normally would.<\/p>\n\n\n\n

Decision Boundaries<\/h3>\n\n\n\n
\"\"
A sample x<\/em><\/strong> with y = 4<\/em><\/strong> is turned into an adversarial sample x\u2019<\/strong> that is classified as a 3<\/em><\/strong>.
Source: Adversarial AI Survey Paper, Xu et al., 2019<\/a><\/figcaption><\/figure>\n\n\n\n

A neural network classifier learns decision boundaries, that allow it to classify an input sample x<\/em><\/strong> as its corresponding class y<\/em><\/strong>. These decision boundaries are represented by the network\u2019s internal parameters. If an attacker knows where the decision boundaries are, they can change the input example x<\/em><\/strong> in such a way that \u201cpushes\u201d it over the boundary. <\/p>\n\n\n\n

Perturbation and Robustness<\/h3>\n\n\n\n
\"\"
Source: Explaining and Harnessing Adversarial Examples, Goodfellow et al., 2015<\/a><\/figcaption><\/figure>\n\n\n\n

In some attack methods like FGSM, an adversarial example x\u2019<\/em><\/strong> is created from a regular input example by adding some perturbation \u03b7<\/em><\/strong>. This perturbation is also often called adversarial noise<\/em>. The noise is then added on top of the original input x<\/em><\/strong>. Usually, an attacker tries to make this noise as imperceptible as possible, while still having an adverse effect on the target\u2019s performance.
Conversely, a model is more robust against adversarial attacks, the more perturbation is necessary to fool it. If the minimum necessary perturbation is large enough, the attack may become obvious to the naked eye and the attack is easily detected.<\/p>\n\n\n\n

Adversarial Patch<\/h3>\n\n\n\n

Other attacks, such as most physical attacks, rely on adversarial patches<\/em>. In such cases, the perturbation isn\u2019t hidden but constrained to some patch within the input image. Here the attacker\u2019s goal is to make the patch as small and unassuming as possible, while retaining its adverse effect.<\/p>\n\n\n\n

Transferability<\/h3>\n\n\n\n

Transferability refers to a property of many adversarial attacks, that allows them to be successful on more than one specific target. This concept can even work across different architectures. Only the domain of the models must be the same.<\/p>\n\n\n\n

\"\"
Transferability example: If an attacker can create a substitute model F<\/em><\/strong> that is close enough to the black-box O<\/em><\/strong>, they can then use F<\/em><\/strong> to create adversarial noise that is effective on O<\/em><\/strong>.<\/figcaption><\/figure>\n\n\n\n

Attacker\u2019s Knowledge<\/h2>\n\n\n\n

How successful an attack can be largely depends on the information available to the attacker. One common approach to model such information is to consider black-box, gray-box and white-box attacks:<\/p>\n\n\n\n

White-box<\/h3>\n\n\n\n

The attacker knows everything about the target model. They have access to the training data, the neural network graph and the trained parameters. The attacker can essentially use the model against itself.<\/p>\n\n\n\n

Gray-box<\/h3>\n\n\n\n

The attacker only has access to parts of the model, such as the architecture or the classifier\u2019s confidence values. This information is then used to recreate the target model as close to the original as possible.<\/p>\n\n\n\n

Black-box<\/h3>\n\n\n\n

The attacker has no knowledge about the target model at all. However, they can<\/strong> make educated guesses. There are a number of attack strategies, that are each able to extract valuable information from a black-box model. Most of these strategies assume that the attacker correctly guesses the target model’s domain and that they can communicate with the black-box, e.g. through an API. They can then send queries to the black-box model, to learn how it classifies inputs. The goal is again to acquire enough information to recreate a close approximation of the target model.<\/p>\n\n\n\n

\"\"
Substitute model example:
1. The attacker finds input data that fits the domain of the black-box (oracle<\/em>).
2. The attacker sends the input data to the oracle, which returns predictions.
3. The attacker uses the input data and the labels predicted by the oracle to train a substitute model.<\/figcaption><\/figure>\n\n\n\n
\n\n\n\n

Defending Against Adversarial AI<\/h1>\n\n\n\n

So what can you do about adversarial attacks? Can they be averted or mitigated? Well, yes and no. Here are some security practices I found – you\u2019ll likely recognize some of the more general ones. Let\u2019s explore what works, what doesn\u2019t, and why.<\/p>\n\n\n\n

Gradient Masking<\/h2>\n\n\n\n

Since many white-box attacks require access to the gradients of a neural network, one way to prevent them is to simply hide their parameters. In practice this boils down to: Don\u2019t put your trained deep neural network up on GitHub for everyone to see, if you want to use it in production. This is what many autonomous car manufacturers do, they keep their network graphs and parameters closed source, which in theory makes it harder to attack them. If that sounds like \u201csecurity through obscurity\u201d to you: That\u2019s because it is. 
The problem with this \u201cdefense\u201d is that many attacks are transferable. An attack on one machine learning model may not work perfectly on another, but just well enough. In addition, there are black-box attacks out there that don\u2019t require full access to the original network parameters at all.<\/p>\n\n\n\n

Verdict: Gradient Masking is better than nothing, but it doesn\u2019t really work as a defense strategy on its own, other than giving you a false sense of security.<\/p>\n\n\n\n

Rate Limiting<\/h2>\n\n\n\n

Say you employ gradient masking and put your neural network behind an API. A user sends some input (e.g. an image), the API returns some output (e.g. a classification). An attacker can only observe the black-box from outside. Without rate limiting, you\u2019re now inviting a substitute model black-box attack. An attacker can send as many queries against the API as they like, until they have labeled a complete dataset. They then train their own substitute model that has the same internal decision boundaries as the original, and attack from there.
Rate limiting can prevent some substitute model attacks, because sending millions of queries would simply take forever. Depending on your use case, you can even use CAPTCHAs or charge money for every API use.
On the flip side, this still doesn\u2019t protect against some of the more advanced black-box attacks. Some of these are especially designed to minimize the number of queries necessary to create a good substitute.<\/p>\n\n\n\n

Verdict: It\u2019s a good idea to use rate limiting, but it doesn\u2019t offer consistent protection either and may significantly impact user experience.<\/p>\n\n\n\n

Input Validation<\/h2>\n\n\n\n

Given that most recent adversarial attack methods use machine learning techniques, it only makes sense to use machine learning techniques to defend against them. To neutralize adversarial examples before they reach their target, one can use a secondary detector model<\/em>. Such a model is either trained to flag and filter adversarial examples, or to turn them into regular, harmless examples. The latter can be done by training a denoising autoencoder<\/em> to remove any adversarial noise from its input examples and pass only the \u201ccleaned\u201d inputs to the primary model. The advantage of input validation is that the actual model (the one with the business logic) can stay as-is. The downside here is that you\u2019ll have to train and maintain yet another model. The second disadvantage is the increase in latency, because every input goes through two neural networks now. Finally, there\u2019s no guarantee that a detector itself cannot be fooled.<\/p>\n\n\n\n

Verdict: Input validation with a detector model can actually be a reasonable protection against known adversarial attacks, but it comes with high recurring costs.<\/p>\n\n\n\n

Adversarial Training<\/h2>\n\n\n\n

Finally, one can try and make the model itself more robust to attacks. Remember that the main problem with adversarial examples is that the target model has never seen that specific type of noise during training. Adversarial training means that adversarial examples are added to the training data, so that some specific attacks become useless against the model. Unfortunately, this can make the model larger and more complex, as it must learn more information. This drives up training cost for no immediate performance gain. Finally, adversarial training might theoretically open the door for poisoning attacks. If detected adversarial examples are used in training, an attacker might thus manipulate the training process itself.<\/p>\n\n\n\n

Verdict: This is a valid defense method, but it comes with high recurring costs, because it must be done again once new attacks are discovered. The space of possible adversarial attacks is very large, so it is intractable to defend against all of them. This means that adversarial training is only a crutch, but cannot fix the deeper problem.<\/p>\n\n\n\n

The Deeper Problem<\/h2>\n\n\n\n

The issue with the \u201csolutions\u201d above is that they are all only symptomatic fixes. The deeper problem lies within a general limitation of state-of-the-art machine learning, that has been documented as early as 2014:<\/p>\n\n\n\n

\n

These results suggest that classifiers based on modern machine learning techniques, even those that obtain excellent performance on the test set, are not learning the true underlying concepts that determine the correct output label. Instead, these algorithms have built a Potemkin village that works well on naturally occuring data, but is exposed as a fake when one visits points in space that do not have high probability in the data distribution. [\u2026]
We regard the knowledge of this flaw as an opportunity to fix it.<\/p>\nGoodfellow et al., Explaining and Harnessing Adversarial Examples<\/a><\/cite><\/blockquote>\n\n\n\n

In other words: Current machine learning models do not really understand the concept of what they see. All they learn is some shallow set of rules that happens to result in the expected behavior. As soon as some pattern appears which they haven\u2019t seen during training, their flaw will be revealed.
This means in practice that there is never a guarantee against adversarial attacks, because there are simply too many ways to create such patterns. Machine learning models cannot be made proactively robust against every possible attack, because that would require them to understand the true concept behind the data. That leaves us in a situation where:<\/p>\n\n\n\n

\n

The current state-of-the-art attacks will likely be neutralized by new defenses, and these defenses will subsequently be circumvented.<\/p>\nXu et al., Michigan State University, 2019 Survey Paper<\/a><\/cite><\/blockquote>\n\n\n\n

Some even call it an adversarial AI arms race<\/a>. A grim outlook? Perhaps, but it is at least something we can work with.<\/p>\n\n\n\n

General Strategy<\/h2>\n\n\n\n

A recent report published by Accenture Labs<\/a> features a three-step plan on how to prepare for adversarial threats. I\u2019ve compiled the items that I found most actionable, as well as a few of my own thoughts:<\/p>\n\n\n\n