{"id":2615,"date":"2017-08-27T21:04:05","date_gmt":"2017-08-27T19:04:05","guid":{"rendered":"https:\/\/blog.mi.hdm-stuttgart.de\/?p=2615"},"modified":"2023-08-06T21:50:31","modified_gmt":"2023-08-06T19:50:31","slug":"fooling-the-intelligence","status":"publish","type":"post","link":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/2017\/08\/27\/fooling-the-intelligence\/","title":{"rendered":"FOOLING THE INTELLIGENCE"},"content":{"rendered":"

Adversarial machine learning and its dangers<\/h2>\n

The world is led by machines, humans are subjected to the robot\u2019s rule. Omniscient computer systems hold the control of the world. The newest technology has outpaced human knowledge, while the mankind is powerless in the face of the stronger, faster, better and almighty cyborgs.<\/em><\/p>\n

Such dystopian visions of the future often come to mind when reading or hearing the latest news about current advances in the field of artificial intelligence. A lot of Sci-Fi movies and literature take up this issue and show what might happen if the systems become more intelligent than humans and develop their own mind. Even the CEO of SpaceX, Tesla and Neuralink, Elon Musk, who is known for his innovative mindset, has a critical opinion towards future progress in artificial intelligence:<\/p>\n

If I were to guess what our biggest existential threat is, it\u2019s probably that. So we need to be very careful with the artificial intelligence. […] With artificial intelligence we are summoning the demon.<\/p>\n

Elon Musk<\/p><\/blockquote>\n


\nFor this reason he started the non-profit organization OpenAI<\/i><\/a> in 2015 aiming at researching and developing friendly artificial intelligence.<\/i> But also other famous scientists and researchers have doubts that the current hype regarding AI and Deep Learning yields only benefits for the people. Stephen Hawking claims, for example, that<\/p>\n

the rise of powerful AI will either be the best or the worst thing ever to happen to humanity. We do not yet know which. The research done by this centre is crucial to the future of our civilisation and of our species.<\/p>\n

Stephen Hawking<\/p><\/blockquote>\n

 <\/p>\n

While these doomsday scenarios are targeted at a potential evolution of artificial intelligence in decades, if not centuries, there are already posing serious dangers and risks from such systems these days. In times of increasingly self-driving vehicles, personal assistants, recommender systems etc., where more and more task are taken over by computers, people rely on them and trust them. Consumers use artificial intelligence because it simplifies their daily lives, it offers convenience and helps them in certain situations. Companies use and develop intelligent systems because they provide multiple benefits compared to human labour: They are cheaper, have more capacity and are more efficient. But above all, we think that these algorithms are free of human errors: They don\u2019t make errors due to fatigue or inattentiveness. They don\u2019t make errors due to miscalculations or misjudgement. They behave like they are told to behave. And this is exactly the reason why the so-called adversarial machine learning<\/b> is such a powerful opportunity that enables attackers to specifically manipulate these systems.<\/p>\n

 <\/p>\n

What is adversarial machine learning?<\/h3>\n

First of all, what is machine learning? Machine learning is a subfield of artificial intelligence, although the terms are often used interchangeably. It can be used to learn from input data in order to make predictions. The heart of each ML application is its underlying model, that defines the architecture of the algorithm and all of its parameters. It is important to mention that machine learning is not tied to a specific algorithm, for example not all ML applications make use of deep neural networks, although this type of algorithm is becoming more and more popular.<\/p>\n

There are two ways of machine learning that you can use, based on the characteristics of the input data and on what you want to learn: With supervised learning classification (deciding between two or more classes, e.g. \u201cIs the object in the image a bus, a car or a pedestrian?\u201d) and regression (learning a function, e.g. \u201cHow long will my journey take?\u201d) tasks can be solved, but they require labeled training data. This means that for each data instance, for example some image, the required target output has to be specified, e.g. \u201cthe image contains a car and pedestrian, but no bus\u201d. Unsupervised learning on the other hand does not require labeled data, which can save a lot of time during data preparation, but the tasks are limited to cluster analysis like finding similarities, which is heavily used in recommender systems.<\/p>\n

In order to get high-quality predictions from the algorithm, one has to train the model first. During the training phase, you feed lots of data into the algorithm, which computes an output based on the current model. In the case of supervised learning, the computed output is compared with the given target output. Based on the deviation, the model parameters are updated (this is basically the \u201clearning\u201d) in order to minimize the error. If the performance of the model is satisfying, it can be leveraged in production in order to predict, classify, cluster or whatever you want to do with your data.<\/p>\n

 <\/p>\n

Now that we have a principal understanding of how machine learning is working, we can get a better understanding of how adversarial machine learning is working. Generally speaking, inputs of machine learning systems are manipulated in a way that the output of the model becomes any desired result with high confidence. By applying smallest perturbation to the inputs, the model can be completely tricked. As a result, attackers can enforce the machine learning application to output any result they want under certain circumstances.<\/p>\n

This opens the door for various types of attacks that are not limited to a specific type of application, technology, or algorithm. But how does it look like in practice?<\/p>\n

 <\/p>\n

Adversarial machine learning in practice<\/h3>\n

To give an example of what adversarial machine learning is capable of, we can take a closer look at fooling image recognition.<\/p>\n

Some years ago, image recognition was done by applying traditional techniques, which includes choosing the suitable filters, applying edge and keypoint detection and clustering the visual words. Although this method worked well for years, it had some drawbacks: First of all, it depends on the specific application; you had to choose the correct setup, filters and other components tailored towards the individual application. Furthermore, the accuracy was acceptable, but not extraordinary. But from the 2010s on, a new technique was developed, strongly outpacing former traditional methods: Image recognition based on deep learning. They simplified and generalized the whole process while setting new benchmarks in the field of image recognition.<\/p>\n

Although their accuracy is above any doubt, they are prone to adversarial machine learning attacks. In this context, the attacker manipulates an image in such a manner that the classification algorithm misclassifies the content of the image in favor of the attacker\u2019s desired output.<\/p>\n

Researchers presented two types of attacks, both fooling the algorithm:<\/p>\n

The first approach is based on a \u201cnormal\u201d image, meaning that the image contains a human-recognizable object or scene. Now the attacker adds a tiny perturbation to the image, which is hardly perceptible by humans, if at all. However, the perturbation is not some random white-noise signal, but results from a targeted calculation. In the figure below, you can see an example:<\/p>\n

\"\"<\/a>
On the left the original image is depicted. To its right you can see the perturbation. These two combined result in the third image. Notice that the manipulation is not perceptible<\/figcaption><\/figure>\n

On the left side, the original image containing a panda is depicted. If you would feed this image in the machine learning algorithm used for this example, it would classify the content with a confidence of 57.7% to be a panda. But instead of classifying the output of that picture, the attacker adds a perturbation to it, pictured in the center. Note that this noisy image is multiplied with a factor of 0.007, so it is actually not visible for human eyes. The combination of the original image and the perturbation is visible on the right: It looks exactly like the original panda image, even if you know that it contains the perturbation, you cannot recognize it. For humans, the object in the manipulated image is still clearly a panda, but the algorithm is almost completely convinced (99.3% confidence) that it contains a gibbon. But this is no coincidence: You can basically take any picture, add the correct perturbation to it and get the desired output classification.<\/p>\n

 <\/p>\n

\"\"
Original images are shown on the left side, the distortions in the center are added. Each of those images is recognized as ostrich.<\/figcaption><\/figure>\n

As you can see in the figure above, the algorithm classifies each of these six different images as ostrich<\/i>, although they are containing something completely different.<\/p>\n

 <\/p>\n

The second approach starts from scratch, without depending on an existing image. A completely random-looking image is generated, looking like a white-noise signal, but is actually recognized with high confidence by the algorithm. How does this look like?<\/p>\n

\"\"<\/a>
The images are recognized to be digits between 0-9 with 99.99% confidence.<\/figcaption><\/figure>\n

In this example, you can see several newly generated images. For humans, it looks more like some kind of  QR code or something, but the algorithm can recognize digits in these images. For example, the images in the first column are classified to contain the digit \u201c0\u201d.  For us it\u2019s impossible to see, but the algorithm yields a confidence score of 99.99%.<\/p>\n

 <\/p>\n

How does it work?<\/h3>\n

As described in the next section,<\/a> adversarial machine learning is not limited to a specific application, e.g. image recognition, but is universally practicable in the field of machine learning. For that reason, it is not possible to explain how all of these attacks exactly work, because it depends on what you want to achieve. Therefore, a short summary is given of how the scientists managed it to produce and manipulate images like in the examples above in order to fool the image classification algorithm:<\/p>\n

 <\/p>\n

In general, the image generation and manipulation process is based on a local search algorithm, known as genetic algorithm.<\/i> It is derived from the biological theory of evolution by Charles Darwin, transferring the concept of \u201csurvival of the fittest\u201d to optimization problems.<\/p>\n

Let\u2019s call the the digit-generation example from above to mind: How can we produce such an image, that looks like white noise, but will be classified by a trained machine learning algorithm to contain any desired digit between 0 to 9 with a confidence of over 99%?<\/p>\n

Imagine we want to craft an image which is classified as the digit \u201c6\u201d. At first, we have to produce an initial population<\/i>. The population is the set of all relevant states, referred to as individuals<\/i>. This means in practice that we have to generate a set of independent, random images. Each image is an individual, all images together form the population.<\/p>\n

The whole process consists of four steps:<\/p>\n

    \n
  1. Evaluation<\/strong>: Each individual is evaluated with a fitness function. The fitness function indicates how good the respective individual is with regard to subjective. In concrete terms, each previously randomly generated image is classified with the machine learning algorithm. The fitness value would be the confidence score of the algorithm for the digit \u201c6\u201d. Since the images are white-noise signals, the fitness evaluation value should be quite low initially for each individual.<\/li>\n
  2. Selection<\/strong>: We select a subset from the complete population for the next step. However, these individuals are not randomly chosen. Since we want to build a better generation, we state that fitter individuals are more likely to be chosen. This means that we select those images, that are recognized to be a \u201c6\u201d with higher confidence, with higher likelihood.<\/li>\n
  3. Crossover<\/strong>: We can only produce a new population, if we modify the selected individuals. Therefore we choose pairs of parent individuals<\/i> which produce a child individual <\/i>by crossover. Regarding our example crossover means that we choose a random crossover point (selecting a random pixel) and then taking all the pixel values before that crossover point from the first parent individual<\/i> and taking all the pixel values after that crossover point from the second individual, combining them into the new child individual<\/i>. By this means we hope to chose the best values from the parents and therefore produce an even stronger child.<\/li>\n
  4. Mutation<\/strong>: With low probability, we modify individual pixel values in the child instance.<\/li>\n<\/ol>\n

    The child individuals replace the weakest individuals from the current population. Now we can evaluate the population again in order to determine the fitness values of all individuals. If the fitness value of one or more individuals is high enough, which means that the ML algorithm is confident enough to recognize a \u201c6\u201d, we can terminate the genetic algorithm. Otherwise, we start the next iteration, constituting the next generation.<\/p>\n

    For the example above, the researchers needed less than 50 generations to produce images with a confidence score of \u2265 99.99% and only 200 iterations to achieve a median confidence of 99%.<\/p>\n

    Using this approach, we don\u2019t need access to the underlying model of the classification algorithm, we just need the output with the confidence score.<\/p>\n

     <\/p>\n

    What else is possible?<\/h3>\n

    Adversarial machine learning is not limited to image recognition tasks. There are many more applications that are vulnerable to these attacks:<\/p>\n

    Researchers showed how to easily fool Google\u2019s new Cloud Video Intelligence API.<\/a> This API offers the functionality to analyse videos and make them searchable like textual content. The main purpose of the Cloud Video Intelligence API is to recognize the the content of videos and provide the associated labels.<\/p>\n

    How could we adversarially use this API? Remember that the principle of adversarial machine learning consists of manipulating inputs of machine learning systems in a way that the output of the model becomes any desired result with high confidence.<\/p>\n

    \"The<\/a><\/p>\n

    Researchers took an image different from the actual video content and inserted it periodically and at a very low rate into the video. The original video contained for example animals like in the figure above. Now they inserted an image of a car, a building, a food plate or a laptop every two seconds for one frame. Although this is a very low rate (one frame every two seconds for a video of 25 fps corresponds to a insertion rate of 2%), the API is deceived into returning only the video labels which are related to the inserted image, but did not return any labels related to the original video.<\/p>\n

     <\/p>\n

    Another team showed how to attack voice recognition systems. Personal assistants are widely and extensively used nowadays, but they are also prone to adversarial attacks. Similar to the image generation example above, they generated voice commands in order to fool systems like the Google Assistant or Apple\u2019s Siri. For their whitebox approach they had complete knowledge of the model, its parameters and architecture. Hence the attack worked out very well: The voice commands were not recognizable by humans, but completely understandable by the system. This way an attacker could attack any voice recognition system without being noticed by the user. However, in reality it\u2019s not common to have complete knowledge about the system\u2019s internals. Therefore they performed another experiment, this time treating the machine learning model as blackbox. Even though the voice commands generated now were easier understandable for humans, they were also harder to process by the voice recognition systems.<\/p>\n