How Cybercriminals are Attacking Machine Learning

How Cybercriminals are Attacking Machine Learning

Featured Image 2In recent years, machine learning has made tremendous strides in the fight against cybercrime. But it’s not foolproof, and criminals have developed techniques to undermine its effectiveness. In today’s adversarial environment, organizations must deploy technologies that are resilient to attacks against machine learning.

Machine Learning—Amazing, but not a Panacea

Machine learning (ML) is getting a lot of attention these days. Search engines that autocomplete, sophisticated Uber transportation scheduling, and recommendations from social sites and online storefronts are just a few of the daily events that ML technologies make possible.

Cybersecurity is another area where ML is having a big impact, providing many benefits. For instance, ML can help security analysts shorten response times, and dramatically improve accuracy when deciding if a security alert is an actual threat or just a benign anomaly. Many view ML as the primary answer to help save organizations from the severe shortage of skilled security professionals, and the best tool to protect companies from future data breaches.

However, as helpful as ML is, it is not a panacea when it comes to cybersecurity. ML still requires a fair amount of human intervention, and hackers have developed multiple methods to deceive and thwart traditional ML technologies.

Unfortunately, it’s not particularly difficult to fool ML into making a bad decision, especially in an adversarial environment. It’s one thing to have a successful ML outcome when operating in a setting where all the designers and data providers are pulling for its success. Executing a successful ML algorithm in a hostile environment full of malware that’s designed to fight back is altogether a different situation.

Machine Learning Ground Truths

Cybercrime continues to grow dramatically in sophistication. While it’s true that older basic hacking techniques and exploits are still widely used, leading cybercriminals quickly adopt each new innovation, including ML, to do their dirty work. Many have a deep understanding of the technology, and this enables them to design attacks that can evade ML-based malware detection systems.

A very common type of machine learning (called “supervised machine learning”) is dependent on a baseline of “ground truth” data. The technology uses this foundation to develop the models that support all decisions, by training with specific examples that have been previously labeled as malicious or benign. This is the foundation against which new instances are compared as part of the ML-supported analysis.

For example, in a very simplified illustration, imagine an ML system designed to detect if the animal in a particular image is a cat. Some of the ground truths identifying a cat are that it has two pointy ears, fur, paws, a tail, and whiskers. By comparing an image against these truths, the system can determine if the image is indeed that of a cat.

Ground Truth:filler insert

Machine Learning Cats Image 5

filler insert


filler insert

Defeating ML-Based Systemsfiller insert

Cybercriminals use three primary techniques to defeat ML-based security controls.

1) Polluting the Ground Truth

Criminals will attempt to pollute an ML system with content that corrupts cat characteristics image 1the ground truth data. If the ground truth data is either incorrect or altered, the system won’t be accurate, because it will literally learn the wrong thing.

For example, a hacker might attempt to loosen the details of what constitute the characteristics of a cat, with the goal of having the system recognize people dressed up as cats as cats. This could be achieved by adding the image above to the ground truth dataset, thereby polluting the ground truth data. Because this image of a person has fur, whiskers, pointy ears, and presumably a tail, it meets the system’s criteria for a cat and it incorrectly classifies it as such.

A subsequent result, therefore, could be:

Image 8The baseline of ground truth data found in ML-based malware and breach detection systems are very complex, and it’s difficult to get everything correct. For example, at Lastline, we took a number of benign executables and packed them using standard compression tools. We then submitted the packed executables to a number of antivirus tools. Although the programs were completely harmless, several of the malware detection systems classified them as malicious because these systems had incorrectly learned that all packed programs are dangerous.

Cybercriminals look for opportunities to influence or corrupt the ground truths that ML systems use so that something bad is identified as good.

2) Mimicry Attacks

image 10The second type of attack against ML-based security controls is a mimicry attack. This is where an attacker observes what the ML system has learned, and then modifies the malware so that it mimics a benign object. Using our above illustration about the system that identifies cats, in a mimicry attack the criminal knows that the system will recognize something with paws, pointy ears, and a tail as a cat, and therefore it creates an image of something that is not really a cat but mimics a cat by meeting the system’s specifications of a cat, such as the image at the right.

To understand how a cybercriminal might use a mimicry attack in the real world, consider the following example. An attacker learns that a system uses entropy associated with the various sections of an executable as a feature to detect malware. Then, the attacker pads specific section with data (e.g., zeroes) to change the entropy so that it will be similar to the entropy of a benign file. Of course, ML systems use many features to characterize a malware sample, but this process can be repeated until the right values are reached for all the features.

3) Stealing the ML Model

A third technique used by cybercriminals to undermine ML-based controls is to steal the actual ML model. In this case, the attacker reverse engineers the system or otherwise learns exactly how the ML algorithms work. Armed with this level of knowledge, malware authors know exactly what the security system is looking for, and how to avoid it.

While this level of attack requires a very determined and sophisticated adversary, researchers have already found a number of novel techniques used to steal ML models with just a few hundred queries (see for example the paper “Stealing Machine Learning Models via Prediction APIs”).

Fighting Back—ML Done Right

There are a number of things an enterprise can do to fight back against cybercriminals and their attempts to defeat ML-based security controls. First and foremost, it’s critical to select and deploy a system that has been specifically designed to operate in a hostile environment. To be effective in the world of cybercrime, ML tools must be attack-resilient and capable of multi-classification and clustering to detect the most sophisticated evasion techniques. Migrating an ML toolset designed for image detection or marketing to a security application will likely be met with defeat.

Data filtering techniques should also be present to guard against pollution of the ground truth data. By constantly revising, upgrading, and evolving the ground truth data, the system will automatically remove inaccuracies and stay up-to-date with respect to emerging attacks and new trends.

It’s also vital that the ML technology incorporates learning from the behaviors of a file as it executes, and not just from the appearance of the file. Returning to our cat detection analogy—while an attacker can create an image of a person that appears something like a cat, it’s another matter to make a person behave like a cat. Cats can easily jump higher than their body length and they can retract their claws—humans can’t do either behavior, and that won’t change. Likewise, malicious files elicit malicious behavior, and that won’t change. Effective ML systems must incorporate behavior into the equation. This will significantly reduce the possibility of a successful mimicry attack.

ML is an Important Arsenal in the Fight Against Cybercrime

ML is changing our world, including cyber. It’s true that it can’t fully operate on its own and needs an element of human intervention and oversight—and perhaps it always will. However, it’s also clear that ML provides many benefits and is here to stay. Although sophisticated cybercriminals have devised very clever attacks against the technology, when designed and built specifically for adversarial environments and combined with other tools and skilled human oversight, ML is an effective technique in the fight against cybercrime.

Giovanni Vigna

Giovanni Vigna

Giovanni Vigna is one of the founders and CTO of Lastline as well as a Professor in the Department of Computer Science at the University of California in Santa Barbara. His current research interests include malware analysis, web security, vulnerability assessment, and mobile phone security. He also edited a book on Security and Mobile Agents and authored one on Intrusion Correlation. He has been the Program Chair of the International Symposium on Recent Advances in Intrusion Detection (RAID 2003), of the ISOC Symposium on Network and Distributed Systems Security (NDSS 2009), and of the IEEE Symposium on Security and Privacy in 2011. He is known for organizing and running an inter-university Capture The Flag hacking contest, called iCTF, that every year involves dozens of institutions around the world. Giovanni Vigna received his M.S. with honors and Ph.D. from Politecnico di Milano, Italy, in 1994 and 1998, respectively. He is a member of IEEE and ACM.
Giovanni Vigna