Detecting Malware in Mac OS X Environments

Detecting Malware in Mac OS X Environments

Mac OS X Malware Over the last few years, we’ve seen a number of families of malware written specifically for the Mac OS X operating system. There was Flashback, and more recently the KeRanger ransomware. We’ve also seen more targeted attacks where Mac OS X malware was written to steal very specific information from a small number of high-value targets. Indeed, we are starting to witness the same range of Mac malware that we see in Windows environments, from blunt tools like ransomware that’s designed to attack as many people as possible, to very targeted, information-stealing malware going after a specific group of users.

Mac OS X Malware Rapidly Gaining Sophistication

When we look at the overall picture, we see that, despite the growing scope of attacks targeting Mac OS X systems, there hasn’t been a dramatic increase in volume as most malware authors still are targeting Windows. But we do see an increase in sophistication. This evolution is reminiscent of the Windows environment during the early 2000s—but there is a difference. Today, authors writing malicious code for the Mac environment can really leverage and learn from the long Windows history. The result is that Mac malware is evolving at a rapidly accelerated pace—much faster than it did on the Windows platform.   

Most Mac OS X Sandboxes are Ineffective

While there is significantly less malware in Mac environments than in Windows, organizations still can’t afford to ignore it. As with Windows, sandboxes and dynamic analysis is usually the best method to detect and defeat advanced malware that’s targeting Mac OS X. However, dynamic analysis is very difficult to do effectively in a Mac OS X environment, and organizations need to be careful when selecting malware detection products.

For instance, when we look at typical dynamic analysis tools or sandboxes on Mac OS X platforms, we find that many of them rely on DTrace as the underlying technology. Unfortunately, DTrace is not a security product. It’s an instrumentation tool that allows one to debug programs by inserting collection probes into the program that gather information about the program and what it is doing.

The problem is that DTrace provides very limited visibility and is unable to observe a number of important behaviors. For example, DTrace can only look at parameters and system calls. It does not look at any individual instructions or kernel code. DTrace also inserts probes, system calls, and other artifacts into the program, which enables malware to easily detect its presence and take evasive action.

This DTrace approach is similar to the sandboxes used on Windows platforms ten years ago. They only checked system calls and couldn’t see CPU instructions. Nor could they change the return value of system calls. While this worked on old Windows malware, and it might work on old Mac OS X malware, today’s sophisticated Mac malware employs techniques learned from Windows malware over many years—techniques that successfully evade and fool most sandboxes.

To Detect Mac OS X Malware—Look for These Features

When selecting a sandbox for the Mac OS X environment, organizations need to determine if the product is simply using DTrace, or if it uses more advanced technologies that provide deep visibility.

We recommend that decision-makers ask the following types of questions:

  • Is the product able to see all of the instructions that the CPU executes?
  • Does it provide full memory access? Can it dynamically inspect the memory?
  • Can the system see all of the objects that are created?
  • Is it capable of viewing the call stack, including all of the system calls that have happened?
  • Can it determine which functions are responsible for actually invoking a specific system call?

To get a complete picture of what the malware is doing, it’s important for the product to do all of the above. Unfortunately, DTrace is not able to provide any of these important capabilities, and products that rely on it are ineffective.

To detect today’s advanced Mac OS X malware, enterprises need to deploy a dynamic analysis solution that fully emulates the operating system, including the CPU. It’s also important that the product is capable of Deep Content Inspection so it has full visibility of everything that occurs within the system.

Additionally, it’s crucial to select a solution that is as invisible to the malware as possible. DTrace and VM based sandboxes introduce artifacts that the malware can detect. See Lastline’s blog Evasive Malware Detects and Defeats Virtual Machine Analysis to learn more about why this is important.

By understanding how malware attacks Mac OS X environments, and by carefully selecting the right malware detection system, organizations can effectively protect their Mac OS X assets as well as their Windows-based equipment.

Bert Rankin

Bert Rankin

Bert Rankin has been leading technology innovation for over 25 years including over 5 years in security solutions that prevent cybercrime. He is a frequent blogger and is often quoted in security-related articles. Bert earned his BA from Harvard University and an MBA at Stanford University.
Bert Rankin

Latest posts by Bert Rankin (see all)