Poll

On Windows 7 (or Vista) I use

Articles

Features of Modern Security Suites – Part 2

Published: 2012/09/14

The second part of the Features of Modern Security Suites series is devoted to Behavior Control features. Application behavior and its monitoring and controlling is a topic that is very close to us as it is also the primary focus of our public research as well as our commercial work for security software vendors. In this article we describe basic functionality of Behavior Control features and we enumerate common categories of application behavior that are being controlled by modern security suites.

Behavior Control

Also called: Proactive Protection, Active Virus Control, Intrusion Guard, Proactive Defense, Host-based Intrusion Prevention System (HIPS), Behavioral Shield

The Behavior Control component monitors actions of all applications in the system and blocks actions that threaten the security of the system or its users. It maintains a database of rules that determine which actions should be allowed or blocked for each application. The protection system takes control and stops the execution of the application when it is about to execute a potentially dangerous action. If there is a rule that determines the particular situation, it is used and the action is either blocked, in which case the execution flow is modified so that the action is not performed at all or its parameters is altered to ensure it is safe to execute it; or allowed, which means that the action is executed as if there was no protection. It may happen that there is no appropriate rule in the database of rules. In such cases, depending on the configuration of the behavior control component, either a user can be asked to make a decision, or the action can be automatically allowed or blocked, or a heuristic-based algorithm can be used to decide whether it is safe to allow the execution of the action in question.

Behavior Control does not scan the application files before they are executed and hence it is unable to detect malicious programs before they are stared. However, it can effectively block dangerous behavior and thus prevent damage and protect the system from both known and unknown malware.

It is the Behavior Control feature that our Proactive Security Challenge 64 (PSC64) focuses on. In this project we try to examine the quality and the level of security of the product's Behavior Control component. Other components such as Anti-virus Engine may be strongly connected to Behavior Control and may be necessary for its correct behavior, but the primary focus of our project is on the behavior blocking – the ability of the product to recognize and block malicious behavior even if the offending program is not known and can not be recognized prior its execution. We have created a set of tests called Security Software Testing Suite 64 (SSTS64). Each test within this suite implements different technique that can be misused by malware. The goal of the security product tested in PSC64 is to block the attacking techniques of the SSTS64 tests.

Operation Modes

Similarly to Firewall Policy which determines how decisions are made by the Program Control component, the Behavior Control feature can operate in several different modes. Most products do not provide separated configuration here and use Firewall Policy to define how the rules are applied by both the Firewall component and the Behavior Control module. Some products do allow the users to configure this feature separately but in general, the available operation modes work on exactly same principles as in case of Firewall Policy – in the Learning mode, the Behavior Control component will automatically allow the actions and create new rules for them; in the Interactive mode, the situations that can not be decided using the current rule set will alert the users and ask for their decisions; in the Silent mode, everything will be decided automatically etc.

Some products allow to set the protection level of Behavior Control. This setting defines which actions of the applications are considered as potentially dangerous. On lower protection levels applications are allowed to freely perform almost everything they want except for the most dangerous actions. On high protection levels the protection is very tight and the applications are monitored so closely that it may happen that even some of their legitimate actions are sometimes blocked.

Sometimes it is possible to set particular reactions to each of the potentially dangerous actions that can occur. Allow, Block or Prompt are the common options here meaning the action will be automatically allowed or blocked or that the user is to be asked when the corresponding event occurs.

The default configuration of many products prefers more automation, less user interaction, while the security is said to be optimal. This usually means that more advanced methods of behavior control are disabled and so only the most common malicious techniques are being watched. If this is the case and the user wants to be fully protected against the most sophisticated types of attacks it is necessary to enable the appropriate setting. Its name varies, for example, it can be called Advanced events monitoring, Anti-leak, but often a unique trademarked name is used for the technology.

Sandbox

When running under one of the automatic modes the security product may allow a malicious action to be executed (in case of Allow All/Allow Most modes), or block some of actions of legitimate programs (in case of Block All/Block Most modes). This situation is not ideal but using the Interactive mode requires users to make various decisions and so this mode assumes that the users are qualified to do so. Also the number of questions in the Interactive mode might be annoying for the users. An alternative approach that helps solving these problems is using a sandbox.

A product that implements a sandbox treats all unknown or suspicious programs in a special way that ensures they can not cause any damage to the system. It creates a special environment, the sandbox, that looks like the real system to the application that runs inside. The application can freely manipulate objects that it could not in the real system. For example, it can modify important registry entries. The sandbox makes sure that no such modification actually happens to the real system but it tries to simulate a behavior of the system so that the sandboxed application can not recognize it. In the example with the changed registry entry, when the application tries to read its value after the change the sandbox can return the modified value instead the one from the real registry that was not changed. There are several reasons why it is not possible to create perfectly authentic sandbox and this is why some of the most critical operations are always blocked in the sandboxed environment. The better the sandbox is the harder it is for the application to recognize that it is sandboxed.

Trusted programs always run outside the sandbox so that they are fully able to do their job without any limitations. When a new or an unknown application is installed by the user and the sandbox environment prohibits it to operate correctly, the user can exclude the particular application from the sandbox.

Some security suites implement the sandbox as a separate feature of the Behavior Control component. These products let users switch the sandbox off and still control the behavior of the applications. Others have it as an integral part of the Behavior Control component. There are products that allow to configure which actions are blocked automatically in the sandbox and which should be decided using the current policy and rules.

Potentially Dangerous Actions and Techniques

Potentially dangerous actions that security suites monitor can be divided into several categories or groups. We will describe the most common types of techniques that security suites control.

DDE communication – Dynamic Data Exchange (DDE) is a method of communication between two applications (so called inter-process communication). The DDE server application can accept messages from client applications and respond to them. Some applications such as Internet Explorer allow other applications to control them using DDE commands. This feature can be misused by malware to perform monitored actions in the context of trusted applications. In SSTS64, this technique is covered by DDEtest.
COM Access Control, OLE Automation Control – OLE Automation technology supersedes DDE. It is a more advanced mechanism of inter-process communication based on Component Object Model (COM). Many important system services provide interfaces for applications through COM/OLE mechanisms. When an interface is misused by malware it seems that it is the trusted service, not the malware process, that performs the dangerous action. In SSTS64, this technique is covered by Flank, BITStest, Schedtest etc.
DNS/RPC Client Services, DNS API Request – Some system services, such as DNS Client, are accessible through technologies called Remote Procedure Call (RPC), Local Procedure Call (LPC), or Advanced LPC (ALPC) that are used for inter-process communication. Similarly to aforementioned DDE communication and COM Access Control, some of these system services can be misused by malware. Monitoring the related communication can thus prevent misusing these services. In SSTS64, these techniques are covered by DNStester, WFPblock etc.
Application Window Control, Windows Messages – Windows Messages is yet another mechanism of inter-process communication. This one is used mostly to control graphical user interfaces (GUIs) of applications and this is exactly what can be misused. Using Windows Messages, it is possible to simulate common users actions, such as a mouse button click. As long as an application has a GUI based on technology that relies on Windows Messages, it can be attacked by malware using this technique. In SSTS64, this technique is covered by Breakout1, Kill3e, and Kill3f.
Code Injection, Process Memory Injection, Interprocess Memory Accesses – An injection of a code to another process running in the system is an easy method of executing malicious code in the context of a trusted process. Malware can be aware of the behavioral protection that limits its functionality and so in order to bypass the protection, it can inject its own code into a trusted process that is likely to be allowed to perform the critical malicious actions. This is why protecting trusted processes against code injection is fundamental for behavioral protection in security suites. In SSTS64, this technique is covered by AWFT4, CopyCat, Kill8 etc.
DLL Injection, Binary Planting – DLL Injection is very similar to Code Injection. The result of a successful attack is the same – an execution of a malicious code in the context of a privileged application. One of the differences is that in case of DLL Injection, a whole executable module is loaded into the target process, while Code Injection usually injects only a small piece of code. DLL Injection is easier to implement for the author of malware but also easier to detect for security products. In SSTS64, this technique is covered by Inject1, Inject2, FireHole2 etc.
Network-enabled Application Launch, Process Launching, Parent Process Control – On Windows systems, a parent process can control its child processes either by specifying their command line arguments or using special methods that are related to the internal functionality of the process launching itself. For malware applications, this feature presents just another way how to take advantage of trusted applications. Security suites react by monitoring the parent process chain either of all processes running in the system or just of the trusted processes. In SSTS64, this technique is covered by Tooleaky, VBStest, NewClass etc.
Process Termination – Process Termination and similar attacks (like Thread Termination, attempts to crash processes/threads) are intended to partially corrupt or even fully disable the protection of security software. The targets of these attacks are processes of the security suite. The actual result of a successful attack highly depends on the implementation of the damaged security suite. It can cause system instability, freezes, crashes, or it can cause some of the protection components to be disabled. Some security suites are able to recognize a corrupted state of their components and lock the computer in order to prevent malware exploiting their current weakness. In SSTS64, this technique is covered by Kill* tests and Crash* tests.
Low-level Network Access, Direct Network Access – Most of the security suites are able to control common network traffic, such as web traffic or email communication, very well but there might be a problem when less common or special purpose protocols are used. It is not rare that a security product allows communication with web sites (i.e. using Hypertext Transfer Protocol – HTTP – protocol) only to trusted programs while requests to send data to a network server using Internet Control Message Protocol (ICMP) are passed automatically without a notice. Malware that uses alternative ways to send or receive network data can be harder to detect for some security software. In SSTS64, this technique is covered by SockSnif, ECHOtest, and ECHOtest2.
Direct Disk Access – A common way to access data on hard disks is through system functions that work with files and directories. Older Windows systems allowed applications to access the disk device and its data directly. Such a way to access on disk data bypasses the common file and directory protection mechanisms. On Windows Vista and later systems, this technique is limited and thus less usable for most types of malware attacks and hence it is not covered by the current version of SSTS64.
Physical Memory Access, Direct Memory Access – Each running process in the system has its own private memory that is inaccessible to other applications by default. In case an access to a remote process memory is required, the system provides special functions that make this possible but at the same time it allows security software to control this kind of access. The operating system kernel also has its own private memory that is not accessible to applications. However, on older Windows systems, it was possible to get access to a special object that mapped the whole physical memory including the system kernel memory. This allowed malware to bypass the common mechanisms of accessing the memory. On Windows Vista and later systems, this technique is forbidden and hence it is not covered by SSTS64.
Device Driver Installation, Driver Load – Applications running on Windows are limited in what they can do, especially with system hardware resources, such as memory, disk, input and output devices etc. When an application wants to use such a resource it has to ask the system kernel which can either allow such a request and perform the action or deny it – this occurs, for example, when the application is run by the user that is not privileged to work with the resource. This mechanism works well for all application code that runs in so called user mode. The system kernel code runs in so called kernel mode, which allows it to directly access any resource without being limited. A code running in the kernel mode can also bypass all types of protection implemented either by system or third party software. A user mode application can load a device driver, the code of which runs in the kernel mode. It is thus necessary to control driver loading and do not allow malicious drivers to get executed. On 64-bit Windows systems, however, this technique is not easily exploitable by malware due to a requirement of a valid digital signature of every kernel driver and hence it is not covered by SSTS64.
Service Installation – System services on Windows are special programs that can run even when no user is logged on. They are usually more privileged than common applications, do not require direct communication with users, and can be started automatically during the system startup. Some of the services do not have their own process and are hosted with other similar services inside special hosting processes. For malware purposes, services are very easy way how to persist in the system. Security products usually also implement one or more services and without control of access to installation of system services the malware could disable important components of security suites. Moreover, the same interface that is used for installation and configuration of system services can be used for installation of kernel mode drivers. In SSTS64, this technique is covered by SvcKill.
HOSTS File Access – The HOSTS file is a special file that maps network names to IP addresses. Most commonly the network names are domains and the mapping between a domain and IP address is obtained using Domain Name System (DNS) protocol. However, it is the HOSTS file that is used first to translate network names, including domains, to IP addresses. It is thus possible to redirect a domain to arbitrary IP address using the HOSTS file. A common malware technique is to redirect anti-virus software update servers to invalid addresses which effectively disables the possibility to update the anti-virus. Other technique is used for phishing – a redirection of domains of various Internet banking and Internet payment systems to malicious servers that look exactly like the original web sites in order to steal the user's credentials. In SSTS64, this technique is covered by HostsBlock.
Active Desktop Changes – On older Windows systems, Active Desktop feature allowed users to insert active content (such as an HTML code) into the desktop. This feature made it possible to create highly customized desktops. Active Desktop could be misused by malware for executing actions in context of the trusted Windows Explorer application. Windows Vista and later systems do not implement the Active Desktop feature and hence it is not covered by SSTS64.
Autoruns, Autostart Locations – There exist many ways how an application can install itself or its own module into the system so that it is started or loaded again after a reboot. Some of these methods make it possible to infect various system processes with a custom DLL – i.e. perform DLL Injection. Common malware programs exploit one or more autorun locations in order to persist in the system. In SSTS64, these techniques are covered by Autorun* tests.
Keylogging, Keyboard Logging – Spying on users activity is another popular activity of malware. Keylogging techniques allow programs to get information about input the users typed into other applications in the system. Using these techniques, it is possible to steal passwords typed into a browser, an email client, or an instant messaging client. Some of the keylogging techniques rely on other techniques, such as Windows Hooking or DLL Injection. In SSTS64, these techniques are covered by Keylog* tests.
Screen and Clipboard Logging – Similarly to Keylogging, Screen and Clipboard Logging is used by malware to steal sensitive information. Taking screenshots can be used, for example, to steal credit card information typed into an otherwise secured web page in a browser. Clipboard Logging is based on stealing data from copy and paste operations that users do very often on Windows systems. Many users do transfer sensitive information, such as passwords, in the clipboard. This happens commonly when non-trivial passwords are required by web applications. One the one hand, using non-trivial passwords is needed so that they can not be cracked easily, but on the other hand a user that works with many of such applications might not be able to memorize all the passwords and uses a password storage software or text file and just copy-and-pastes the password from the storage to the form through the clipboard. In SSTS64, this technique is covered by Screenlog and Cliplog.
Window Hooking, Windows and WinEvent Hooks – Hooking Windows messages and so called WinEvents are mechanisms that operating system offers through specialized API functions to applications that want to monitor sending Windows messages in the system or receive generated notification events. These functions, however, can also be used by malware to implement various dangerous techniques like Keylogging or DLL Injection. In SSTS64, this technique is covered by CPILSuite2, FireHole, Keylog3 etc.

Component Control

Also called: Known Components

Every application uses one or more executable modules, sometimes called components. The main module is usually a .EXE file that requires several dynamic-linked libraries (DLLs) to be loaded into the same address space. Common applications rely on core system modules, such as Kernel32.dll, KernelBase.dll, ntdll.dll, Advapi32.dll, user32.dll etc. Many applications also rely on their custom libraries that are installed to the system with the main application module. One of the basic characteristics of DLLs is that they can be loaded during the initialization of the application's process as well as to be loaded later, just before their functionality is required.

Every application thus has more or less fixed set of DLLs that it loads and depends on. The Component Control feature recognizes these dependencies and controls loading of modules into applications' processes. When a malicious program wants to inject its DLL into another process the component control can recognize a new module being loaded to the target process and it can deny the action.

The component control is also responsible to guarantee the integrity of the trusted modules. Any attempts to modify the on-disk files of the known modules can be recognized and denied. This applies for both the main executable modules as well as for DLLs.

System Guard

Some security products implement different level of protection for the third party applications and the operating system and its components. System Guard is the part of the behavior control protection that is responsible for protecting the operating system itself from being infected by malware, for preserving the integrity of the operating system's parts. Important system files, critical registry keys (including autorun locations), and other system resources, such as COM interfaces, that can be attacked or misused are watched by the system guard. Any attempt to change the system or infect it is denied.

Removable Media Protection

The basic functionality of a security suite when dealing with removable media is to disable AutoRun and AutoPlay features. When a removable medium is inserted and its root directory contains Autorun.inf file, it is possible that a custom program is automatically started by the system. This can lead to a silent infection of the computer.

Many security suites also define a special set of rules for all programs stored on removable media. It is assumed that files stored on removable media may come from other computers that are not protected sufficiently and may be infected. This is why removable media programs are untrusted by default and their actions are usually highly restricted. Some suites can recognize programs that are digitally signed by trusted certificates and do not limit such applications even if they are stored on removable media.

Self-protection

One of the most critical features that the behavior control engine is responsible for is the product's self-protection. Any security protection the product implements becomes useless if malware can disable it. Modern security suites protect all their components so that they can not be switched off or damaged by malware. Self-protection requires protection of the product's processes and threads, files and directories, registry keys and values, installed system services and drivers, COM interfaces, and other resources created by the product that could be accessed by other processes running in the system.

It is vital for the security suite to prevent its own most trusted processes to be infected with malicious code or terminated. Many security suites also rely on regular updates of their databases. The update mechanisms must be designed securely so that malware can not subvert fake update files or block downloading or installing the updates.

Self-protection can be implemented naturally using common behavior control rules that prohibit manipulation of the product's resources; or it can be implemented separately from the security model provided to other applications, in which case the components of the security product are protected better than any other application and resource in the system. Both approaches are common in today's security suites.

Continue to Features of Modern Security Suites – Part 3

www.matousec.com