Poll
How much system resources could your security products consume at most?
Windows Personal Firewall Analysis
- What do we analyse?
- How do we analyse and what do we offer?
- Methodology reference
- Results of our analyses
- More about personal firewalls
- Design of ideal personal firewall
- Links
- Comparison of top five personal firewalls
- Introduction to Firewall Leak-testing
- Plague in (security) software drivers
- Leak-tests results
Methodology reference
This page is outdated! Current information is available on pages of Firewall Challenge project.
Contents:
- Attributes
- Groups
- Features
- Installation
- Hardware requirements
- User features
- Design features
- Testing programs
- Bugs
In the following text there is a list of attributes we investigate in every tested product. There are also definitions of terms and idioms that are used in the whole text and also on other pages related to this project. In the following text we use the following (old) notation 1 MB = 1024 kB, 1 kB = 1024 B (bytes), 1 B = 8 b (bits).
Definitions:
- Description
- The description is a block of text about described object. We write description immediately below the name of described object without any title.
- Product, firewall, software, application
- All these terms are used interchangeably and represent the personal firewall product we analyse.
- Vendor, producer, company
- All these terms are used interchangeably and represent the vendor of the personal firewall family we analyse.
Attributes
An attribute in our analysis is either a feature or a bug. Attributes can be united to groups. Some features and every bug in the final analysis have Testing program or Testing method.
Definitions:
- Name
- Every attribute and group has its own name which give a true picture of it.
- Code
- Every attribute has its own code. This code is unique in the context of this project, thus it can be referenced from any other text.
- Description
- Every attribute and group has its own description. It is a detailed text that fully describes the purpose of the attribute or group. The attribute description is written immediately after its name and code.
- Penalty
- Some features and every bug or bug ategories have their penalty. It is an integer value of points with a negative meaning that express the penalized fact. The penalty of a software bug is a sum of the bug penalty and penalties of all bug groups the bug belongs to. The penalty of a product is a sum of penalties of all its bugs and features. The penalty of a vendor is a sum of penalties of all its products.
- Penalty formula
- Usually features do not have their penalty tightly set. A concrete value of the penalty of the feature usually depends on their classification. In such case these features have Penalty formula instead of the penalty itself. After the feature classification is measured for the tested product its penalty value is counted using Penalty formula.
Groups
Groups serve us only to make the text easier to read. Groups can not be referenced (do not have a code) but have a name and a decription. A group can contain one or more attributes or other groups.
Features
To be able to think about security problems in a product we need to define some features of this kind of software. When we analyse a product we measure and evaluate these features. If we define the set of these features well we can deduce many other features from them and we may also see how is the tested software designed.
Definitions:
- Measurement
- The measurement of a feature explains how to examine or measure the given feature. If the measurement is not trivial for some feature it could be better to implement a Testing program that would test the feature.
- Classification
- The classification of a feature explains how to evaluate the results of its measurement.
Overall rating
The Overall rating is the fastest way how to compare two products. It covers all examined features and found bugs into a signle value.
Measurement:
The value of product Overall rating is the sum of penalties of all features and its bugs. This sum is updated every time a new feature or a bug is discovered.
Classification:
The Overall rating is a non-negative integer value of any size. The zero value is only theoretical value for ideal personal firewall.
Leak-testing score
Leak-testing examine the firewall's ability to prevent sensitive data to be send to the Internet without user's knowledge. These tests are very complex and this is why we have published a whole article about leak-testing.
Measurement:
The measurement procedure is described on a separate page devoted to leak-testing results.
Classification:
The classification is also decribed on the leak-testing result page. This page contains a table with the Product Score of tested firewalls. Leak-testing penalty of each firewall is equal to the theoretical maximum of the points the firewall can score minus its Product Score.
Penalty formula:
9625 - Product Score
Installation
On the clean installation of the operating system we install the tested product. We leave the default settings of the product after the installation and we only initialize privileges and access rights for common programs and utilities (e.g. an Internet access for Internet browsers). We watch the following features during the installation process. When we search for bugs later we usually harden security settings of the tested product.
Installation speed
The Installation speed is a time from the first run of a software installation package up to the moment when the software is ready to work. Some products are in so called learning mode after the installation process. When they are in this mode a user is asked to decide every single action which has not been decided before. The learning mode is not a part of the installation and does not affect the installation speed.
Measurement:
The Installation speed is a subjective view of the number and duration of installation steps that are necessary to complete the installation process. For example a product that requires to reboot the system several times has not as fast installation process as a product that does not require the system reboot at all.
Classification:
0-100%, where 100% means that the installation process is fast enough, and an explanation if it is not 100%.
Penalty formula:
10 * (1 - Classification)
Definitions:
- Installation end
- The above mentioned moment when the product is ready for common use.
- Installation start
- The first run of the software installation package.
- Learning mode
- The mode or state of some firewalls after the installation in which these firewalls ask user to make a decision about every action that was not decided before or for which there is no rule set yet. This mode can be usually switched to a normal firewall mode in which all undecided actions are denied.
Trouble-free installation
The trouble-free installation should be a commonplace. Security products should be tested properly because their affect on the system behaviour is extensive. A possible problem during the installation process can cause a damage of data, the system instability or even the system crash.
Measurement:
The trouble-free installation is the subjective view of the installation process. If there are any errors, compatibility issues, system crash during the installation we can hardly say it is a trouble-free process. We monitor the installation problems from the installation start up to the installation end.
Classification:
0-100%, where 100% means that the installation was completely trouble-free, and an explanation if it is not 100%.
Penalty formula:
30 * (1 - Classification)
Installation easiness
Often, especially in home offices, the administrator of the firewall (the person who installas the product) is a common user. For such cases the installation package should contain easy installation process and if it requires some advanced decision from the user it should be clearly explained. A software for general public should not rely upon users knowledge.
Measurement:
The Installation easiness is a subjective view of the installation process. We judge the understandability of installation questions and their explanations for common users.
Classification:
0-100%, where 100% means that the installation process is easy and understandable, and an explanation if it is not 100%.
Penalty formula:
10 * (1 - Classification)
Reboot requirement
The security software often requires a system reboot after the installation.
Measurement:
An attempt to commonly use the software without the system reboot but only in case the installation program does not ask for the reboot.
Classification:
YES / NO. YES, if the reboot is necessary.
Automatic database initialization
Some firewalls with the database fill this database during the installation with selected or all programs on the system hard disk. Other firewalls leave the database blank and fill it only during run in learning mode.
Measurement:
The check of the database immediately after the installation end.
Classification:
YES / NO. YES, in case the database contains any programs or objects that does not belong to the personal firewall itself.
Hardware requirements
Hardware requirements of a software can assist as a secondary characteristic during the decision-making which product to buy. If we have more alternatives with the same functionality and price we can follow Hardware requirements with our final decision. They can be also used as a filter criterion that eliminates unsuitable products for our hardware from our selection. Values of following features are measured immediately after the installation end.
Memory
Memory hardware requirements.
Measurement:
We measure memory requirements after the reboot and system initialization (process userinit.exe finished) using Windows Task Manager. From this value we subtract the value that was measured on the same hardware before the installation process. During the measurement there run no other programs except those that were included in the measurement before the installation. The measurement is repeated at least twice.
Classification:
The measured value is an approximated value range in MB between separated measurements. The smaller value the better result.
Penalty formula:
5 * min(Classification in MB)
Disk
The on disk size of the installed software.
Measurement:
We measure the on disk size of the installed software. We do not take into consideration the size of potential updates of system components.
Classification:
The measured value in MB. The smaller value the better result.
Penalty formula:
2 * (Classification in MB)
Performance
The common methods of personal firewall implementations natively reduces the performance of the target operating system. In the ideal case the performance impact is very small so that the end-user does not even know about it, not even if demanding applications run. That is why it is a good to know how the tested product affects the system performance.
Measurement:
Running Testing programs for performance measurement before the installation and after its end and comparing measured values. We do not take into consideration the first run of the testing program after the installation. During the first run a user can be asked by the firewall to make a decision about some action performed by the testing program. It is needed to allow all these actions. The measurement can not be affected by any user reaction and this is why we do not count with values from the first run. An output of the testing program is always redirected to a file. Before every run of the testing program the system has to be rebooted so that the result of the run is not affected by a previous one. We make three measurements plus the first initial measurement. The result is an arithmetic mean of measured values.
Classification:
0-100%, where 100% is only a theoretical value and it would mean that the product does not affect the performance in any way. The result is an arithmetic mean of measured values. With every such value we compare the time before and after the installation and we express the performance impact as a result of the division of the measured time before and after the installation. For example the final value of 50% (which is 0.5 in fact) for some action means that the measured value after the installation was two times greater than before the installation.
Penalty formula:
300 * (1 - Classification)
Testing programs:
User features
Among User features we include such features that are not important from the security point of view. The classification of these values is usually subjective. We do include here features like the application appearance and easy of use.
Easy of use
The prime quality security product should offer well arranged and understandable control environment too. If a user interface is chaotic it can also present a security risk. For example if the user is asked to make a decision of potentially dangerous action and the user will not understand the severity or the core of the question they can decide in the opposite sense as desirable. End-users also care about the design and the application appearance with which they often work. Another important feature is the possibility of easy setup of rules and behaviour. For example the user should be able to set the firewall into fully automated mode.
Measurement:
The Easy of use is a subjective view of the application behaviour to the end-user, its communication interface, its appearance, its pleasentness, understandability of its questions and other similar features.
Classification:
0-100%, where 100% holds for the product with well arranged and understandable control environment. The classification can be extended with an additional comment.
Penalty formula:
20 * (1 - Classification)
Design features
Design features of a security product are those that affect a system security. Designs of personal firewalls' security differs. There are many places in the system that should be protected by every such software. But there also exist many places that can be but do not have to protected by a personal firewall depending on its security design. What is more, many of these places can be reliably protected with several different techniques. That is why we can not say that the tested program has a bad or insecure design only because it does not implement all of the Design features we investigate.
To differentiate between key and superfluous features we have to understand the security design of the tested product. It is rather appropriate combination of selected features that determines the quality of the application design not only from the security point of view but also from the user and administrator usability point of view. However, there are several features that are important regardless the design and so the absence of such features can be immediately called a bug in the security design. Some features may depend on the existence of others or can be affected by them. It should be clear from a context how an absence of dependent feature affects the one being described. For example if we require some action to be executed by a program with some restrictive feature (e.g. the feature that it is a parent process), and this feature is not implemented in the tested product, then we execute the action with an arbitrary program.
Among Design features we do not include common user features and other features like Popup/Cookie/Spyware Blocking, Parental Control or Content Filtering which are often main elements of comparative tables of firewall products. We consider these features as irrelevant from the security point of view.
Known program database
The majority of today's personal firewalls implements a database of programs (or modules) that are somehow known to the firewall. The firewall assigns a set of access rights and restrictive rules to these programs. These process rights and rules are used by the firewall engine to decide whether to allow or deny a specific action. Known programs can be identified by a few different methods e.g. a hash of a binary module, an on disk path of the module, a name of the module, by dependencies on the other modules etc. or by a combination of mentioned methods.
Measurement:
An attempt to find a database in the tested software. Usually the user interface of the firewall allows you to look into the database and to edit its records.
Classification:
YES / NO. YES, in case the product contain such database.
Definitions:
- Database
- The above mentioned list of programs.
- Known process
- A process (an instance) of a known program.
- Known program
- A program included in the database.
- Protected object
- An object that is guarded by a personal firewall engine. It is not necessary only a program or a process, it can be a thread, a file, a directory, a registry key or value etc. Some of these objects can be hidden to a common user and the user usually can not see (or at least can not modify) the key objects. However, other objects like programs can be arbitrarily modified in the database.
- Unknown process
- A process (an instanace) of an unknown program.
- Unknown program
- A program which is not included in the database.
Unknown process start control
When an unknown program is executed the firewall may ask a user for a permission and thus prevent the execution of the unwanted program.
Measurement:
To an arbitrary unprotected directory we copy a program which is different in its name, hash and functionality from all other programs on the disk. We run this program using an arbitrary parent process.
Classification:
YES / NO. YES, if the firewall alerts the user or denies the execution of the unknown program.
Modified process start control
For known programs, especially if they are somehow privileged, it is necessary to secure their integrity or at least to check it. If an attacker changed a module of privileged program and if the firewall did not check its integrity then such a modified program could execute a malicious code with all the rights of the original program.
Measurement:
We change an arbitrary known program with an arbitrary unknown program and we run it with an arbitrary parent process.
Classification:
YES / NO. YES, if the firewall alerts the user or denies the execution of the modified known program.
Process start with modified dependent module control
For the known programs the firewall can control modifications of dependent modules. In case a dependent module is change by an attacker and the program is run the final effect can be the same as if the original program was modified.
Measurement:
We take an arbitrary known program and we replace one of its dependent modules on the disk with a modified copy. We run the program using an arbitrary parent process.
Classification:
YES / NO. YES, if the firewall alerts the user or denies the execution of the program of which the dependent module was modified.
Control of automatically started programs
A malicious software usually wants to persist in the system for as long as possible. There are many ways how to configure or modify system components (e.g. registry) such that the malicious program will be executed regularly (e.g. every system start or after user logon). The automatic execution in the system which is not controled by the firewall can present a security risk. This is why many personal firewalls implements the control of automatically started programs.
Measurement:
An attempt to find a place in the system which may serve to a malicious program to be executed regularly and which is protected by the firewall.
Classification:
YES / NO. YES, in case there exists such a protected place in the system.
Directories protection
There can be directories in the system in which an improper file modification, deletion or creation by an attacker can lead to a change of behaviour of privileged programs and such change can be a dangerous for the system security. As an example we can take 'Startup' folder. Files or links in this folder are automatically started after every user logon. The firewall can be designed to protect some important directories and alerts the user or denies the action if an unprivileged program attempts to access such directory. Names of protected directories are usually maintained in the dabatase as protected objects.
Measurement:
An attempt to create or find a protected directory on disk. Usually, protected directories are system directories (Windows, System32, Documents and Settings, Program Files) or the firewall installation directory.
Classification:
YES / NO. YES, in case we were able to create or find the directory on disk which was somehow protected by the firewall.
Definitions:
- Protected directory
- The above mentioned directory.
- Unprotected directory
- The directory that is not protected.
Files protection
A similar object to a protected directory with the only exception that it is a file.
Measurement:
An attempt to create or find a protected file on the disk. Protected files are usually those in system directories (Windows, System32, Documents and Settings, Program Files) or in protected directories.
Classification:
YES / NO. YES, in case we were able to create or find a file that is somehow protected by the personal firewall.
Definitions:
- Protected file
- The above mentioned file.
- Unprotected file
- The file that is not protected.
Registry keys protection
Likewise in Directory protection we have defined protected directories there can be some keys protected in the system registry too. The analogy of 'Startup' directory is the registry key 'Windows\CurrentVersion\Run' in which the registry values represent programs that are run after every user logon. And so the registry key can also be the protected object.
Measurement:
An attempt to create or find a protected registry key.
Classification:
YES / NO. YES, in case we were able to create or find a key in the registry that was somehow protected by the firewall.
Definitions:
- Protected registry key
- The above mentioned registry key.
- Unprotected registry key
- The registry key that is not protected.
Registry values protection
A similar object to protected registry key with the only exception that it is a registry value.
Measurement:
An attempt to create or find a protected registry value.
Classification:
YES / NO. YES, in case we were able to create or find a registry value that was somehow protected by the firewall.
Definitions:
- Protected registry value
- The above mentioned registry value.
- Unprotected registry value
- The registry value that is not protected.
System service protection
There are processes with special privileges and behaviour in the user mode of the operating system called system services. The firewall security design may restrict a communication with these services or their creation, modification or other manipulation. It is very common that some part of the personal firewall is implemented as a system service.
Measurement:
An attempt to create or find a protected system service and an attempt to manipulate system services.
Classification:
YES / NO. YES, in case we were able to create or find a system service which was somehow protected by the firewall or in case the firewall restricts the manipulation with services or their creation.
Definitions:
- Protected system service
- The above mentioned system service.
- Unprotected system service
- The system service that is not protected.
System driver protection
There are modules with special implementation, privileges and behaviour in the system called system drivers. Drivers have a direct access to a system kernel and privileged processor instructions and they are considered to be reliable. The firewall security design may restrict a communication with these drivers or their creation, modification or other manipulation. It is very common that some part of the personal firewall is implemented as a system driver.
Measurement:
An attempt to find a protected system driver. An attempt to load a new driver.
Classification:
YES / NO. YES, in case we were able to create or find a system driver which was somehow protected by the firewall or in case the firewall restricts loading new drivers.
Definitions:
- Protected system driver
- The above mentioned system driver.
- Unprotected system driver
- The system driver that is not protected.
Protection of static objects of different type
The system defines whole range of static objects that do not correspond any of protected objects described in Directories protection, Files protection, Registry keys protection, Registry values protection, System service protection or System driver protection. As examples we can take system objects called Event, Mutant/Mutex, Section etc. Some of these objects may be important in a security design of the personal firewall thus they may be protected by the firewall.
Measurement:
An attempt to find a protected object of different type.
Classification:
YES / NO. YES, in case we were able to find a object of different type that was somehow protected by the firewall.
Definitions:
- Protected object of different type
- The above mentioned object of different type.
Parent process control
The firewall design may require a dependence control of process execution. To create a new process can be implement as a privilege for special or selected processes only. Processes that can create a child processes are called parent processes. In some firewalls the control of the process creation can be limited to known processes only, other processes can be created by an arbitrary process there. Such a control may prevent an attacker to run privileged program and execute a malicious activity with it.
Measurement:
Running an arbitrary unknown program using another unknown program. Running known program using an unkown program.
Classification:
YES-YES / YES-NO / NO-YES / NO-NO.
YES-YES, in case, the firewall alerts the user or denies the creation of both known and unknown processes using the unknown program.
YES-NO, in case the firewall alerts the user or denies the creation of known process using the unknown program but it allows the creation of unknown process using the unknown program.
NO-YES, in case the firewall alerts the user or denies the creation of unknown process using the unknown program but it allows the creation of known process using the unknown program.
NO-NO, in case the firewall allows the creation of both known and unknown process using the unknown program.
Definitions:
- Parent process
- The above mentioned process that is allowed to create another processes.
Open process control
Executed programs run in the system as processes. Processes can be limited by the firewall based on the rules in the database. The system implements several types of process opening. For example a process can be opened by another one for memory reading, for memory writing, for a new thread creation etc. It is a good idea to prevent unprivileged processes to open other processes (or at least to restrict some types of openeing).
Measurement:
Running Testing programs for open process control on an arbitrary known process.
Classification:
YES / NO. YES, if the firewall alerts the user or forces at least one testing program to fail.
Testing programs:
Testing programs for open process control
Definitions:
- Opening the process
- The above mentioned action of a process.
Open thread control
Every process consist of one or more threads that execute a code. Similarly to the situation described in Open process control the system implements several types of thread opening and it can be a good idea to restrict this action.
Measurement:
Running Testing programs for open thread control on a thread of an arbitrary known process.
Classification:
YES / NO. YES, if the firewall alerts the user or forces at least one testing program to fail.
Testing programs:
Testing programs for open thread control
Definitions:
- Opening the thread
- The above mentioned action of a process.
Control of function calls that can corrupt process integrity
CreateRemoteThread, WriteProcessMemory, VirtualAllocEx, SetThreadContext, SuspendThread etc. are system functions that allow one process to violate an execution of another process (creation of a new thread in the process, rewriting its memory, allocation of its resources etc.). Such an action can be dangerous. All of these functions require a handle to a process which is the result of the process opening or a handle to a thread which is the result of the thread opening. It depends on the design of the firewall whether to implement a process protection based on Open process control and Open thread control or whether it protects the above mentioned possible dangerous functions.
Measurement:
Running Testing programs that can corrupt process integrity on an arbitrary known process.
Classification:
YES / NO. YES, if the firewall alerts the user or forces at least one testing program to fail.
Testing programs:
DLL injection control
DLL injection is a technique that can be implemented using functions described in Control of function calls that can corrupt process integrity. It is a loading of a new module to the existing process. This can be used to execute a malicious code with the privileges of the target process.
Measurement:
Running Testing programs that can corrupt process integrity on an arbitrary known process.
Classification:
YES / NO. YES, if the firewall alerts the user or forces at least one testing program to fail.
Testing programs:
DLL injection testing programs
Definitions:
- DLL injection
- The above mentioned technique.
Inbound connection control
The attacker who wants to control his malicious program that is already installed in the system may implement this program as a network server. The program listens on a port of some network interface (if TCP or UDP protocols are used) until it is connected by a client from another place in the network. This behaviour can be dangerous. The attacker can also transfer stolen sensitive data and files using such communication channel. This is why it is convenient to control network servers and their communication by the firewall.
Measurement:
Running Inbound connection testing programs.
Classification:
YES / NO. YES, if the firewall alerts the user or forces at least one testing program to fail.
Testing programs:
Outbound connection control
The attacker who wants to control his malicious program that is already installed in the system may implement this program as a network client. The client connects to a server on the network and fulfils commands it receives. This behaviour can be dangerous. The attacker can also transfer stolen sensitive data and files using such communication channel. This is why it is convenient to control outbound connections by the firewall.
Measurement:
Running Outbound connection testing programs.
Classification:
YES / NO. YES, if the firewall alerts the user or forces at least one testing program to fail.
Testing programs:
TCP/UDP port management
Instead of or together with Inbound connection control of single programs the firewall may offer an interface for a global management of open TCP and UDP ports. And similar interface can be implemented instead of together with Outbound connection control such that only specific ports (only specific network services) can be connected from local machine.
Measurement:
An attempt to find the above mentioned management in the user interface of the firewall.
Classification:
YES / NO. YES, if the firewall administrator is able to define rules for TCP and UDP protocols.
Other protocols control
For the unwanted communication described in Inbound connection control and in Outbound connection control the attacker can use also other protocols like ICMP instead of TCP or UDP. Some personal firewall implementations are oblivious of these protocols.
Measurement:
Running Testing programs using other protocols.
Classification:
YES / NO. YES, if the firewall alerts the user or forces at least one testing program to fail.
Testing programs:
Other protocols management
Instead of or together with Other protocols control of single programs the firewall can offer an interface for a global management of these protocols. If this management is implemented it is usually a connected with the interface for TCP/UDP port management.
Measurement:
An attempt to find the above mentioned management in the user interface of the firewall.
Classification:
YES / NO. YES, if the firewall administrator is able to define rules for above mentioned protocols.
Token privilege elevation control
Every process is started with limited set of privileges which depends on the user who run the process. Every started thread is given the same set as the process it belongs to. There are many privileges that are not in the privilege set of the process even if it is run by an administrator. If the process wants these privileges it has to ask the system for them. Examples of such privileges are the debugging privilege, load driver privilege and system shutdown privilege. Some of these privileges can be misused by a malicious program hence it is a good idea to protect requests for these privileges.
Measurement:
Running Testing programs for token privilege elevation control for all possible privileges.
Classification:
YES / NO. YES, if the firewall alerts the user or forces at least one testing program to fail.
Testing programs:
Settings protection
Settings of the personal firewall have to be protected. the protection is usually implemented using a password lock. For more complex products it may be useful to create system of users and privileges. In many products a user is given a chance to protect firewall settings during the installation process.
Measurement:
An attempt to protect firewall settings.
Classification:
YES / NO. YES, in case the firewall allows its administrator to protect its settings against unwanted modification by a user or program.
SSDT hooks
Hooking System Service Descriptor Table is the most common method of personal firewalls implementation. It is a replacement of some internal system services in the kernel mode. New functions are usually implemented to check privileges of the calling process before they call the original function code. The check itself is implemented as a lookup to the firewall database or as a query to the user in case there is no matching rule in the database and the personal firewall is in learning mode.
Measurement:
Looking into SSDT and enumeration of all modified functions.
Classification:
The result of the measurement is a set of functions which were changed in SSDT by the firewall.
Usermode hooks
Usermode hooking is a method for the implementation of personal firewall or its parts. It is a replacement of some system functions very similar to SSDT hooks but in the usermode. For some features it is the easiest method how to implement them. However, this method can be used only for special features. If this method is used for implementing basic features it is a fatal design error.
Measurement:
Running Testing programs for usermode hooks enumeration without any limitations. If the firewall asks to permit or deny some action we always permit it.
Classification:
The result of the measurement is a set of functions that are hooked in usermode by the firewall. Some functions can be hooked only in some process (i.e. hook is not global for all processes), in such case we also write down the process name.
Testing programs:
SSDT GDI hooks
System Service Descriptor Table GDI hook is very similar to SSDT hook. It is a method of implementation of some features of personal firewalls. GDI functions are used by the system to manage GUI (graphic user interface).
Measurement:
Looking into SSDT GDI and an enumeration of all modified functions. Unlike SSDT hooks these functions are not exported and thus they do not have public names. That is why we use only their indexes in the result of this measurement. However, with some functions we can guess their purpose and in such case we name the function.
Classification:
The result of the measurement is a set of functions that are hooked in SSDT GDI by the firewall.
Testing programs
With some features the easiest way how to examine them is to write a special but simple program. This program has only one purpose: to test the given feature. It makes the measurement instead of the analyst. The results depends on a classification of the given feature. However, in most cases the tested firewall does have (or implement) the feature if the action of the testing program fails because of a firewall activity including a user alert. There can be more than one testing program for one feature. In most cases if at least one testing program fails the firewall does have (or implement) the feature. All the testing programs are naturally unknown programs and they are run by a parent process unless otherwise indicated.
Definitions:
- Usage
- The usage of a testing program is a syntax of its calls in format 'prog -OPT ARG' where 'prog' stands for the name of the testing program (the common name of testing program is 'test'), 'OPT' is a list of options and switches (this list can be empty, if it is non-empty it starts with symbol '-'), 'ARG' is a list of arguments (this list can be empty). Possible values of both lists 'OPT' and 'ARG' are described in the usage too.
- Note
- Additional comment or explanation of nonstandard behaviour of testing program or its feature.
Testing programs for open process control
Download: TST00OPC.zip
This program opens a process specified by PID with OpenProcess function. Flags are used to open the process with all possible access rights. If the process is opened the program reports success. This program is used to test following features: Open process control.
Usage:
prog PID PID - process identifier - number that uniquely identifies every running process
Testing programs for open thread control
Download: TST01OTC.zip
This program opens a thread specified by TID with OpenThread function. Flags are used to open the thread with all possible access rights. If the thread is opened the program reports success. This program is used to test following features: Open thread control.
Usage:
prog TID TID - thread identifier - number that uniquely identifies every thread in the system
Testing programs that can corrupt process integrity
Download: TST02PIC.zip
At first OpenProcess or OpenThread is used to open process or thread specified by PTID. Such action can be guarded by firewall hence the least possible and needed rights for the following action are used. CreateRemoteThread, WriteProcessMemory, VirtualAllocEx, SetThreadContext and SuspendThread are called then. If the call is successful the program reports success. This program is used to test following features: Control of function calls that can corrupt process integrity.
Usage:
progK PTID
progK - stands for program names "prog1", "prog2", ...
PTID - process or thread identifier
- number that uniquely identifies every process and thread
DLL injection testing programs
Download: TST03INJ.zip
This program implements the DLL injection technique. API functions OpenProcess, VirtualAllocEx, WriteProcessMemory, CreateRemoteThread and LoadLibrary are used. The program reports success if the module is injected. This program is used to test following features: DLL injection control.
Usage:
prog PID PID - process identifier - number that uniquely identifies every running process
Inbound connection testing programs
Download: TST04ICC.zip
This program creates TCP server listening on given port on all available network interfaces. The program reports success if the server is created and listenting. This program is used to test following features: Inbound connection control.
Usage:
prog PORT PORT - TCP port number in range 1-65535 for program to listen on
Outbound connection testing programs
Download: TST05OCC.zip
This program tries to establish connection with TCP server given by its address and port. The program reports success if the connection is established. This program is used to test following features: Outbound connection control.
Usage:
prog IP PORT IP - IPv4 address of target computer PORT - TCP port number in range 1-65535 for program to connect to
Testing programs using other protocols
Download: TST06OPC.zip
This program tries to send ICMP request to the target computer and receive a response. If it succeeds program reports success. This program is used to test following features: Other protocols control.
Usage:
prog IP IP - IPv4 address of target computer
Testing programs for token privilege elevation control
Download: TST07TEC.zip
This program tries to obtain system privilege specified in argument for its token. If the token gets the privilege the program reports success. This program is used to test following features: Token privilege elevation control.
Usage:
prog PRIV PRIV - name of system privilege defined in winnt.h
Testing programs for performance measurement
Download: TST08TPM.zip
This program tests the performance of several system functions. It measures time needed for searching, reading, creating, writing and deleting files and directories on the disk and keys and values in the registry. It also measures the performance of working with processes. The result of each test is an integer value that can be compared to the other tests. Only tests made on the same system and hardware can be compared because these factors can highly affect the results. This program is used to test following features: Performance.
Usage:
prog (the program is executed without special arguments)
Testing programs for usermode hooks enumeration
Download: TST09UHM.zip
This program works in two phases. In the first one it detects global function hooks. It runs helper program for every system library. This helper program tries to load the library to the memory and then it compares memory images of this library with its on disk image. During the first phase there are quite a lot of libraries loaded that throws exception because of various reasons. These libraries are not interesting in this test and we ignore all exceptions that raise in helper program. In the second phase it detects function hooks in other processes. It opens every process and reads their virtual memory. Then it compares memory images of libraries in target processes to their on disk images. This program is used to test following features: Usermode hooks.
Usage:
prog (the program is executed without special arguments)
Bugs
Bugs are inseparable part of software products. There are many categories of bugs and many of them can not be judged without the knowledge of the context they appear in. By a clever combination of a few smaller bugs an attacker can cause greater damage than by an exploitation of a few unrelated bugs with a higher individual severity. That is why we think it is a good idea to describe every bugs separately and rate them without possible relations with other bugs and after then to describe all their possible combinations that lead to higher risks than single bugs. Descriptions of bugs categories follow. A penalty is set for every single bug category and additionally every discovered bug has its own unique penalty which is set in the context of the product it appears in. Then the total penalty of the bug is a sum of this unique penalty and penalties of all bug categories the bug belongs to.
Definitions:
- Testing methods
- In some special cases a bug can have the Testing method instead of a Testing program. Testing method is a description of a procedure that is used by an analyst or a tester to see whether the tested product contains the bug or not. It is used very rarely.
Bug type
The Bug type classifies a bug by its origin. To determine the Bug type helps us to understand the cause of the bug.
Coding bugs
Coding bugs are the most common types of bugs. They arise during the ordinary programmer's activity. These bugs are often corrected and their fixes provided as product updates or patches. The risk of these bugs ranges from unimportant to critical (e.g. the famous buffer overflow). If the bug of this type is located in the code it is usually easy to fix it.
Penalty: 10
Implementation bugs
The Implementation bug arises during the implementation of some feature of the product when a dangerous programmer's technique is used, or when the programmer does not fully understand the correct way of using some external function or its technical details, or using the code which does not support the required level of security, or when an inappropriate technique is used for the implementation.
Penalty: 20
Design bugs
The Desing bug is very unpleasant type of bug because it is very hard to fix it later. Sometimes it is even impossible without rewriting the majority or the base of the code. In a way these bugs are the worst possible bugs. They arise when programmers are not aware of the center of some problem or when they neglect or do not know about some important technological facts. This cathegory includes bugs in design logic too. Design bugs are usually critical.
Penalty: 40
Incomplete design implementation bugs
The Incomplete design implementation bug arises in case the product implementation covers only a part of its (security) design. For example we can take an arbitrary object in the system that has to be protected by the personal firewall and of which there are more ways to access it. The firewall should protect all these ways otherwise it contains the Incomplete design implementation bug. This category also contains bugs that were caused by an incomplete rule set for some protected object. This type of bug is very common in security software. The risk of Incomplete design implementation bug is usually serious or even critical but it is usually easy to fix it.
Penalty: 20
Bugs combinations
The bug risk of many bugs depends on additional conditions especially on the existence of other bugs. By a combination of two originally not so serious bugs we can gain a critical bug. For example we can think of a malicious software which sniffs and logs users activity. Such a software is practically harmless unless it is able to send its logs to the attacker. The most (or only) interesting bug combinations are those of which the risk is greater than the risk of the most risky bug in the combination. A special piece of information in the description of the bug combination is the enumeration of combined bugs. The penalty of the combination is included in the unique penalty of the bug.
Bug character
The Bug character reflects the impact and the possible result of bug exploitation. The Bug character is closely associated with the Bug risk.
Complete system control
By the exploitation of the bug the attacker gains a complete system control, obtains an access to all protected objects and data and can execute arbitrary operations.
Penalty: 300
System crash
The exploitation of the bug cause a system crash and the reboot of the system is required, unsaved data are lost.
Penalty: 100
Denial of service
By the exploitation of the bug the system resources become insufficient and some system service becomes unusable or unstable. The system or service reboot is required. To this character category we also include bugs that fully or partially disable some functionality of the affected software or service but can not lead to a complete system control.
Penalty: 50
Exposure of sensitive information
By the exploitation of the bug the attacker gains an access to a sensitive information such as passwords or user data.
Penalty: 220
Damage of sensitive data
By the exploitation of the bug some sensitive data are deleted or damaged.
Penalty: 150
Damage of insensitive data
By the exploitation of the bug some insensitive data are deleted or damaged. Among these data we include data with only informative character.
Penalty: 30
Privilege escalation
By the exploitation of the bug the attacker gains some extra privileges. These bugs often arise when a privileged program or a part of the system is not protected enough and it is misused or replaced by the attacker to execute protected actions.
Penalty: 150
Supportive bug
The Supportive bug itself does not present a security problem. This bug just gives more options to the attacker and is often a part of bug combinations.
Penalty: 20
Resource leak
By the exploitation of the bug some system resources leak (e.g. famous memory leaks).
Penalty: 20
Bug status
The Bug status changes during the lifetime of the bug and it depends on the reaction of the product vendor. A list of bugs for a specific product indicates the willingness of the product vendor and how it reacts to a bug discovery. Penalties of bugs in the fixed state are counted only in penalties of products that are vulnerable.
Definitions:
Unpatched bugs
The Unpatched bug is every newly discovered bug. The bug is unpatched until the vendor releases an update or a patch for it or a new product version in which the bug does not appear anymore.
Penalty: 50
Fixed bugs
The bug is fixed since the vendor released a public update or a patch for it or a new product version in which the bug does not appear anymore. The bug is also considered as fixed if the vendor published a procedure how to modify product's settings that prevent the exploitation of the bug and that do not limit the product functionality or usability.
Penalty: 50
Partially fixed bugs
The bug is partially fixed if it is not completely fixed but if the vendor released a public update or a patch for it or a new product version in which the bug was limited such that the bug risk was lowered.
Penalty: 50
Bug risk
The Bug risk differentiates bugs among several risk factor categories that follow. If a bug appears when a product is set to its default settings (settings that are used immediately after the installation before user changes them) and if it is possible to fix the bug only by adjusting the product settings and if this action does not limit the functionality or usability of the software in any way then the bug risk is decreased by one to the lower category. In such case this fact is also mentioned in the bug description.
Unimportant bugs
Unimportant bugs do not affect security in any way.
Penalty: 5
Minor bugs
Minor bugs present only small security problems. They often require more conditions to be fulfilled or nonstandard settings to be applied. These bugs can not damage the system or data. To this category we also include bugs that require user interaction (but not regularly executed actions e.g. the system reboot).
Penalty: 20
Serious bugs
Serious bugs present higher security problems. They can crash the system, set the system into an unstable state or damage important data. By the exploitation of the serious bug the attacker can not gain the Complete system control or reveal sensitive information (or in such cases some special requirement has to be fulfilled).
Penalty: 80
Critical bugs
The bug is critical in case it allows the attacker to gain the Complete system control, to reveal sensitive data, discredit the system, violate system integrity or violate security rules that are necessary for the system reliability or violate rules because of which the security product was installed.
Penalty: 240
Bug exploitability
The Bug exploitability from the attacker point of view. One bug can attend one or more following categories.
Locally exploitable bugs
The bug is locally exploitable if the attacker that has a physical or programmatic access to a target system (the attacker has an account on the system or is able to execute a code on the system) is able to exploit this bug.
Penalty: 20
Remotely exploitable bugs
The bug is remotely exploitable if the attacker that has neither a physical nor programmatic access to a target system (the attacker has neither an account on the system nor is able to execute a code on the system) is able to exploit this bug. These bugs are usually exploited by sending malicious network packets to the target system. Some of Remotely exploitable bugs are also Locally exploitable bugs.
Penalty: 80
Bug discoverability
The Bug discoverability reflects a complexity of contemplations that preceded the bug discovery. We can also say that the Bug discoverability gives some additional information to the bug risk and thus extends bug risk categories. Obviously a critical bug which is Easily discoverable present a higher security risk than another critical bug which is Hardly discoverable. However, this differentiation tells nothing about the level of difficulty needed to exploit this bug if we already discovered the bug and we have a full description of it. In such case it is usually easy to implement a working code that can be use to exploit the bug. The Bug discoverability also gives us some information about the quality of affected software. If we concede that every software (even the security software) may contain serious bugs then these bugs should not be easily discoverable. Security products should be properly tested before they are sold.
Easily discoverable bugs
The discovery of Easily discoverable bug does not require any special knowledge or technical resources. It is often enough only to think about the security design of the product or about the implementation of its features. To this category we also include bugs that can be discovered using some well known or old method that is or was commonly used to test similar products. The presence of such a bug indicates the ignorance or negligence during the implementation and or about insufficent testing of the product. We can assume that even casual attacker will be able to exploit this bug.
Penalty: 50
Medium discoverable bugs
The discovery of Medium discoverable bug requires a special knowledge or trick but it does not require any special technical resource. A method used to reveal this bug may be known and well described but in such case it is recent. Bugs that belong to this category can be prevented during the application design or implementation by proper testing and examinating the above mentioned methods. We can assume that the majority of casual attackers do not have a necessary resources to exploit these bugs unless their descriptions were published.
Penalty: 30
Hardly discoverable bugs
The discovery of Hardly discoverable bug requires a special technical resources or thorough knowledge of the design and implementation of the security software. We can assume that only pointed attacks of experts can exploit these bugs if their descriptions were not published.
Penalty: 10