matousec.com (site map)

Poll

On Windows 7 (or Vista) I use

  unlimited administrator's account (58.03%)

  limited administrator's account (16.51%)

  common user's account (13.67%)

  nothing (I do not use Win 7/Vista) (14.07%)

more

results

Proactive Security Challenge 64

Frequently asked questions

Contents:

Testing guidelines

Question: What is the exact procedure to reproduce your results of Proactive Security Challenge 64?

Answer: We start with a clean machine with Microsoft Windows 7 Service Pack 1 installed. The 64-bit version of Microsoft Internet Explorer 9 is used as the default browser. Every Proactive Security Challenge 64 test is performed by at least two independent testers who must agree on the results unanimously.

To reproduce the results, follow these steps:

  1. Use the SSTS64 Configurator tool to create a snapshot of the system prior the tested product's installation. Make sure the 64-bit version of Internet Explorer and Task Manager are running while the Configurator tool runs.
  2. Install the tested product. Install all its components.
  3. Reboot the system.
  4. Update the tested product if possible.
  5. Start the 64-bit version of Internet Explorer and surf the web, use Windows Explorer to visit an Internet page and also to check information about the code signing certificate of arbitrary signed application, use ping.exe to ping an Internet address, schedule a task using schtasks.exe, run a visual basic script with wscript.exe, run Task Manager and terminate some processes with it, reboot the machine. Run the programs using cmd.exe as a parent process, especially the 64-bit version of Internet Explorer and Windows Explorer which are to be executed with parameters. If the tested product asks you about any action choose Allow and remember if possible. You can also set to trust system processes and services if applicable.
  6. Configure the tested product to its highest usable security settings as defined in Methodology and rules.
  7. Make sure the tested product runs in an interactive mode or is set to a similar policy, in which all undecided actions of applications cause the product to ask user about the decision.
  8. Use the SSTS64 Configurator tool to create the configuration file now. Make sure the 64-bit version of Internet Explorer and Task Manager are running while the Configurator tool runs. Make sure the tested product's GUI is opened. Do not forget to configure the LAN inteface address in the generated ssts64.conf configuration file. Modify the generated configuration file so that the lists of files and registry entries contain only those objects that were created by the tested product or its installer.
  9. Enable a password protection for product's termination if available.
  10. Reboot the machine and repeat the step 5, then perform Windows Update. Make sure that every common action is allowed without any questions from the tested product. If it still asks, change its settings to satisfy this condition and repeat this step.
  11. If there are no more alerts the machine is ready for testing. Run the 64-bit version of Internet Explorer and Task Manager and let them run especially during the testing. Make sure the tested product's GUI is opened.
  12. Copy the test files to the testing machine. If there are problems with the product's anti-virus engine create exceptions for the copied files. If it is not possible, disable the anti-virus engine.
  13. Save the system state.
  14. Run a test. If the product passed the test, repeat the test and if the product passes it for the second time, mark the test as passed and continue with the next test. If the product failed the test, mark the test as failed, restore the saved system state and continue with the next test. If the test caused BSOD or damaged the system in a way it is not bootable again or if the system was seriously damaged in any other way, proceed as if the test was failed.
  15. If a technique of a termination test was successful but the test did not report a failure because no processes were terminated, try to repeat the test several times and try to use the product as much as possible. This includes attempts to perform the product update, file system scanning if available, various actions with products logs and reports etc. Do not blindly trust the result reported by the test, if you are not sure about the test result follow the scoring definition for the given test. The My leaks page or a network sniffer may be handy too.

This approach is the same for most of the products. Occasionally, however, we encounter products which require individual approach. If the product is suspected that it is designed to protect only against a specific implementation of a test or it offends the testing suite in another way, we implement a modification of the test in order to bypass the protection. Other special cases are usually mentioned in the Further notes section of the PDF report.

Note that the above information may be updated, improved or changed in case of updates of the testing suite, the testing process or the project rules.

Note that the current implementation of the tests requires to use the 64-bit version of Internet Explorer, not the 32-bit version. However, this limitation is artifical and can be removed. We have decided to set this requirement only in order to keep the implementation of the tests simple.

Product requirements

Question: What kind of products are suitable for Proactive Security Challenge 64 testing and which are not?

Answer: We often receive requests to test security products that are not suitable for Proactive Security Challenge 64. It is important to understand what kind of products we test. The primary requirement is that the product implements application-based security model and behavior blocking. This means that it allows its users to control selected actions of applications. Among behavior blocking capabilities, the product must be able to control applications' network access. Then we require the product's project to be alive and intended to be used by desktop users. We are not interested in already dead projects without a future although exceptions may appear. Finally, we require the tested version of the product to be stable, publicly available in English and run on Windows OS that is currently supported by the challenge. Most of the products called an Internet security suite, a personal firewall, a HIPS, a behavior blocker do meet all these criteria and hence they are suitable for Proactive Security Challenge 64 testing.

On the other hand, there are many products that are not suitable for our project. Security products that are built to protect only a single process are not suitable – various Internet browser security add-ons, sandboxes or virtualizations, for example. Also behavior blockers that focus on a single type of malware are not suitable – e.g. anti-keyloggers, malware removers. All pattern based systems that are not based on application behavior are not suitable – this includes all anti-virus and anti-malware solutions that are not delivered with application-based security module or do not implement at least a minimal application-based security model.

The security software that is NOT suitable for Proactive Security Challenge 64 testing just because it is not publicly available or stable can be tested in private (commercial) Proactive Security Challenge 64 without a chance to publish its results, however. Any other security software that is NOT suitable can be tested on commercial basis outside Proactive Security Challenge 64.

Testing request

Question: How can I request a testing of my favorite product?

Answer: Simply visit our contact us form and send us names of the products you would like us to test. Your votes for testing these products will be added to our database. When we receive a significant number of votes for a single product, we will include it in our tests.

Security products versus Proactive Security Challenge 64, instead of malware

Question: How is avoided a danger that security product vendors will start to focus on fighting against Proactive Security Challenge 64 tests instead of malware? The reason can be immediate positive business impact of successfully passed tests.

Answer: We have faced this problem since our first projects. Some vendors really fight the tests and not the methods the tests implement – their attacking techniques. Some vendors optimize against the given set of tests rather than solving the causes.

If we have a suspicion that the tested product attacks some test directly, we use internally modified versions of the tests to prove it. If we can prove such a behavior, we mention this in the testing report and the product fails the test.

Another situation is when the vendors blindly add functionality to their software to pass some technique. In such a case, their users might be confused by absurd, false, misleading or somehow bad alerts, popups and questions. In this case, such a product might get through our tests but it would be unusable for most of the users. We hope that vendors will not do this for their own good. It is likely that the end-users would reject such a product.

Finally, we have also set a fixed rules about the frequency of testing, this should also help. The free testing is limited and we also limit a number of results that can be published for a single product even when the testing is paid by the vendor.

Termination tests' methodology

Question: The methodology for termination tests seems to indicate that termination of any of the security product's processes results in a failure in the test. I disagree with that methodology as the main features of the product may be unaffected by the termination (e.g. if the process that was terminated was only the tray icon) or the product may have some kind of "fail-safe" meachanism implemented (e.g. blocking all connections if the processes are not running). I think a test (e.g. "leaktest.exe") should be run after a termination to see if the protection is still working or not. If the product stopped the test after the termination it should receive a partial score (e.g. 50% of the normal score for the termination test).

Answer: The idea behind our scoring system is to allow the tests to be as simple as possible. We can not really say how the termination of one component affects the whole protection system unless we analyse the system deeply. We do not do that in Proactive Security Challenge 64.

Imagine a product that implements the GUI component which communicates with the user. Imagine that if this component is terminated, the product blocks all connections to the Internet. You say that if we run "leaktest.exe" to verify the protection, it will tell us whether the protection is weakened.

In a classic model of a driver, service and GUI component there are communications channels opened between these components. And these channels may be implemented so that only one connection is allowed to prevent malicious software to connect to the channel and send fake requests over it. If the GUI component is terminated, it may become possible to connect to these channels and attack the service or driver component through them. The verification you suggest does not reveal this case and there are many other situations that should be verified before we could say that the protection was not weakened at all.

Termination of any of the product's component is a security issue. In our scoring system it is penalized and we are not aware of any easy modification that would make the system more accurate or more fair.

Administrator's or limited account

Question: I'm just curious if these tests are carried out under an administrative or limited account?

Answer: According to our poll most of Windows 7/Vista users work under the fully privileged administrator's account. This is why we perform our tests under administrator's account with UAC turned off, to be as close to the real scenario as possible. This choice is also better to distinguish between the products that are effective only if users take advantage of the security mechanisms of the operating system and those that work regardless the operating system configuration and users' habits.