matousec.com (site map)

Poll

On Windows 7 (or Vista) I use

  ↓ unlimited administrator's account (55.76%)

  → limited administrator's account (19.12%)

  ↑ common user's account (11.25%)

  → nothing (I do not use Win 7/Vista) (15.39%)

more

results

Proactive Security Challenge 64

Introduction

Contents:

Latest news

Back to contents

Introduction

This project examines security software for Windows OS that implement application-based security model – i.e. most of the products called Internet security suites, personal firewalls, HIPS, behavior blockers, and similar products on the market. A product must meet some fixed criteria in order to be included in this project. The list of products suitable for this project that we are aware of is available on the product list page.

The goal of this project is to evaluate various abilities of security products to protect the user's data and the operating system based on the application behavior control and similar features. A product that succeeds in Proactive Security Challenge 64 is able to block well known techniques used by malware to steal or corrupt the user's identity or data, to infect and persist in the system, to participate in botnets, and to circumvent the protection implemented by the security product itself.

Back to contents

Methodology and rules

Installation and configuration

The tested products are installed on a virtual machine running Windows 7 Service Pack 1 with Internet Explorer 9 set as the default browser and with UAC turned off. The products are configured to their highest usable security settings and tested with this configuration only. We define the highest usable security settings as follows. The user must be able to do the configuration of the product without need of expert knowledge of the operating system and the computer security. This means that the user, with the skills and knowledge we assume, is able to go through all forms of the graphic user interface of the product and enable or disable or choose among several therein given options, but is not able to think out names of devices, directories, files, registry entries etc. to add to various table of protected objects manually, not even if such a configuration is suggested on the product's support forum or website. The product is configured to interact with the user as much as possible, reducing the number of automatic decisions made by the product as much as possible. To meet the usability requirement it must be possible to use the computer with the configured product for all legitimate tasks as if there was no security product installed. It is also required that the user is not forced to predict behavior of any unknown application and that under normal circumstances (i.e. no malware attack is in the progress) and once the product is set up properly, the product does not bother the user too often.

Testing levels

There are several testing levels in Proactive Security Challenge 64. Each level contains a selected set of tests and it also contains a score limit that is necessary to pass this level. All products are tested with the level 1 set of tests. Products that reach the score limit of level 1 and thus pass this level will be tested in level 2 and so on until they reach the highest level or until they fail a limit of some level.

Testing suite and scoring

Proactive Security Challenge 64 uses tests from Security Software Testing Suite 64 only. All the tests in this suite are available with source codes. Using the open set of tests makes the testing transparent as much as possible. For each test the tested product can get a score between 0 % and 100 %. Currently, all the tests can be simply passed or failed only and so the product can get 0 % or 100 % score only. It should be noted that the testing programs are not perfect and in many cases they use methods that do not guarantee recognizing whether the tested product passes or fails the test. This means that it might happen that the testing program reports that the tested system passed the test even if it failed, this is called a false positive result. The official result of the test is always determined by an experienced human tester in order to filter false positive results. The opposite situations of false negative results are rare and are also eliminated by the human tester.

All tests on the levels a tested product reaches are run at least once. If a product passes a test, this test is repeated at least once in order to mitigate false passing. For more information about the testing process see the testing guidelines.

Every test has a defined type. Tests of the same type usually attempt to achieve the same goal. Here is a list of the defined types and their goals:

The types of tests are defined for information purposes only. They do not determine the process of evaluation of whether the test was passed or failed. Each test implements one or more attacking techniques that can be used for various malicious purposes. A test implemented as a leak-test may use a more general technique that can be used to permanently infect the system. A tested product may be able to block the leak-testing part of the test and it still may fail the test because the core technique of the test may be usable for a different malicious purpose. It happens quite often that we use modified version of tests in order to check whether the tested product really protects against the specific attacking technique of the test or is just able to prevent the current test's implementation from succeeding.

All tests are equal to the intent that their scores are not weighted by their level or something else. The total score of the tested product is counted as follows. For all tests in all levels that the product did not reach, the product's score is 0 %. For all other tests the score is determined by the testing. The total score of the product is a sum of the scores of all tests divided by the number of all tests and rounded to a whole number. It may happen that a new test is added to Proactive Security Challenge 64 when some products already have their results. In such case, the result of this test for already tested products is set to N/A, which means that it is not counted for this product and does not affect its score or level passing. Neither the number of the tests, nor the number of levels is final. We may create new tests and levels in the future. We are also open to your ideas of new testing techniques or even complete tests.

Product's selection and vendors rights

Products for testing are selected from those that were requested for tests by their vendors or often suggested for tests by our visitors, more times than other products. If there are no such products, we will select products for tests ourselves, preferentially taking products that have not been tested at all yet and have a real chance to succeed in our tests. Every vendor has a right for its product to be tested in Proactive Security Challenge 64 for free once in a six months period and this right is valid only for stable and publicly available versions of the products. If a vendor offers more than one product it still has a right of only one free test per six months. Moreover, next free testing of a product will be performed no sooner than three months after the last free testing. This rule should prevent vendors from using Proactive Security Challenge 64 testing as a free beta testing service. We reserve the right to postpone the testing request and in exceptional cases also to completely refuse the testing request with or without a reason. The only exception from the free testing request rule is for vendors that offer a product with an anti-virus or an anti-malware engine and mark any of the tests of Security Software Testing Suite 64 as a virus, an infected code, an unwanted or a malicious application, or offend any part of the suite directly using pattern recognitions or any other form of blacklisting, or offend the Proactive Security Challenge 64 project in any other way. This approach deceives the users of such anti-virus or anti-malware engines and make the testing more difficult for us. The vendors who offend the testing suite have no right for free testing at all but can still request a paid testing.

Every vendor has a right to request the paid Proactive Security Challenge 64 testing, in which case its product will be tested in all levels regardless the results on each of the levels. After the vendor receives the results of the paid testing, it can either keep them private or request their publishing on our website, but such a request will be satisfied only if the previously published results for the tested product, if any, are at least one month old and if the tested version of the product is stable and publicly available. There are no limits of the frequency of the paid tests.

How you can help us

Do you enjoy Proactive Security Challenge 64? Do you want to help us but you do not consider yourself to be a security expert? Still you can help us! If you intend to buy a security software, you may be interested to buy one of the products we recommend in Proactive Security Challenge 64. Have a look at Products' ratings on the results page. The Recommendation column in the table contains links to the online stores or products' web pages of the vendors that we have affiliate agreements with. If you click on any of these links and then buy the tested product or other product offered on the target webpage, we will profit from it. So, if you are going to buy a security software and you like our projects, you can help us! Even if you have the licence already, we can profit if you prolong the licence after you visit the vendor's website through our recommendation links. Thank you!

The rules for the recommended products are simple. The first condition is that the product's Protection level is at least Very good, which means that its final score is at least 80 %. The second condition is that we have an affiliate agreement with its vendor. It is important to note that if the recommended product is retested and does not reach the 80 % limit, it will not be recommended anymore, at least not until the next retesting.

Back to contents

For vendors

We provide various services to vendors of security software. Besides the above mentioned paid Proactive Security Challenge 64 testing, we provide commercial testing based on the original methodology for advanced analyses. We also offer consulting services and research related to Windows internals, implementation of security software, design of security software, reverse engineering and malware analyses. Get more information about the services we offer.

Back to contents

News archive

Back to contents