matousec.com (site map)

Poll

Should software vendors reward independent researchers for finding vulnerabilities in their software?

  → Yes, by money and credit. (78.26%)

  → Yes, by credit only. (11.55%)

  → No. (6.78%)

  → Yes, by money only. (1.84%)

  → Other answer. (1.77%)

more

results

Proactive Security Challenge

Introduction

Contents:

Latest news

Back to contents

Introduction

This project examines security software for Windows OS that implement application-based security model – i.e. most of the products called Internet security suites, personal firewalls, HIPS, behavior blockers and similar products on the market. A product must meet some fixed criteria in order to be included in this project. The list of products suitable for this project that we are aware of is available on the product list page.

Back to contents

Methodology and rules

Installation and configuration

The tested products are installed on Windows XP Service Pack 3 with Internet Explorer 8 set as the default browser. The products are configured to their highest usable security settings and tested with this configuration only. We define the highest security settings as settings that the user is able to set without advanced knowledge of the operating system. This means that the user, with the skills and knowledge we assume, is able to go through all forms of the graphic user interface of the product and enable or disable or choose among several therein given options, but is not able to think out names of devices, directories, files, registry entries etc. to add to some table of protected objects manually, not even if such a configuration is suggested on the product's support forum or website.

Testing levels

There are several testing levels in Proactive Security Challenge. Each level contains a selected set of tests and it also contains a score limit that is necessary to pass this level. All products are tested with the level 1 set of tests. Those products that reach the score limit of level 1 and thus pass this level will be tested in level 2 and so on until they reach the highest level or until they fail a limit of some level.

Testing suite and scoring

Most of the tests are part of Security Software Testing Suite, which is a set of small tests that are all available with source codes. Using this open suite makes the testing transparent as much as possible. For each test the tested product can get a score between 0 % and 100 %. The tests can be simply passed or failed only and so the product can get 0 % or 100 % score only. It should be noted that the testing programs are not perfect and in many cases they use methods, that are not reliable on 100 %, to recognize whether the tested system passes or failed the test. This means that it might happen that the testing program reports that the tested system passed the test even if it failed, this is called a false positive result. The official result of the test is always set by an experienced human tester in order to filter false positive results. The opposite situations of false negative results should be rare but are also eliminated by the tester.

To be able to make right decisions in disputable situations, we define the test types. Every test has a defined type. Tests of the same type usually attempt to achieve the same goal. Here is a list of the defined types and their goals:

All tests are equal to the intent that their scores are not weighted by their level or something else. The total score of the tested product is counted as follows. For all tests in all levels that the product did not reach, the product's score is 0 %. For all other tests the score is determined by the testing. The total score of the product is a sum of the scores of all tests divided by the number of all tests and rounded to a whole number. It may happen that a new test is added to Proactive Security Challenge when some products already has their results. In such case, the result for already tested product is set to N/A for this new test, which means that it is not counted for this product and does not affect its score or level passing. Neither the number of the tests, nor the number of levels is final. We intend to create new tests in the future. We are also open to your ideas of new testing techniques or even complete tests.

All tests on the levels a tested product reaches are run at least once. If a product passes a test, this test is repeated at least once in order to mitigate false passing. For more information about the testing process see the testing guidelines.

Product's selection and vendors rights

Products for testing are selected from those that were requested for tests by their vendors or often suggested for tests by our visitors, more times than other products. If there are no such products, we will select products for tests ourselves, preferentially taking products that have not been tested at all yet and have a real chance to succeed in our tests. Every vendor has a right for its product to be tested in Proactive Security Challenge for free once in a six months period and this right is valid only for stable and publicly available versions of the products. If a vendor offers more than one product it still has a right of only one free test per six months. Moreover, next free testing of a product will be performed no sooner than three months after the last free testing. This rule should prevent vendors from using Proactive Security Challenge testing as a free beta testing service. The only exception from this rule is for vendors that offer a product with an anti-virus or an anti-malware engine and mark any of the tests of Security Software Testing Suite as a virus, an infected code, an unwanted or a malicious application, or offend any part of the suite directly using pattern recognitions or any other form of blacklisting. This approach deceives the users of such anti-virus or anti-malware engines and make the testing more difficult for us. The vendors who offend the testing suite have no right for free testing at all but can still request a paid testing.

Every vendor has a right to request a paid Proactive Security Challenge testing, in which case its product will be tested in all levels regardless the results on each of the levels. After the vendor receives the results of the paid testing, it can either keep them private or request their publishing on our website, but such a request will be satisfied only if the previously published results for the tested product, if any, are at least one month old and if the tested version is stable and publicly available. There are no limits of the frequency of the paid tests.

How you can help us

Do you enjoy Proactive Security Challenge? Do you want to help us but you do not consider yourself to be a security expert? Still you can help us! If you intend to buy a security software, you may be interested to buy one of the products we recommend in Proactive Security Challenge. Have a look at Products' ratings on the results page. The Recommendation column in the table contains links to the online stores or products' webpages of the vendors that we have affiliate agreements with. If you click on any of these links and then buy the target product or other product offered on the target webpage, we will profit from it. So, if you are going to buy a security software and you like our projects, you can help us! Even if you have the licence already, we can profit if you prolong the licence after you visit the vendor's website through our recommendation links. Thank you!

The rules for the recommended products are simple. The first condition is that the product's Protection level is at least Very good, which means that its final score is at least 80 %. The second condition is that we have an affiliate agreement with its vendor. It is important to note that if the recommended product is retested and does not reach the 80 % limit, it will not be recommended anymore, at least not until the next retesting.

Back to contents

For vendors

We provide various services to vendors of security software. Besides the above mentioned paid Proactive Security Challenge testing, we provide commercial testing based on the original methodology for advanced analyses. We also offer consulting services and research related to Windows internals, implementation of security software, design of security software, reverse engineering and malware analyses. Get more information about the services we offer.

Back to contents

News archive

Back to contents