matousec.com (site map)

Poll

How much system resources could your security products consume at most?

  → 0% - 1% (25.23%)

  → 2% - 5% (40.19%)

  → 6% - 10% (14.02%)

  → 11% - 15% (7.01%)

  → 16% - 20% (5.61%)

  → 21% - 25% (1.87%)

  → 25% - 30% (0.93%)

  → 31% and more (5.14%)

more

results

Firewall Challenge

Introduction

Contents:

Latest news

Back to contents

History and introduction

Firewall Challenge is a project that replaces our older project Window Personal Firewall Analysis and its subproject Leak-testing. As a part of Window Personal Firewall Analysis project we have deeply analysed security products but we found out soon that such a testing was extremely time consuming. It was not possible to test as many products as we wanted to. On the other hand, Leak-testing seemed to be a very easy way how to test many products in reasonable time. However, Leak-testing is not able to cover many of the important features of the desktop security products. We have decided to combine the simplicity and effectivity of Leak-testing with the scope of our deeper analyses and created this project – Firewall Challenge.

This project examines personal firewalls, Internet security suites and other similar products for Windows OS that implement process-based security. We call all such products personal firewalls or simply firewalls. In our opinion, personal firewalls should prevent spying and data and identity theft. So, we require personal firewalls to include host protection features too. The list of personal firewalls we are aware of is available on the product list page. We know that our terminology may be in conflict with the common understanding of what the firewalls are. To distinguish between personal firewalls and firewalls in the common sense, we call the later packet filters. A typical example of a packet filter is WIPFW. Most of the personal firewalls include a packet filter component. Simple packet filters are not worse than personal firewalls, they are just different kind of software – for different kind of users. This project does not examine stand-alone packet filters.

Back to contents

Methodology and rules

The tested firewalls are installed on Windows XP Service Pack 2 with Internet Explorer 6.0 set as the default browser. The products are configured to their highest usable security settings and tested with this configuration only. We define the highest security settings as settings that the user is able to set without advanced knowledge of the operating system. This means that the user, with the skills and knowledge we assume, is able to go through all forms of the graphic user interface of the product and enable or disable or choose among several therein given options, but is not able to think out names of devices, directories, files, registry entries etc. to add to some table of protected objects manually.

There are several testing levels in Firewall Challenge. Each level contains a selected set of tests and it also contains a score limit that is necessary to pass this level. All products are tested with the level 1 set of tests. Those products that reach the score limit of level 1 and thus pass this level will be tested in level 2 and so on until they reach the highest level or until they fail a limit of some level.

Most of the tests are part of Security Software Testing Suite, which is a set of small tests that are all available with source codes. Using this open suite makes the testing transparent as much as possible. For each test the tested firewall can get a score between 0% and 100%. Many of the tests can be simply passed or failed only and so the firewall can get 0% or 100% score only. A few tests have two different levels of failure, so there is a possibility to get 50% score from them. The rest of the tests have their specific scoring mapped between 0% and 100%. It should be noted that the testing programs are not perfect and in many cases they use methods, that are not reliable on 100%, to recognize whether the tested system passes or failed the test. This means that it might happen that the testing program reports that the tested system passed the test even if it failed, this is called a false positive result. The official result of the test is always set by an experienced human tester in order to filter false results. The opposite situations of false negative results should be rare but are also eliminated by the tester.

To be able to make right decisions in disputable situations, we define the test types. Every test has some defined type. Tests of the same type always attempt to achieve the same goal. Here is a list of the defined types and their goals:

All tests are equal to the intent that their scores are not weighted by their level or something else. The total score of the tested product is counted as follows. For all tests in all levels that the product did not reach, the product's score is 0%. For all other tests the score is determined by the testing. The total score of the product is a sum of the scores of all tests divided by the number of all tests and rounded to a whole number. It may happen that a new test is added to Firewall Challenge when some products already has their results. In such case, the result for already tested product is set to N/A for this new test, which means that it is not counted for this product and does not affect its score or level passing. Neither the number of the tests, nor the number of levels is final. We intend to create new tests in the future. We are also open to your ideas of new testing techniques or even complete tests.

Products for testing are selected from those that were requested for tests by their vendors or often suggested for tests by our visitors, more times than other products. If there are no such products, we will select products for tests ourselves, preferentially taking products that have not been tested at all yet and have a real chance to succeed in our tests. Every vendor has a right for its product to be tested in Firewall Challenge for free two times in six months period and this right is valid only for stable and publicly available versions of the products. If a vendor offers more than one product it still has a right of only two free tests per six months. Moreover, next free testing of a product will be performed no sooner than one month after the last free testing. This rule should prevent vendors from using Firewall Challenge testing as a free beta testing service. The exception from this rule is for vendors that offer two versions of the same product, from which one is available free of charge and the other one is a commercial version with some extra functionality and these version are likely to have different results in Firewall Challenge. The second and the last exception from this rule is for vendors that offer a product with an antivirus engine and mark any of the tests of Security Software Testing Suite as a virus, an infected code, an unwanted or malicious application, or offend any part of the suite directly using pattern recognitions or any other form of blacklisting. We have experienced such a behaviour in the past in case of leak-tests, this approach deceives the users of such antivirus engines and make the testing more difficult for us. The vendors who offend the testing suite have no right for free testing at all but can still request a paid testing.

Every vendor has a right to request a paid Firewall Challenge testing, in which case its product will be tested in all levels regardless the results on each of the levels. After the vendor receives the results of the paid testing, it can either keep them private or request their publishing on our website, but such a request will be satisfied only if the previously published results for the tested product, if any, are at least one month old and if the tested version is stable and publicly available. There are no limits of the frequency of the paid tests.

How you can help us

Do you enjoy Firewall Challenge? Do you want to help us but you do not consider yourself to be a security expert? Still you can help us! If you intend to buy a security software, you may be interested to buy one of the products we recommend in Firewall Challenge. Have a look at Firewalls' ratings on the results page. The Recommendation column in the table contains links to the online stores or products' webpages of the vendors that we have affiliate agreements with. If you click on any of these links and then buy the target product or other product offered on the target webpage, we will profit from it. So, if you are going to buy a personal firewall or a similar security software and you like our projects, you can help us! Even if you have the licence already, we can profit if you prolong the licence after you visit the vendor's website through our recommendation links. Thank you!

The rules for the recommended products are simple. The first condition is that the product's Protection level is at least Very good, which means that its final score is at least 80%. The second condition is that we have an agreement with its vendor. In case of free products, we need at least a consent of the product's vendor. It is important to note that if the recommended product is retested and does not reach the 80% limit, it will not be recommended any more, at least not until the next retesting.

Back to contents

For vendors

We provide various services to vendors of personal firewalls and similar security software. Besides the above mentioned paid Firewall Challenge testing, we provide commercial testing based on the original methodology for advanced analyses. We also offer consulting services and research related to Windows internals, implementation of security software, design of security software and malware analyses. Get more information about the services we offer.

Back to contents

News archive

Back to contents