matousec.com (site map)

Poll

On Windows Vista I use

  → unlimited administrator's account (26.11%)

  → limited administrator's account (9.09%)

  → common user's account (3.26%)

  → nothing (I do not use Vista) (61.54%)

more

results

Articles

Comparison of top five personal firewalls

Published: 2006/10/31

This page is outdated! Current information is available on pages of Firewall Challenge project.

This article is a final report of the first phase of Windows Personal Firewall Analysis project. It is based on our analyses of these five personal firewalls: ZoneAlarm, Kerio, Norton, BlackICE and Outpost. You can find a brief comparison of these products not only from the security point of view in this article. We also mention responses from product vendors and reactions we have received from end users.

Contents:

Back to contents

Tested products

The method how we chose personal firewalls for this phase of our project is described on the methodology page. Exact versions of tested products were:

All products were tested either with a full licence that we received from the vendor for our analysis or as trial versions without any limitations in their functionality.

Back to contents

Testing conditions

All personal firewalls were installed over a clean installation of Microsoft Windows XP Professional with Service Pack 2. No third party software was installed on these machines during the tests.

Before the analysis of the first product, the exact methodology was set. All products were analysed step-by-step using this internal methodology. This methodology defined an exact set of situations that all products were tested against. As a part of this procedure we have also developed automatic tools for finding vulnerabilities in personal firewalls.

Our analyses were concentrated on Self-protection, Verification of own components, Process protection, File and component protection, Driver protection, Service protection, Registry protection, Protection of other system resources, and Control of automatically started programs. These features were described in our article called Design of ideal personal firewall and are related to the personal firewall security. However, most of these topics were not fully examined, so there are many possibilities for further future analyses. The time spent on the analysis of each product depended on its complexity. The average analysis took about two weeks.

Back to contents

Final results

The final results are available on the result page of our project. The comparison table on that page shows a brief overview of tested products and links to our reviews of these personal firewalls. A big surprise for us was our finding that in all cases our automatic tools were able to find serious bugs in the implementation of important components of tested products. Our prior expectation that programmers of personal firewalls are familiar with basic principles of writing this type of software was wrong. This fact is one of the most alarming results of our tests.

The clear winner of our tests was ZoneAlarm PRO. Its security design was the best of all tested products and was nearly perfect. The reason why we did not recommend this product was an extensive number of bugs in the implementation of its security design and other features. ZoneAlarm was almost perfect in all other aspects that are not related to the security. Our review of ZoneAlarm.

Very close to ZoneAlarm with its final score was Outpost, which took the second place in our tests. This product was very modern and was also great in security unrelated aspects. Its security design was quite good but worse than the design of ZoneAlarm. Apart from the security design imperfections, this product had poor implementation in some parts and hence it tended to be unstable or incompatible with other software. Another big problem of this product was its performance. Our review of Outpost.

The third place was taken by Norton. The quality of this product was much worse than the quality of ZoneAlarm or Outpost. The security design was poor and incomplete and problems had appeared since its installation. The enormous complexity of this product resulted in problems with security. Some components did not work well together or were excluded from the security design. The whole solution was half-baked. Our review of Norton.

BlackICE took the fourth place. This product was unusable from the security point of view. Its security design was very poor and its implementation was not better. The performance and the hardware requirements were the only positives of this product. Our review of BlackICE.

The last place was taken by Kerio. This very popular software had a major problem and that was the absence of almost all security elements we would expect in a personal firewall software. Kerio could be classified as an average packet filter with a nice interface, not a personal firewall. Our review of Kerio.

Back to contents

Vendors' responses

Another interesting part of our tests was a chance to see reactions of vendors on our mostly negative reviews. We always contacted the vendor at least twice, before and after the analysis of its product. Vendors' attitudes were always different.

The most positive reaction came from Sunbelt Software, the vendor of Kerio, even though it came after we have published the first vulnerability for Kerio. However, after then we have received a response and felt a great interest in our results. Finally, Sunbelt Software bought our analysis and we believe they work on improvements for their software. Moreover, they were able to fix mentioned vulnerability very quickly. On the other hand, the fix for the second vulnerability we have published recently is still not available.

We have experienced almost the same scenario with similar results when we contacted Zone Labs, L.L.C., the vendor of ZoneAlarm. We have received the real response after the first vulnerability was published and the result of our communication was that they bought our analysis and expressed an interest in further cooperation with our group. They also analyse our report and probably work on issues presented in our analysis.

We have received a possitive reaction from Agnitum Ltd., the vendor of Outpost. They acknowledged our interest in their product with thanks but expressed no interest in our private results. However, they care about vulnerabilities that are published.

Symantec Corp., the vendor of Norton, was able to react and expressed an interest in the published vulnerabilities. The response time differed, sometimes they were very quick, other times very slow. We have not noticed any interest in other vulnerabilities that we keep private.

Internet Security Systems, Inc., the vendor of BlackICE, was not interested in our findings. Moreover, they were not interested in any of vulnerabilities we had published. These issues remain open and users of BlackICE should rather change to a better product.

Back to contents

Public reactions and our comments

We have received many reactions related to Windows Personal Firewall Analysis project. These reactions confirmed our assumption that people will be divided into two irreconcilable groups in their opinions. The first group understands the situation in similar way as we do and genarally do agree with our project and the way it is presented. The second group strictly disagrees. It is worth to say that neither of these groups is bigger than the other one. Here we state some of Reactions we have received and our comments (M):

R: Your model is blackmailing, you are terrorists, vendors should sue you ...
M: We strictly disagree with this point of view. Many vulnerabilities are found on the field of the security research in all kind of software every day. These vulnerabilities are reported to vendors as well as to public forums, conferences, news groups etc. The only difference in our model is that at first, we ask vendors whether or not they are interested in our project and findings for a fee. In case they are not interested, we behave as all other researchers and publish our findings. Our goal is to improve the security of end users. If vendors are interested in our project we are ready to help them with fixing issues in their software. In this case end users only profit from our project. If the vendor is not interested we publish the vulnerability which press the vendor to fix the issue. In this case end users only profit from our project. And finally, if the vendor is neither interested in our findings before nor after we publish the vulnerability, end users receive the information that the vendor of the product they use does not take the security seriously. In all cases end users profit from our project.

R: You sell the information for the tenth of its price, hence it seems to be a scam.
M: We agree that our prices are very low but since our goal is to improve the security of the tested products we only need something to live from. We suppose that if vendors buy our analyses they will be interested in further cooperation with our group. For example, we offer beta-testing on the security level to these vendors.

R: You sell the information for ten times more than its value is, hence it seems to be a scam.
M: We disagree. Our findings are made for vendors of professional security software, not for common users who are able to read our free reviews that lack technical details that most of people would not understand. These reviews are intended for common users who do not have deep technical knowledge. What is more, vendors of this kind of software have enough resources to buy our results and we believe it is worth for them. Our analyses are not trivial and are relatively time consuming, this is why we can not release all our findings for free.

R: You sell the information to hackers, they have enough resources to buy your findings.
M: This is not true. We always contact and deal with the vendor in the first place. The possibility to buy our product is open to everyone because we think that this can be an indication for vendors that our findings are real. Moreover, if we sell our reports we always require to know who the buyer is, so there is no chance to buy our reports anonymously. We also want to offer the possibility to buy our reports for independent researchers who are interested in this field. If any of our results are misused we will always cooperate with authorities if they contact us.

R: Secunia marks your critical bugs as not critical. How is this possible?
M: Secunia is on the top of the vulnerability research field. On this level they have to judge vulnerabilities from the global scope. Simply said, they are more interested in vulnerabilities that can not be prevented, they are more interested in remotely exploitable bugs. Our project is related to Windows personal firewalls. This environment is very specific. Firstly, the majority of users of personal firewalls on Windows OS are running under administrator's account. This is why personal firewalls are designed to protect the user even if administrator's account is used for common user's work. It is known fact that if users followed recommendations of security experts and worked only under unprivileged accounts the situation would be different. However, even many programmers of application software implicitly assume that their program is run under administrator's account. And even if this is not required by the character of their software they force users to use administrator's account when they run their software. Secunia can not mark vulnerabilities that requires administrator's account as serious. Our model of bugs classification focuses on real users. Their personal firewall is usually implemented to protect them even if administrator's account is used. This is right because it reflects the real situation. But then if the security model can be bypassed on this level we mark this problem as serious or critical and this is the real severity for those users we described.

R: Your reviews do not include leaktest results.
M: We intend to step into the leaktest field in near future. We will present our own leaktest programs and we will also test many products against well known leaktests too. These results will be available for free and will be included in rating of the tested products.

R: I like your project, it will help to improve security solutions. How can I help you?
M: We would like to thank all people who are able to see our real intentions and understand our point of view or at least do not a priori condemn it. The easiest way how you can help us is to spread a word about our project. If you want to help us with your knowledge and or time to improve our project, feel free to contact us.

R: More or less, your results say that all five tested products are not good. Then what do you recommend to end users to use?
M: It is true that our findings are very negative and we can not recommend any of the tested products to you. However, we clearly state that based on our tests, ZoneAlarm PRO is the best product from those five and we recommend it as a temporary solution until we find a better one. Since its vendor bought our results, we believe that they will fix issues in ZoneAlarm and will present a better product soon. We also think that users should watch vendors' responses. It is natural that every non-trivial software cointans bugs. Then the vendor's ability to react is also what should be considered when you choose which product to use. Back to contents