matousec.com (site map)

Poll

How much system resources could your security products consume at most?

  ↑ 0% - 1% (23.58%)

  ↑ 2% - 5% (39.02%)

  ↑ 6% - 10% (17.89%)

  ↑ 11% - 15% (8.94%)

  → 16% - 20% (4.07%)

  → 21% - 25% (1.63%)

  → 25% - 30% (0.81%)

  → 31% and more (4.07%)

more

results

Windows Personal Firewall Analysis

Leak-tests results

This page is outdated! Current information is available on pages of Firewall Challenge project.

Contents:

Back to contents

Latest news

Back to contents

Introduction

As a part of our Windows Personal Firewall Analysis project we have performed our own leak-testing. This page contains results of these tests and a comparison of anti-leak features of the most popular personal firewalls. To learn more about leak-tests, we recommend you to read our Introduction to Firewall Leak-testing.

We present more tests than any other leak-testing website. We have tested 21 personal firewalls from our list against 26 leak-tests. For this purpose, we have also implemented our own leak-tests. Our leak-tests are called Coat, Fake Protection Revealer (FPR) and Runner. They are downloadable from the download page of the Windows Personal Firewall Analysis project and their source code is included. Later, we have tested even more products and retested new versions of already tested products. Against this arsenal neither of the tested products was able to gain 100% score until 2nd November 2007.

We would like to thank José Pascoa who provided us a free licence of Atelier Web Firewall Tester 3.2 for our tests.

Back to contents

Methodology

Each firewall was tested twice against 26 leak tests - once with its default, out-of-the-box settings, and once with its highest security settings. Each firewall was then awarded an overall score derived from its pass/fail result against each test. The higher the score, the better the firewall performed against the range of leak tests. For every test the firewall passed on its default settings it gained 125 points. For those tests that the firewall failed on its default settings but passed on its highest security settings it gained 100 points. The number of tests per firewall settings is 77. Thus the maximum score is 77 * 125 = 9625 points. The tested firewalls were installed on Windows XP SP2, Internet Explorer 6.0 was set as the default browser and was running during the tests.

Back to contents

Result table

In the table below, products and leak-tests are sorted lexicographically.

Explanation of used symbols

PASSED_DEF - The tested product passed the test on its default settings (125 points).
PASSED_HI - The tested product failed the test on its default settings but passed on its highest security settings (100 points).
FAILED - The tested product failed the test. (0 points)
X/Y - Some tests are more complex because they consist of multiple subtests. Such complex tests have (?/MAX) in their name where MAX is the number of subtests (i.e. the maximum number of subtests the firewall can pass). Results for each firewall are presented as X/Y - a green and a blue number separated by a slash. If any number is omitted, it means it is zero. If both numbers are zero, a red minus sign is displayed. X (the green one) is the number of subtests the tested product passed on its default settings, Y (the blue one) is the number of subtests the firewall failed on its default settings, but passed on its highest security settings. The remainder (MAX - X - Y) is the number of subtests which firewall failed on either settings. For example, if a complex test has (?/10) in its name, the result 3/5 means that the firewall passed 3 subtests on its default settings, failed 5 subtests on its default settings, but passed these subtests on its highest security settings, and that it failed on the remaining 2 (10 - 3 - 5 = 2) subtests on either settings. The final score for the firewall for the one complex test is equal to X*125 + Y*100.

 
Product / Test I. Breakout I. CopyCat I. DNStest I. FPR (?/38) I. LeakTest I. pcAudit2 I. Surfer I. Wallbreak.(?/4)
II. BITStester II. Coat II. CPILSuite (?/3) II. FireHole II. Jumper II. pcAudit II. Runner II. TooLeaky II. ZAbypass
III. AWFT (?/10) III. Breakout2 III. CPIL III. DNStester III. Ghost III. OSfwbypass III. PCFlank III. Thermite III. YALTA
III. II. I. III. II. I. III. II. I. III. II. I. III. II. I. III. II. I. III. II. I. III. II. I. III. II.
Armor2net NE NO NO NO NO NO NO NE NO NO NO 2 NO NO YES NO NO NO NO NO NO NO NO NE YES NO
Ashampoo NE NO NO NO NO NO NO NE NO NO NO 2 NO NO YES NO NO NO NO NO NO NO NO NE YES NO
AVG NE NO NO NO NO NO NO NE NO NO NO 2 NO NO YES NO NO NO NO NO NO NO NO NE YES NO
Avira 1 NO NO NO YES NO NO NE YES NO OK 5/6 OK NO YES NO NO NO NO YES OK NO OK 1 YES NO
BitDefender NE NO NO NO YES NO NO NE NO NO NO 3 NO NO YES NO NO NO NO NO NO NO NO NE YES NO
BlackICE 10 NO NO NO YES NO NO NE YES NO YES 23 YES YES YES NO YES NO NO YES YES NO YES 1 YES YES
Blink 4 NO NO NO YES YES NO NE NO NO NO 9 NO NO YES NO NO NO NO NO NO YES NO NE YES NO
CA NE NO NO NO YES NO NO NE NO NO NO 4 NO NO YES NO NO NO NO YES NO NO NO NE YES NO
Comodo 10 YES YES YES YES YES YES 3 YES YES YES 35/3 YES YES YES YES YES YES YES YES YES YES YES 1/3 YES YES
DSA 10 NO YES YES NO YES YES 1 YES YES YES 24 YES YES YES YES YES YES YES YES YES YES YES 4 YES YES
F-Secure 9/1 NO NO NO OK YES YES 1 OK NO YES 17/12 OK YES OK NO YES OK NO NO OK YES OK 4 OK NO
Filseclab NE NO NO NO NO NO NO NE NO NO NO 2 NO NO YES NO NO NO NO NO NO NO NO NE YES NO
FortKnox 10 NO NO YES YES YES YES 1 YES NO YES 14 YES YES YES NO NO NO NO YES YES YES YES 2 YES NO
G DATA 7/2 OK NO OK YES NO NO 2 NO NO YES 18/8 YES YES YES NO OK OK NO YES YES NO YES 2 YES NO
GSS 10 NO NO NO YES YES YES 1 YES NO YES 30 YES YES YES NO YES YES NO YES YES YES YES 4 YES NO
Jetico v1 10 YES NO NO NO YES YES 2 YES NO YES 31 YES YES YES YES YES YES NO YES YES YES YES 3 YES NO
Jetico v2 10 YES NO YES YES YES YES 3 YES YES YES 37 YES YES YES YES YES YES YES YES YES YES YES 4 YES YES
Kaspersky 10 YES YES OK OK YES YES 1/2 YES OK YES 19/17 OK OK OK OK YES OK YES OK OK YES OK 4 OK OK
Lavasoft 10 YES YES YES YES YES YES 3 YES YES YES 30 YES YES YES NO YES YES YES YES YES YES YES 4 YES YES
Look 'n' Stop 1/5 OK NO NO YES NO NO NE YES NO YES 12/6 YES NO YES NO OK YES NO YES YES NO YES 1 YES NO
McAfee 1 NO NO NO YES NO NO NE NO NO YES 7/1 NO YES YES NO YES NO NO OK NO NO NO 2 YES YES
Norman NE YES NO NO NO NO NO NE NO NO NO 3 NO NO YES NO NO NO NO NO NO NO NO NE YES NO
Norton 6 NO NO NO OK NO NO NE OK OK OK 18 OK NO OK NO OK NO NO OK OK NO OK 1 OK NO
Online Armor 10 YES YES YES YES YES YES 3 YES YES YES 38 YES YES YES YES YES YES YES YES YES YES YES 4 YES YES
Outpost Free NE NO NO NO YES NO NO NE NO NO NO 4 NO NO YES NO NO NO NO YES NO NO NO NE YES NO
Outpost Pro 10 YES YES YES YES YES YES 3 YES YES YES 38 YES YES YES YES YES YES YES YES YES YES YES 4 YES YES
Panda NE NO NO NO OK NO NO NE NO NO NO 1/2 NO NO OK NO NO NO NO YES NO NO NO NE NO NO
PC Tools 1/9 NO NO NO YES OK OK 1 YES NO YES 10/14 YES OK YES OK OK OK OK YES YES OK YES 1 YES NO
PC-cillin 10 NO NO NO NO YES YES NE YES YES YES 30 YES NO YES NO YES YES NO NO YES YES NO 4 YES NO
Prisma NE NO NO NO NO NO NO NE NO NO NO 1 NO NO YES NO NO NO NO NO NO NO NO NE NO NO
Privatefirewall 10 YES YES YES NO YES YES 1 YES YES YES 25 YES YES YES YES YES YES YES YES YES YES YES 4 YES YES
ProSecurity 10 YES NO NO YES YES YES 3 YES NO YES 35 YES YES YES YES YES YES YES YES YES YES YES 4 YES YES
Rising NE NO NO NO NO NO NO NE NO NO NO 2 NO NO YES NO NO NO NO NO NO NO NO NE YES NO
Safety.Net 1/5 OK NO NO OK NO NO NE OK NO YES 6/12 OK NO YES NO OK OK NO OK OK NO YES 1 YES NO
SensiveGuard 5 OK NO NO YES NO NO NE NO NO NO 9 NO NO YES NO NO NO NO YES NO NO NO NE YES NO
Sunbelt (Kerio) 10 NO NO NO OK YES NO 1 YES NO YES 4/15 OK NO OK NO YES YES NO OK OK YES OK 4 OK NO
Sygate 1 OK NO NO NO NO NO NE NO NO YES 7/3 NO NO YES NO OK OK NO NO NO NO YES 1 YES YES
SSM 10 YES NO NO OK YES YES 3 YES NO YES 29/3 YES YES OK NO YES YES NO YES YES YES YES 4 OK NO
TheGreenBow NE NO NO NO NO NO NO NE NO NO NO 3 NO NO YES NO NO YES NO NO NO NO NO NE YES NO
Windows NE NO NO NO NO NO NO NE NO NO NO NE NO NO NO NO NO NO NO NO NO NO NO NE NO NO
ZoneAlarm Free NE NO NO NO YES NO NO NE NO YES NO 6 NO NO YES NO NO NO NO YES NO NO NO NE YES YES
ZoneAlarm Pro 10 OK YES YES YES YES YES 1 YES YES YES 34 YES YES YES NO YES YES NO YES YES YES YES 4 YES YES
Back to contents

Firewalls' ratings

The table below sorts the tested firewalls by their final score. This table also shows the exact version of every tested product.


Product Product Score Level of Anti-leak Protection
Online Armor Icon Online Armor Personal Firewall 2.1.0.19 FreeFREE 9625 Excellent - 100%
Outpost Pro Icon Outpost Firewall Pro 2008 6.0.2162.205.402.266 9625 Excellent - 100%
Comodo Icon Comodo Firewall Pro 2.4.18.184FREE 9475 Excellent
Jetico v2 Icon Jetico Personal Firewall 2.0.0.35 9375 Excellent
ProSecurity Icon ProSecurity 1.40 beta 1 8875 Very good
ZoneAlarm Pro Icon ZoneAlarm Pro 7.0.408.000 8600 Very good
Lavasoft Icon Lavasoft Personal Firewall 2.0.1019.7604 (700) 8500 Very good
Kaspersky Icon Kaspersky Internet Security 7.0.0.125 8475 Very good
SSM Icon System Safety Monitor 2.4.0.617 beta 7975 Very good
Jetico v1 Icon Jetico Personal Firewall 1.0.1.61 FreewareFREE 7750 Very good
Privatefirewall Icon Privatefirewall 5.0.8.11 7625 Very good
GSS Icon Ghost Security Suite [BETA] 1.110 7500 Very good
DSA Icon Dynamic Security Agent 1.0.8.8FREE 7375 Good
PC-cillin Icon Trend Micro PC-cillin Internet Security 2007 15.30.1151 7000 Good
F-Secure Icon F-Secure Internet Security 2007 7.01.128 6625 Good
G DATA Icon G DATA InternetSecurity 2007 6100 Good
PC Tools Icon PC Tools Firewall Plus 3.0.0.36FREE 5825 Poor
BlackICE Icon BlackICE PC Protection 3.6.cpv 5750 Poor
Sunbelt (Kerio) Icon Sunbelt Personal Firewall 4.5.916 5200 Poor
FortKnox Icon FortKnox Personal Firewall 2007 2.0.205.0 5125 Poor
Look 'n' Stop Icon Look 'n' Stop 2.06 4300 Poor
Safety.Net Icon Safety.Net 3.61.0002FREE 4000 Poor
Norton Icon Norton Internet Security 2008 15.0.0.60 3600 Very poor
Avira Icon Avira Premium Security Suite 7 build 98 2450 Very poor
SensiveGuard Icon SensiveGuard 1.06FREE 2350 Very poor
Sygate Icon Sygate Personal Firewall 5.6.2808FREE 2350 Very poor
McAfee Icon McAfee Internet Security Suite 2006 8.0 2325 Very poor
Blink Icon Blink Personal Edition 3.0.8.1496FREE 2250 Very poor
ZoneAlarm Free Icon ZoneAlarm Free 7.0.302.000FREE 1500 Very poor
CA Icon CA Personal Firewall 2007 3.0.0.196 1000 None
Outpost Free Icon Outpost Firewall Free 1.0.1817.1645FREE 1000 None
BitDefender Icon BitDefender Internet Security 10.108 750 None
Norman Icon Norman Personal Firewall 1.42 750 None
TheGreenBow Icon TheGreenBow Personal Firewall 2.60.005 750 None
Panda Icon Panda Antivirus + Firewall 2007 6.00.00 650 None
AVG Icon AVG Anti-Virus plus Firewall 7.5.431 500 None
Armor2net Icon Armor2net Personal Firewall 3.13.30 500 None
Ashampoo Icon Ashampoo FireWall Pro 1.14 500 None
Filseclab Icon Filseclab Personal Firewall 3.0.0.8686FREE 500 None
Rising Icon Rising Personal Firewall 2007 19.33.10 500 None
Prisma Icon Prisma Firewall 2.4.4.0 250 None
Windows Icon Windows Firewall XP SP2FREE 0 None
Back to contents

Interpretation of results

The clear winners of our tests are Online Armor Personal Firewall 2.1.0.19 Free and Outpost Firewall Pro 2008 6.0.2162.205.402.266. Both products reached absolute scores even on their default settings. Congratulations!

Another important result of our tests is firewall scoring against FPR. FPR stands for Fake Protection Revealer. This leak-test was implemented to reveal cheating on leak-tests. FPR does nothing but unhooks user mode hooks and thus bypasses protection that relies on this kind of hooks. Our article Design of ideal personal firewall clearly says that user mode hooks can not be used for security critical features. Following firewalls uses user mode hooks improperly and sometimes only to bypass some leak-tests: FortKnox Personal Firewall 2007 2.0.205.0, Privatefirewall 5.0.8.11, Lavasoft Personal Firewall 2.0.1019.7604 (700), Sunbelt Personal Firewall 4.5.916, Kaspersky Internet Security 7.0.0.125, PC Tools Firewall Plus 3.0.0.36.

Twenty of the tested firewalls were marked with a Very poor or None anti-leak protection. This result is quite worrying because it shows that even today, when the malware programs are very sophisticated, still a lot of vendors simply do not care about the outbound connection control too much.

It should be noted that leak-tests probe only a few features of personal firewalls. If a firewall passes all leak-tests it does not mean it is perfect, bug free or secure in other aspects! However, if a personal firewall fails most of leak-testing techniques, it means that it is insecure. This does not hold for packet filters! To learn more about leak-tests, we recommend you to read our Introduction to Firewall Leak-testing.

Back to contents

Curiosities and interesting notes

Some firewalls totally failed tests made against their default settings but their results on the highest security settings were much better. Norton Internet Security 2007 is the product with the biggest difference between the default settings score and the highest security settings score. Another such product is Safety.Net.

Some products like BitDefender, F-Secure, McAfee, Panda, G DATA etc. include antivirus engines. The sad and funny thing in once is that lots of them mark leak-testing software as viruses or malware. The better engines mark leak-testing software only as potentially unwanted software, which is much better, but still it seems that these products worry about leak-tests. Why? To perform our tests against these products we had to switch antivirus engines of such products off to get real results of their anti-leak protection. Such behaviour can be also marked as cheating on leak-tests. Fortunately, it was always possible to disable the antivirus protection.

The most successful leak-test was OSfwbypass. It failed only against seven firewalls on its own and only against six when run via FPR. Good job! Another very successful leak-tests were Breakout, 3rd test of CPILSuite and PCFlank.

The least successful leak-test was LeakTest. It was able to score only against Windows XP SP2 firewall and against the default settings of a few other firewalls.

Testing Blink with FPR was not easy. Blink implements hundreds of user mode hooks in very unusual way. FPR was implemented to fix hooks that are at most 12 bytes long. We had to make a special compilation of FPR to be able to run it against Blink.

Testing ZoneAlarm Pro 7.0.337.000 was also a difficult task in some cases. ZoneAlarm implements anti-spyware scanning engine that we were not able to disable in its graphic user interface. When every single component of ZoneAlarm was turned off, some leak-tests were still forbidden to run. This was both weird and unpleasant. Why there was no chance for users to make their own decision? Anyway, some advanced techniques were used to bypass anti-spyware protection of ZoneAlarm and thus finally, all tests were performed successfully.

Another strange thing with ZoneAlarm is that it might seem that it passes PCFlank test, but in fact it does not. This leak-test tries to establish network connection with www.pcflank.com. ZoneAlarm invisibly includes this Internet address in its Spy Site Blocking list. So, if PCFlank contacted another website instead of the original one, it would bypass the protection. ZoneAlarm does not block the technique PCFlank presents, it blocks the target website which is harmless in fact.

Back to contents

Vendors' responses

We have received responses on our leak-tests from these vendors:

Agnitum Ltd. - the vendor of Outpost Firewall Pro

2007-11-09: Here is the response we have received from this vendor:

Although nobody can guarantee 100% security on a computer that is switched on, Agnitum's goal has always been to get as close to that 100% as possible. After almost two years of hard work, we're delighted with these perfect test results for our new generation of proactive protection - the Vista-compatible Outpost Firewall Pro 2008 and Outpost Security Suite Pro 20