Poll
How much system resources could your security products consume at most?
Firewall Challenge
- Introduction
- Results and comments
- Testing levels
- List of products
- My leaks
- Frequently asked questions
Results and comments
Contents:
Firewalls' ratings
The table below sorts the tested products by their total score, which is displayed in the Product score column.
The second number in this column, separated by a slash, represents a number of tests that were in the system when the given
product was tested, i.e. the number of tests with a valid test result value (other value than N/A).
This table also shows the exact version of every tested product. The Level reached column presents the highest
level that the product reached in Firewall Challenge. If it passed all levels, this number is suffixed with a plus sign.
For products that score at least 80% in Firewall Challenge, the Recommendation column contains links to the online stores
or products' webpages of the vendors that we have affiliate agreements with. If you click on any of these links and then buy
the target product or other product offered on the target webpage, we will profit from it. This is one of the ways how you can
support this project. The PDF document icon allows you to download the testing report in PDF format for the tested product.
| Product | Product score | Level reached | Protection level | Recommendation | Report | |
|---|---|---|---|---|---|---|
 |
Comodo Firewall Pro 3.0.21.329FREE | 100% / 62 | 10+ | Excellent – 100% | N/A |  |
 |
Online Armor Personal Firewall 2.1.0.119 FreeFREE | 100% / 62 | 10+ | Excellent – 100% | GET IT NOW! |
|
 |
ProSecurity 1.43 | 93% / 62 | 10 | Excellent | N/A | |
 |
Outpost Firewall Pro 2008 6.0.2302.264.0490 | 91% / 62 | 10 | Excellent | GET IT NOW! | |
 |
Kaspersky Internet Security 7.0.1.325 | 85% / 62 | 10 | Very good | GET IT NOW! |
|
 |
System Safety Monitor 2.3.0.612 | 77% / 62 | 7 | Good | Not recommended | |
 |
ZoneAlarm Pro 7.0.462.000 | 74% / 62 | 10 | Good | Not recommended | |
 |
Privatefirewall 6.0.11.30 | 65% / 62 | 7 | Poor | Not recommended | |
 |
Norton Internet Security 2008 15.5.0.23 | 32% / 62 | 4 | Very poor | Not recommended | |
 |
Jetico Personal Firewall 2.0.1.5.2216 | 29% / 62 | 3 | None | Not recommended | |
 |
Sunbelt Personal Firewall 4.5.916 | 18% / 62 | 3 | None | Not recommended | |
 |
FortKnox Personal Firewall 2008 3.0.195.0 | 16% / 62 | 2 | None | Not recommended | |
 |
Look 'n' Stop 2.06 | 15% / 62 | 2 | None | Not recommended | |
 |
Avira Premium Security Suite 8.1.00.206 | 11% / 70 | 2 | None | Not recommended | |
 |
AVG Internet Security 8.0.93 | 6% / 62 | 1 | None | Not recommended | |
 |
PC Tools Firewall Plus 3.0.1.9FREE | 6% / 62 | 1 | None | Not recommended | |
 |
McAfee Internet Security Suite 2008 9.1.108 | 6% / 70 | 1 | None | Not recommended | |
 |
ESET Smart Security 3.0.621.0 | 5% / 62 | 1 | None | Not recommended | |
 |
Rising Personal Firewall 2007 19.66.0.0 | 5% / 62 | 1 | None | Not recommended | |
 |
Windows Live OneCare 2.0.2500.22 | 5% / 62 | 1 | None | Not recommended | |
 |
BitDefender Internet Security 2008 11.0.16 | 4% / 70 | 1 | None | Not recommended | |
 |
BullGuard Internet Security 8.0.0.13 | 4% / 70 | 1 | None | Not recommended | |
 |
iolo Personal Firewall 1.5.2.7 | 3% / 62 | 1 | None | Not recommended | |
 |
Steganos Internet Security 2008 7.5.509 | 3% / 70 | 1 | None | Not recommended | |
 |
Filseclab Personal Firewall 3.0.0.8686FREE | 1% / 70 | 1 | None | Not recommended | |
Detailed results
The following links take you to pages with detailed products' results on each level. The level pages also contain important information about the given level and short information about its tests.
- Level 1 – Breakout2, Coat, ECHOtest, Kill1, Kill2, Leaktest, PerfTCP, PerfUDP, Tooleaky, Wallbreaker1, Yalta
- Level 2 – AWFT1, DNStest, Ghost, Jumper, Kill3, Kill3b, Kill6, Wallbreaker3, Wallbreaker4
- Level 3 – AWFT3, AWFT4, DNStester, Kill3f, Kill4, Kill7, SSS2, Suspend1, Thermite
- Level 4 – CopyCat, CPIL, CPILSuite1, Keylog1, Kill3e, Kill8, Kill9, SSS, Suspend2
- Level 5 – Breakout1, CPILSuite2, Crash1, Crash2, Crash3, Keylog2, Kill3c, Kill3d, VBStest
- Level 6 – CPILSuite3, DDEtest, ECHOtest2, FireHole, Flank, Keylog3, Keylog4, Kill10, Kill11, Runner
- Level 7 – BITStest, Keylog5, Keylog6, OSfwbypass, Runner2, Schedtest, SSS3
- Level 8 – Keylog7, Kill5, NewClass, Schedtest2, SockSnif, SSS4
- Level 9 – Driver Verifier
- Level 10 – BSODhook, ShadowHook
Interpretation of results
Comodo Firewall Pro 3.0.21.329 and Online Armor Personal Firewall 2.1.0.119 Free are leading with a perfect score. Both these products are free. ProSecurity 1.43 and Outpost Firewall Pro 2008 6.0.2302.264.0490 were also excellent in the tests.
The results of Jetico Personal Firewall 2.0.1.5.2216 show that Firewall Challenge testing is not only about leak-testing. Jetico v2 had excellent performance against leak-tests but this fact alone is insufficient for success in Firewall Challenge.
It also seems that Firewall Challenge tests make a big difference between really good products and the rest of the world. Most of the products are filtered in very low levels which means that they probably miss some critical features.
Vendors' responses
We have received the following responses to Firewall Challenge:
Agnitum Ltd. – the vendor of Outpost Firewall Pro
2008-03-18 (Outpost Firewall Pro scored 91% and took 4th place): We'd like to thank Matousec – Transparent Security Lab for doing research on Agnitum's product – Outpost Firewall Pro 2008. Thanks to the results revealed in the latest tests (based on the lab's new methodology), we'll make related corrections and updates in the next version of our software. Those should help us resist new threat types discovered in Matousec's report – to the benefit of security products users.
Alexey Belkin,
Chief Software Architect,
Agnitum
AVG Technologies – the vendor of AVG Internet Security
2008-04-24 (AVG Internet Security scored 6%): Thank you for including our product in your test. Our firewall was never designed to be used in a stand-alone environment, but as a part of an integrated endpoint security solution that delivers our recommended layered approach to security. We will, however, review these results and implement those changes we believe to be necessary in future upgrades.
Larry Bridwell
Global Security Strategist
AVG Technologies
Comodo Group – the vendor of Comodo Firewall Pro
2008-03-18 (Comodo Firewall Pro scored 98% and took 1st place): We do appreciate the good work. We believe security researchers such as you guys, who try to poke holes in our products, are going to make them stronger and make us give our users even better products. Keep up the good work!
Egemen TAS,
Sr. Research Scientist,
COMODO Research Labs.
Soft4Ever – the vendor of Look 'n' Stop
2008-04-08 (Look 'n' Stop 2.06 scored 15%): Look 'n' Stop Firewall is a pure firewall, which filters at TDI & NDIS levels. Look 'n' Stop is not an HIPS application and therefore is not designed to block most of the leaktests considered by the "Firewall Challenge" project from Matousec.
The Look 'n' Stop Team.
Jetico, Inc. – the vendor of Jetico Personal Firewall
2008-03-27 (Jetico Personal Firewall 2.0.1.5.2216 scored 29%): Thank you for testing our software. Results can be explained easily: JPF lacks self-protection. We shall add it in upcoming versions. We greatly appreciate your work on new test suite creation.
Nail Kaipov
Jetico, Inc.
NETGATE Technologies s.r.o. – the vendor of FortKnox Personal Firewall
2008-04-08 (FortKnox Personal Firewall 2008 3.0.195.0 scored 16%): We really appreciate testing of our product and we will try to incorporate protection for new tests in the next build releases. Thank you.
Martin Pekar
NETGATE Technologies s.r.o.
Sunbelt Software – the vendor of Sunbelt Personal Firewall
2008-03-24 (Sunbelt Personal Firewall scored 18%): Thanks for including us in your testing. Sunbelt Software is currently testing a new version of Sunbelt Personal Firewall (SPF), planned for release in Q2 2008, that addresses the majority of the leak tests that SPF 4.5.916 fails.
Phil Owens
Product Manager
Sunbelt Personal Firewall
Sunbelt Software
Tall Emu Pty Ltd – the vendor of Online Armor Personal Firewall
2008-03-18 (Online Armor Personal Firewall scored 94% and took 2nd place): We're very pleased to receive this position in the first round of Firewall Challenge tests. Matousec's work on security is a service to both users and vendors of personal firewall products and we are pleased to participate..
Mike Nash,
CEO Tall Emu Pty Ltd