On Windows 7 (or Vista) I use
Proactive Security Challenge
- Results and comments
- Testing levels
- List of products
- Archive of results
- My leaks
- Frequently asked questions
Results and comments
Proactive Security Challenge has been replaced with Proactive Security Challenge 64!
The table(s) below sort(s) the tested products by their total score, which is displayed in the Product score column.
There are two possible views of the total scores. The default view separates the results by the number of tests
that were in the system when the products were tested, i.e. the number of tests with a valid test result value (other value
than N/A). There is one table for each number of tests. The second view mixes all the results in one table.
In the second view, the Product score column consists of two numbers separated by a slash –
the actual score and the number of tests that were in the system when the given product was tested. You can switch between the views
using the link below. This table also shows the exact version of every tested product. The Level reached column presents the highest
level that the product reached in Proactive Security Challenge. If it passed all levels, this number is suffixed with a plus sign.
For products that score at least 80 % in Proactive Security Challenge, the Recommendation column contains links to the online stores
or products' webpages of the vendors that we have affiliate agreements with. If you click on any of these links and then buy
the target product or other product offered on the target webpage, we will profit from it. This is one of the ways how you can
support this project. The PDF document icon allows you to download the testing report in PDF format for the tested product.
The last column of the table shows stars for products that received Proactive Security Challenge Awards. These awards are given
to products that were tested against the latest set of tests and reached at least Very good protection level.
You can see the award logo if you put your mouse cursor over the award star. The vendors of awarded products may ask
us for copies of award logo images to be used on their web sites or products' boxes.
Products tested against the suite with 148 tests
|Product||Product score||Level reached||Protection level||Recommendation||Report||Award|
|Comodo Internet Security 5.3.176757.1236FREE||100 %||10+||Excellent – 100 %||GET IT NOW!|
|Online Solutions Security Suite 1.5.14905.0||99 %||10+||Excellent||GET IT NOW!|
|Privatefirewall 188.8.131.52FREE||98 %||10+||Excellent||N/A|
|Outpost Security Suite Free 184.108.40.20618.520.1245.401FREE||97 %||10+||Excellent||GET IT NOW!|
|Outpost Security Suite Pro 220.127.116.1191.596.1681||97 %||10+||Excellent||GET IT NOW!|
|BitDefender Internet Security 2011 18.104.22.1687||97 %||10+||Excellent||GET IT NOW!|
|Kaspersky Internet Security 2012 22.214.171.1244||93 %||10+||Excellent||GET IT NOW!|
|Malware Defender 2.7.3.0002FREE||91 %||10||Excellent||N/A|
|PC Tools Internet Security 2011 126.96.36.1995||90 %||10+||Very good||GET IT NOW!|
|Jetico Personal Firewall 188.8.131.52.2451||88 %||10||Very good||N/A|
|ZoneAlarm Extreme Security 2012 10.0.250.000||72 %||9||Good||Not recommended||–|
|Rising Internet Security 2011 23.00.41.42||34 %||5||Very poor||Not recommended||–|
|CA Internet Security Suite Plus 2011 184.108.40.2069||30 %||5||Very poor||Not recommended||–|
|Norton Internet Security 2012 220.127.116.11||20 %||3||None||Not recommended||–|
|avast! Internet Security 6.0.1000||15 %||3||None||Not recommended||–|
|Dr.Web Security Space Pro 6.0.2.07290||14 %||3||None||Not recommended||–|
|F-Secure Internet Security 2011 10.51.106||9 %||2||None||Not recommended||–|
|Trend Micro Internet Security Pro 2010 17.50.1647.0000||9 %||2||None||Not recommended||–|
|FortKnox Personal Firewall 18.104.22.168||7 %||2||None||Not recommended||–|
|ZoneAlarm Free Firewall 9.2.076.000FREE||7 %||2||None||Not recommended||–|
|ESET Smart Security 22.214.171.124||6 %||2||None||Not recommended||–|
|AVG Internet Security 2011 10.0.1153||3 %||1||None||Not recommended||–|
|Avira Premium Security Suite 10.0.0.608||3 %||1||None||Not recommended||–|
|Look 'n' Stop 2.07||3 %||1||None||Not recommended||–|
|Sunbelt Personal Firewall 4.6.1861.0||3 %||1||None||Not recommended||–|
|G Data InternetSecurity 2011 126.96.36.199||2 %||1||None||Not recommended||–|
|McAfee Internet Security 2011 11.5.141||2 %||1||None||Not recommended||–|
|Panda Internet Security 2011 16.00.00||2 %||1||None||Not recommended||–|
|TrustPort Internet Security 2011 188.8.131.5284||2 %||1||None||Not recommended||–|
|Product||Product score||Level reached||Protection level||Recommendation||Report||Award|
|Comodo Internet Security 5.3.176757.1236FREE||100 % / 148||10+||Excellent – 100 %||GET IT NOW!|
|Online Solutions Security Suite 1.5.14905.0||99 % / 148||10+||Excellent||GET IT NOW!|
|Privatefirewall 184.108.40.206FREE||98 % / 148||10+||Excellent||N/A|
|Outpost Security Suite Free 220.127.116.1118.520.1245.401FREE||97 % / 148||10+||Excellent||GET IT NOW!|
|Outpost Security Suite Pro 18.104.22.16891.596.1681||97 % / 148||10+||Excellent||GET IT NOW!|
|BitDefender Internet Security 2011 22.214.171.1247||97 % / 148||10+||Excellent||GET IT NOW!|
|Kaspersky Internet Security 2012 126.96.36.1994||93 % / 148||10+||Excellent||GET IT NOW!|
|Malware Defender 2.7.3.0002FREE||91 % / 148||10||Excellent||N/A|
|PC Tools Internet Security 2011 188.8.131.525||90 % / 148||10+||Very good||GET IT NOW!|
|Jetico Personal Firewall 184.108.40.206.2451||88 % / 148||10||Very good||N/A|
|ZoneAlarm Extreme Security 2012 10.0.250.000||72 % / 148||9||Good||Not recommended||–|
|Rising Internet Security 2011 23.00.41.42||34 % / 148||5||Very poor||Not recommended||–|
|CA Internet Security Suite Plus 2011 220.127.116.119||30 % / 148||5||Very poor||Not recommended||–|
|Norton Internet Security 2012 18.104.22.168||20 % / 148||3||None||Not recommended||–|
|avast! Internet Security 6.0.1000||15 % / 148||3||None||Not recommended||–|
|Dr.Web Security Space Pro 6.0.2.07290||14 % / 148||3||None||Not recommended||–|
|F-Secure Internet Security 2011 10.51.106||9 % / 148||2||None||Not recommended||–|
|Trend Micro Internet Security Pro 2010 17.50.1647.0000||9 % / 148||2||None||Not recommended||–|
|FortKnox Personal Firewall 22.214.171.124||7 % / 148||2||None||Not recommended||–|
|ZoneAlarm Free Firewall 9.2.076.000FREE||7 % / 148||2||None||Not recommended||–|
|ESET Smart Security 126.96.36.199||6 % / 148||2||None||Not recommended||–|
|AVG Internet Security 2011 10.0.1153||3 % / 148||1||None||Not recommended||–|
|Avira Premium Security Suite 10.0.0.608||3 % / 148||1||None||Not recommended||–|
|Look 'n' Stop 2.07||3 % / 148||1||None||Not recommended||–|
|Sunbelt Personal Firewall 4.6.1861.0||3 % / 148||1||None||Not recommended||–|
|G Data InternetSecurity 2011 188.8.131.52||2 % / 148||1||None||Not recommended||–|
|McAfee Internet Security 2011 11.5.141||2 % / 148||1||None||Not recommended||–|
|Panda Internet Security 2011 16.00.00||2 % / 148||1||None||Not recommended||–|
|TrustPort Internet Security 2011 184.108.40.20684||2 % / 148||1||None||Not recommended||–|
The following links take you to pages with detailed products' results on each level. The level pages also contain important information about the given level and short information about its tests.
- Level 1 – Autorun1, Autorun3, Breakout2, Coat, ECHOtest, FileDel2, Kill1, Kill2, Leaktest, Tooleaky, Wallbreaker1, Yalta
- Level 2 – Autorun12, Autorun2, Autorun20, Autorun30, AWFT1, DNStest, FileMov2, Ghost, HostsBlock, Jumper, Kill3, Kill3b, Kill6, RegDel1, Wallbreaker3, Wallbreaker4
- Level 3 – Autorun16, Autorun24, Autorun31, Autorun4, AWFT3, AWFT4, DNStester, FileRep1, Kernel1, Kill3f, Kill4, Kill7, RegSet1, SSS2, Suspend1, Thermite, Wallbreaker2
- Level 4 – Autorun14, Autorun17, Autorun26, Autorun36, Autorun37, Autorun6, Autorun9, CopyCat, CPIL, CPILSuite1, FileRep2, Inject2, Inject3, Kernel1b, Keylog1, Kill3e, Kill8, Kill9, SSS, Suspend2
- Level 5 – Autorun15, Autorun18, Autorun21, Autorun28, Autorun5, Breakout1, CPILSuite2, Crash1, Crash2, Crash3, Crash4, FileWri1, Kernel2, Kernel3, Keylog2, Kill3c, Kill3d, RegDel2, Svckill, VBStest
- Level 6 – Autorun22, Autorun25, Autorun27, Autorun29, Autorun32, Autorun7, CPILSuite3, Crash5, Crash6, DDEtest, ECHOtest2, FileWri2, FireHole, Flank, Kernel4, Keylog3, Keylog4, Kill10, Kill11, Runner
- Level 7 – Autorun10, Autorun19, Autorun33, Autorun35, Autorun8, BITStest, Crash4b, FileDel1, FileMov1, FileWri3, FireHole2, Inject1, Keylog5, Keylog6, Kill12, OSfwbypass, RegAcc1, Runner2, Schedtest, SSS3
- Level 8 – Autorun11, Autorun13, Autorun23, Autorun34, FileDel3, FileOpn1, FileOpn2, Kernel4b, Kernel5, Kernel5b, Keylog7, Kill5, NewClass, Schedtest2, SockSnif, SSS4
- Level 9 – Crash7, Driver Verifier, FileAcc1, FileCtl1, FileWri4
- Level 10 – BSODhook, ShadowHook
Interpretation of results
The best product tested against the latest set of tests and the Proactive Security Challenge leader is currently Comodo Internet Security 5.3.176757.1236 with the perfect 100% score. The leader is closely followed by Online Solutions Security Suite 1.5.14905.0 with 99 %, Privatefirewall 220.127.116.11 with 98 %, Outpost Security Suite Pro 18.104.22.16891.596.1681, Outpost Security Suite Free 22.214.171.12418.520.1245.401, and BitDefender Internet Security 2011 126.96.36.1997 with 97 %. Kaspersky Internet Security 2012 188.8.131.524 with 93 % and Malware Defender 2.6.0 with 91 % come next. These are the only products that reached the Excellent protection level. PC Tools Internet Security 2011 184.108.40.2065 with 90 %, Jetico Personal Firewall 220.127.116.11.2451 with 88% come next. All these products have been awarded with Proactive Security Challenge awards.
It seems that Proactive Security Challenge tests make a big difference between really good products and the rest of the world. Most of the products are filtered in very low levels which means that they probably miss some critical features.
However, it is crucial to know what does it mean if a product succeeds in our tests and what does it mean if it fails. Before you start interpreting the results, you should be familiar with the information on the index page, especially with the methodology and rules. You should also know what kind of products do we test before you start to interpret the results. We have received a lot of reactions from people who are not familiar with that information and simply do not understand the results and misinterpret them. All the tested products have one common feature – the application-based security model. In combination with their packet filtering capabilities, the tested products attempt to block attacks from other machines on the network as well as attacks performed by malicious codes that might run inside the protected machine. This is definitely not an unusual situation. People who use email clients, instant messengers, or web browsers face attacks that exploit the vulnerabilities in this kind of software very often. It happens that a malicious code gets inside the machine. And then it may try to install itself silently to the system, to steal users' data or sniff their passwords, or to join the target machine to the botnet. This is what the products we test want to prevent. This is why they are used. The problem is that although the goal is common, not all the products implement a sufficient protection.
We require the products tested in Proactive Security Challenge to prevent data and identity theft. They should also implement a packet filter functionality to prevent direct online attacks – i.e. not to let the malware get in. The products should control the software installed on the computer to prevent the malware to integrate itself into the operating system. Then the malware should not be able to get the user's private data, thus anti-sniffing, anti-keylogging and personal data protection features should be implemented too. And even if the malware succeeded to collect the information it should not be allowed to send it outside the protected system and this means an implementation of the outbound network traffic control. To achieve all these is much harder task than it seems. The protection system also has to prevent attacking trusted processes and other components in the system. Otherwise, the malware would be able to use trusted parts of the system to integrate into the operating system, to collect or steal sensitive data and/or to send the data outside the system without being noticed. So the next feature that is required here is a control of untrusted processes' activities and that is the hardest task for the tested products. It also includes an implementation of self-protection mechanisms because the malware should not be able to terminate the protection, which implies some other features to be implemented and so on.
So, what does it mean if the product fails even the most basic tests of our challenge? It means that it is unable to do what its vendor claims it can. Such a product can hardly protect you against the mentioned threats. On the other hand, if the product succeeds in all our tests, it does not mean that it is perfect. Our tests are focused on the security and stability, but there are many other aspects important for the users like performance, hardware requirements, easy to use, availability of support, price, vendor's reaction time to new threats etc. It should be also noted that although our testing suite is quite large, it is not complete and there are many other ways to bypass the tested products. Moreover, the products are tested on systems with almost no third party software, which limits the stability tests we perform. We are working constantly on extending the suite to be able to provide more accurate information about the security and stability of the tested products. If the tested product fails only a few tests in our challenge, it still might be a great product. This is why we can recommend, from the security point of view, the products that reached at least 80% score in the challenge. You should try them yourself and choose the one best for you, the one that you would be happy with, the one you would be able to configure and use everyday.
We have received the following responses to Proactive Security Challenge:
Agnitum Ltd. – the vendor of Outpost Firewall Pro
2009-02-17 (Outpost Security Suite Pro 2009 2009 6.5.2514.381.0685 scored 93 % and took shared 1st place):
We'd like to thank Matousec lab for another thorough examination of
Outpost security, this time the all-in-one solution - Outpost Security
Suite Pro 2009. By becoming one of the leaders in this latest test we
demonstrated the futility of opposing our standalone firewall product
and security suite when it comes to proactive protection and resistance
93 % is a good result but we regret that level 9 turned out to be an obstacle for the product whereas our internal testing revealed that level 10 was passed with flying colors. Nevertheless, we are positive that improving the product's performance on level 9 is not a question of whether but when. Users and security experts may rest assured we'll address this issue in the nearest future and come up with a relevant update.
2008-07-04 (Outpost Firewall Pro 2008 6.0.2302.264.0490 scored 99 % and took 1st place): As always, we are grateful to the Matousec team for their job! The latest tests show there is still a gap for improvements in Outpost's proactive protection, but it's just a 1% gap, and we feel fully capable to accomplish the 100% result shortly.
PR Manager, Agnitum Ltd.
2008-03-18 (Outpost Firewall Pro scored 91 % and took 4th place): We'd like to thank Matousec – Transparent Security Lab for doing research on Agnitum's product – Outpost Firewall Pro 2008. Thanks to the results revealed in the latest tests (based on the lab's new methodology), we'll make related corrections and updates in the next version of our software. Those should help us resist new threat types discovered in Matousec's report – to the benefit of security products users.
Chief Software Architect,
AVG Technologies – the vendor of AVG Internet Security
2008-04-24 (AVG Internet Security 8.0.93 scored 6 %): Thank you for including our product in your test. Our firewall was never designed to be used in a stand-alone environment, but as a part of an integrated endpoint security solution that delivers our recommended layered approach to security. We will, however, review these results and implement those changes we believe to be necessary in future upgrades.
Global Security Strategist
BitDefender – the vendor of BitDefender Internet Security
2008-06-07 (BitDefender Internet Security 2008 11.0.16 scored 4 %): Thank you for testing BitDefender Total Security 2008. Bitdefender Firewall doesn't exist as a stand-alone product because it has been designed to work and be sold with the Antivirus as a package. These tools you have been using in your test are automatically caught by our Antivirus module. We will analyze the results in this test and plan to cover this type of test in the future.
Our response: According to VirusTotal, not even the latest version of BitDefender detects our tests. But it is a good thing that it does not detect them because they are not malware and you would scare your users with false positive alerts if they were marked as infected.
Comodo Security Solutions, Inc. – the vendor of Comodo Internet Security
2010-05-01 (Comodo Internet Security 4.0.141842.828 scored 100 % and took 1st place): With the latest set of tests, you have proven yourself as the only testing organization who is capable of assessing the prevention capabilities of the existing security suites by creating targeted dynamic tests rather than relying on the "outdated" static tests.
Director, Desktop Security Products
2009-10-10 (Comodo Internet Security 3.12.111745.560 scored 100 % and took 1st place): Thank you very much for testing our product. Keep up the good work!
2009-04-08 (Comodo Internet Security 3.8.65951.477 scored 96 % and took 1st place): Thank you for testing our product and keep up the good work of testing the products frequently.
2008-12-01 (Comodo Internet Security 3.5.55810.432 scored 90 % and took 2nd place): Thank you for testing our product. We are happy to report that the bug identified during your tests is fixed.
Sr. Research Scientist,
2008-05-21 (Comodo Firewall Pro 18.104.22.1689 scored 95 % and took 1st place): Thank you very much for pointing out the "windows shutdown race condition" bug introduced with the tested version, which is the main reason for failing in SSS tests. We will be addressing this with the planned release on 05/20/2008. Keep up the good work.
Sr. Research Scientist,
COMODO CP Inc
2008-03-18 (Comodo Firewall Pro 22.214.171.1248 scored 98 % and took 1st place): We do appreciate the good work. We believe security researchers such as you guys, who try to poke holes in our products, are going to make them stronger and make us give our users even better products. Keep up the good work!
Sr. Research Scientist,
COMODO Research Labs.
Emsi Software GmbH – the vendor of Mamutu
2008-11-29 (Mamutu 126.96.36.199 scored 2 %): In our opinion Mamutu is completely misplaced in this test. Mamutu is not a firewall, but a behavior blocker, designed to detect and block real malware samples only, not to pass a firewall leaktest. For us, the test results are useless because the product was tested for features that Behavior Blockers are not intended to provide in general. In oposite to Matousec, we think that Firewalls, Behavior Blockers and HIPS are not the same type of software and therefore they can not be tested and compared as they were the same.
Emsi Software GmbH
Our response: We are testing a specific kind of security software which must meet some fixed criteria in order to be included to our project. The main criterion is to implement a process-based security. Proactive Security Challenge is designed to test Internet security suites, personal firewalls, HIPS products, behavior blockers and other behavior based systems. Mamutu met all the required criteria and hence there was no why not to include Mamutu to our project after we received several requests from our visitors. All the products included to our project implement similar features. These security features are tested in our project. We believe that using a set of open tests is the only objective way to compare all the products that implement the very same features. There are various tests used in Proactive Security Challenge, only a part of the used testing suite is based on leak-tests.
Filseclab Corporation – the vendor of Filseclab Personal Firewall
2008-06-18 (Filseclab Personal Firewall 188.8.131.5286 scored 1 %): Thank you for testing our firewall. Filseclab Personal Firewall 184.108.40.20686 is designed for inbound protection, it is worse for outbound protection, we recommend our users to use the Anti-virus or Anti-spyware software to protect the outbound. However, we will make a new version to improve those features in the end of this year.
Jetico, Inc. – the vendor of Jetico Personal Firewall
2008-12-04 (Jetico Personal Firewall 220.127.116.11.2311 scored 72 %): Thank you for making SSTS available. We glad to see that the test suite is growing. Keep up your good work.
2008-07-07 (Jetico Personal Firewall 18.104.22.168.2264 scored 78 %): We do appreciate your testing efforts. Your test suite reveals weaknesses in security software and helps vendors to make more reliable products. As soon as you added performance tests, please publish full technical specifications of your test system along with test results.
Our response: Currently, we have only two performance tests in FWC, which are designed to compare the network performance of the products on 100 Mbit full duplex Ethernet. These tests are performed on the testing system without the tested product installed and also with the tested product installed. The measured results are compared then.
2008-03-27 (Jetico Personal Firewall 22.214.171.124.2216 scored 29 %): Thank you for testing our software. Results can be explained easily: JPF lacks self-protection. We shall add it in upcoming versions. We greatly appreciate your work on new test suite creation.
Lavasoft – the vendor of Lavasoft Personal Firewall
2008-05-20 (Lavasoft Personal Firewall 3.0.2293.8822 scored 70 %): Thank you very much for your inclusion of the Lavasoft Personal Firewall 3.0 in the Matousec research. Upon review of the results, we were surprised to find that the Lavasoft Personal Firewall program received a 'good' rating and could not be a recommended firewall, whereas our firewall technology partner, Agnitum, received an 'excellent' score with recommendations for the same firewall technology. We hope that you will take this under consideration with your next round of research, and continue your good work.
Our response: Outpost Firewall Pro was tested when there were 62 tests in the system, Lavasoft Personal Firewall was tested when there were 73 tests in the system. This is the first major difference. The second one is that Outpost Firewall Pro usually contains a newer version of their engine and so Outpost firewall may pass a few more tests than Lavasoft firewall. To mitigate the differences, we intend to retest Outpost Firewall Pro in the near future.
NETGATE Technologies s.r.o. – the vendor of FortKnox Personal Firewall
2008-04-08 (FortKnox Personal Firewall 2008 126.96.36.199 scored 16 %): We really appreciate testing of our product and we will try to incorporate protection for new tests in the next build releases. Thank you.
NETGATE Technologies s.r.o.
PC Tools Pty Ltd – the vendor of PC Tools Firewall Plus
2009-09-11 (PC Tools Firewall Plus 188.8.131.52 scored 99 % and took shared 1st place): Thank you for the detailed and thorough test. We are pleased with our results and are now working on resolving the issues found in the Matousec tests.
2008-09-05 (PC Tools Firewall Plus 184.108.40.206 scored 85 %): We would like to thank Matousec for conducting the Proactive Security Challengetests. PC Tools has invested substantial resources to improve its support in this latest version, and will continue to do so in the future. We are pleased to hear that in this test PC Tools Firewall Plus came out as the fastest firewall. We recognize the importance of minimizing the firewalls influence on the user experience which is a testament to this latest test result. With regard to Keyloggers, PC Tools Firewall Plus does not focus on the detection of keyloggers as these types of threats fall into the Anti-Spyware category of protection. However, there are circumstances in which PC Tools Firewall Plus will detect a Keylogger, this being the point at which it attempts to access the Internet.
PC Tools Firewall Plus Team Leader
PWI, Inc. – the vendor of Privatefirewall
2011-10-07 (Privatefirewall 220.127.116.11 scored 98 % and took 3rd place): We are naturally very pleased with the latest PSC test results, but will investigate and remedy the three outstanding identified flaws. Privacyware remains committed to continually strengthening Privatefirewall for our direct and OEM customers and the effort and dedication of the entire Matousec team has been instrumental in helping us do so. No other service provider possesses the level of expertise regarding leak, spy, malware, termination and so many other attack techniques - or the breadth of knowledge of the relevant software products to effectively assess these vulnerabilities. We extend our deepest gratitude and admiration for the role Matousec plays in helping vendors like Privacyware develop better products.
CEO - Privacyware: www.privacyware.com
2008-12-03 (Privatefirewall 18.104.22.168 scored 82 %): Thank you for the latest report and for continuing to push the envelope on desktop defense quality and performance. We will review the new tests and plan to modify Privatefirewall to correct the deficiencies we are able to validate and address.
CEO - Privacyware: www.privacyware.com
Soft4Ever – the vendor of Look 'n' Stop
2008-04-08 (Look 'n' Stop 2.06 scored 15 %): Look 'n' Stop Firewall is a pure firewall, which filters at TDI & NDIS levels. Look 'n' Stop is not an HIPS application and therefore is not designed to block most of the leaktests considered by the "Proactive Security Challenge" project from Matousec.
The Look 'n' Stop Team.
Sunbelt Software – the vendor of Sunbelt Personal Firewall
2008-03-24 (Sunbelt Personal Firewall 4.5.916 scored 18 %): Thanks for including us in your testing. Sunbelt Software is currently testing a new version of Sunbelt Personal Firewall (SPF), planned for release in Q2 2008, that addresses the majority of the leak tests that SPF 4.5.916 fails.
Sunbelt Personal Firewall