matousec.com (site map)

Poll

On Windows 7 (or Vista) I use

  unlimited administrator's account (58.05%)

  limited administrator's account (16.34%)

  common user's account (12.76%)

  nothing (I do not use Win 7/Vista) (13.72%)

more

results

Proactive Security Challenge

Results and comments





Proactive Security Challenge has been replaced with Proactive Security Challenge 64!



Contents:


Back to contents

Products' ratings

The table(s) below sort(s) the tested products by their total score, which is displayed in the Product score column. There are two possible views of the total scores. The default view separates the results by the number of tests that were in the system when the products were tested, i.e. the number of tests with a valid test result value (other value than N/A). There is one table for each number of tests. The second view mixes all the results in one table. In the second view, the Product score column consists of two numbers separated by a slash – the actual score and the number of tests that were in the system when the given product was tested. You can switch between the views using the link below. This table also shows the exact version of every tested product. The Level reached column presents the highest level that the product reached in Proactive Security Challenge. If it passed all levels, this number is suffixed with a plus sign. For products that score at least 80 % in Proactive Security Challenge, the Recommendation column contains links to the online stores or products' webpages of the vendors that we have affiliate agreements with. If you click on any of these links and then buy the target product or other product offered on the target webpage, we will profit from it. This is one of the ways how you can support this project. The PDF document icon allows you to download the testing report in PDF format for the tested product. The last column of the table shows stars for products that received Proactive Security Challenge Awards. These awards are given to products that were tested against the latest set of tests and reached at least Very good protection level. You can see the award logo if you put your mouse cursor over the award star. The vendors of awarded products may ask us for copies of award logo images to be used on their web sites or products' boxes.




Products tested against the suite with 148 tests


Product Product score Level reached Protection level Recommendation Report Award
ico Comodo Internet Security 5.3.176757.1236FREE 100 % 10+ Excellent – 100 % GET IT NOW! pdf
ico Online Solutions Security Suite 1.5.14905.0 99 % 10+ Excellent GET IT NOW! pdf
ico Privatefirewall 7.0.25.4FREE 98 % 10+ Excellent N/A pdf
ico Outpost Security Suite Free 7.0.4.3418.520.1245.401FREE 97 % 10+ Excellent GET IT NOW! pdf
ico Outpost Security Suite Pro 7.5.1.3791.596.1681 97 % 10+ Excellent GET IT NOW! pdf
ico BitDefender Internet Security 2011 14.0.30.357 97 % 10+ Excellent GET IT NOW! pdf
ico Kaspersky Internet Security 2012 12.0.0.374 93 % 10+ Excellent GET IT NOW! pdf
ico Malware Defender 2.7.3.0002FREE 91 % 10 Excellent N/A pdf
ico PC Tools Internet Security 2011 8.0.0.655 90 % 10+ Very good GET IT NOW! pdf
ico Jetico Personal Firewall 2.1.0.10.2451 88 % 10 Very good N/A pdf
ico ZoneAlarm Extreme Security 2012 10.0.250.000 72 % 9 Good Not recommended pdf
ico Rising Internet Security 2011 23.00.41.42 34 % 5 Very poor Not recommended pdf
ico CA Internet Security Suite Plus 2011 7.0.0.279 30 % 5 Very poor Not recommended pdf
ico Norton Internet Security 2012 19.1.1.3 20 % 3 None Not recommended pdf
ico avast! Internet Security 6.0.1000 15 % 3 None Not recommended pdf
ico Dr.Web Security Space Pro 6.0.2.07290 14 % 3 None Not recommended pdf
ico F-Secure Internet Security 2011 10.51.106 9 % 2 None Not recommended pdf
ico Trend Micro Internet Security Pro 2010 17.50.1647.0000 9 % 2 None Not recommended pdf
ico FortKnox Personal Firewall 6.0.205.0 7 % 2 None Not recommended pdf
ico ZoneAlarm Free Firewall 9.2.076.000FREE 7 % 2 None Not recommended pdf
ico ESET Smart Security 4.2.64.12 6 % 2 None Not recommended pdf
ico AVG Internet Security 2011 10.0.1153 3 % 1 None Not recommended pdf
ico Avira Premium Security Suite 10.0.0.608 3 % 1 None Not recommended pdf
ico Look 'n' Stop 2.07 3 % 1 None Not recommended pdf
ico Sunbelt Personal Firewall 4.6.1861.0 3 % 1 None Not recommended pdf
ico G Data InternetSecurity 2011 21.1.1.0 2 % 1 None Not recommended pdf
ico McAfee Internet Security 2011 11.5.141 2 % 1 None Not recommended pdf
ico Panda Internet Security 2011 16.00.00 2 % 1 None Not recommended pdf
ico TrustPort Internet Security 2011 11.0.0.4584 2 % 1 None Not recommended pdf


Mixed results


Product Product score Level reached Protection level Recommendation Report Award
ico Comodo Internet Security 5.3.176757.1236FREE 100 % / 148 10+ Excellent – 100 % GET IT NOW! pdf
ico Online Solutions Security Suite 1.5.14905.0 99 % / 148 10+ Excellent GET IT NOW! pdf
ico Privatefirewall 7.0.25.4FREE 98 % / 148 10+ Excellent N/A pdf
ico Outpost Security Suite Free 7.0.4.3418.520.1245.401FREE 97 % / 148 10+ Excellent GET IT NOW! pdf
ico Outpost Security Suite Pro 7.5.1.3791.596.1681 97 % / 148 10+ Excellent GET IT NOW! pdf
ico BitDefender Internet Security 2011 14.0.30.357 97 % / 148 10+ Excellent GET IT NOW! pdf
ico Kaspersky Internet Security 2012 12.0.0.374 93 % / 148 10+ Excellent GET IT NOW! pdf
ico Malware Defender 2.7.3.0002FREE 91 % / 148 10 Excellent N/A pdf
ico PC Tools Internet Security 2011 8.0.0.655 90 % / 148 10+ Very good GET IT NOW! pdf
ico Jetico Personal Firewall 2.1.0.10.2451 88 % / 148 10 Very good N/A pdf
ico ZoneAlarm Extreme Security 2012 10.0.250.000 72 % / 148 9 Good Not recommended pdf
ico Rising Internet Security 2011 23.00.41.42 34 % / 148 5 Very poor Not recommended pdf
ico CA Internet Security Suite Plus 2011 7.0.0.279 30 % / 148 5 Very poor Not recommended pdf
ico Norton Internet Security 2012 19.1.1.3 20 % / 148 3 None Not recommended pdf
ico avast! Internet Security 6.0.1000 15 % / 148 3 None Not recommended pdf
ico Dr.Web Security Space Pro 6.0.2.07290 14 % / 148 3 None Not recommended pdf
ico F-Secure Internet Security 2011 10.51.106 9 % / 148 2 None Not recommended pdf
ico Trend Micro Internet Security Pro 2010 17.50.1647.0000 9 % / 148 2 None Not recommended pdf
ico FortKnox Personal Firewall 6.0.205.0 7 % / 148 2 None Not recommended pdf
ico ZoneAlarm Free Firewall 9.2.076.000FREE 7 % / 148 2 None Not recommended pdf
ico ESET Smart Security 4.2.64.12 6 % / 148 2 None Not recommended pdf
ico AVG Internet Security 2011 10.0.1153 3 % / 148 1 None Not recommended pdf
ico Avira Premium Security Suite 10.0.0.608 3 % / 148 1 None Not recommended pdf
ico Look 'n' Stop 2.07 3 % / 148 1 None Not recommended pdf
ico Sunbelt Personal Firewall 4.6.1861.0 3 % / 148 1 None Not recommended pdf
ico G Data InternetSecurity 2011 21.1.1.0 2 % / 148 1 None Not recommended pdf
ico McAfee Internet Security 2011 11.5.141 2 % / 148 1 None Not recommended pdf
ico Panda Internet Security 2011 16.00.00 2 % / 148 1 None Not recommended pdf
ico TrustPort Internet Security 2011 11.0.0.4584 2 % / 148 1 None Not recommended pdf

Detailed results

The following links take you to pages with detailed products' results on each level. The level pages also contain important information about the given level and short information about its tests.


Interpretation of results

The best product tested against the latest set of tests and the Proactive Security Challenge leader is currently Comodo Internet Security 5.3.176757.1236 with the perfect 100% score. The leader is closely followed by Online Solutions Security Suite 1.5.14905.0 with 99 %, Privatefirewall 7.0.25.4 with 98 %, Outpost Security Suite Pro 7.5.1.3791.596.1681, Outpost Security Suite Free 7.0.4.3418.520.1245.401, and BitDefender Internet Security 2011 14.0.30.357 with 97 %. Kaspersky Internet Security 2012 12.0.0.374 with 93 % and Malware Defender 2.6.0 with 91 % come next. These are the only products that reached the Excellent protection level. PC Tools Internet Security 2011 8.0.0.655 with 90 %, Jetico Personal Firewall 2.1.0.10.2451 with 88% come next. All these products have been awarded with Proactive Security Challenge awards.

It seems that Proactive Security Challenge tests make a big difference between really good products and the rest of the world. Most of the products are filtered in very low levels which means that they probably miss some critical features.

However, it is crucial to know what does it mean if a product succeeds in our tests and what does it mean if it fails. Before you start interpreting the results, you should be familiar with the information on the index page, especially with the methodology and rules. You should also know what kind of products do we test before you start to interpret the results. We have received a lot of reactions from people who are not familiar with that information and simply do not understand the results and misinterpret them. All the tested products have one common feature – the application-based security model. In combination with their packet filtering capabilities, the tested products attempt to block attacks from other machines on the network as well as attacks performed by malicious codes that might run inside the protected machine. This is definitely not an unusual situation. People who use email clients, instant messengers, or web browsers face attacks that exploit the vulnerabilities in this kind of software very often. It happens that a malicious code gets inside the machine. And then it may try to install itself silently to the system, to steal users' data or sniff their passwords, or to join the target machine to the botnet. This is what the products we test want to prevent. This is why they are used. The problem is that although the goal is common, not all the products implement a sufficient protection.

We require the products tested in Proactive Security Challenge to prevent data and identity theft. They should also implement a packet filter functionality to prevent direct online attacks – i.e. not to let the malware get in. The products should control the software installed on the computer to prevent the malware to integrate itself into the operating system. Then the malware should not be able to get the user's private data, thus anti-sniffing, anti-keylogging and personal data protection features should be implemented too. And even if the malware succeeded to collect the information it should not be allowed to send it outside the protected system and this means an implementation of the outbound network traffic control. To achieve all these is much harder task than it seems. The protection system also has to prevent attacking trusted processes and other components in the system. Otherwise, the malware would be able to use trusted parts of the system to integrate into the operating system, to collect or steal sensitive data and/or to send the data outside the system without being noticed. So the next feature that is required here is a control of untrusted processes' activities and that is the hardest task for the tested products. It also includes an implementation of self-protection mechanisms because the malware should not be able to terminate the protection, which implies some other features to be implemented and so on.

So, what does it mean if the product fails even the most basic tests of our challenge? It means that it is unable to do what its vendor claims it can. Such a product can hardly protect you against the mentioned threats. On the other hand, if the product succeeds in all our tests, it does not mean that it is perfect. Our tests are focused on the security and stability, but there are many other aspects important for the users like performance, hardware requirements, easy to use, availability of support, price, vendor's reaction time to new threats etc. It should be also noted that although our testing suite is quite large, it is not complete and there are many other ways to bypass the tested products. Moreover, the products are tested on systems with almost no third party software, which limits the stability tests we perform. We are working constantly on extending the suite to be able to provide more accurate information about the security and stability of the tested products. If the tested product fails only a few tests in our challenge, it still might be a great product. This is why we can recommend, from the security point of view, the products that reached at least 80% score in the challenge. You should try them yourself and choose the one best for you, the one that you would be happy with, the one you would be able to configure and use everyday.


Vendors' responses

We have received the following responses to Proactive Security Challenge:

Agnitum Ltd. – the vendor of Outpost Firewall Pro

2009-02-17 (Outpost Security Suite Pro 2009 2009 6.5.2514.381.0685 scored 93 % and took shared 1st place): We'd like to thank Matousec lab for another thorough examination of Outpost security, this time the all-in-one solution - Outpost Security Suite Pro 2009. By becoming one of the leaders in this latest test we demonstrated the futility of opposing our standalone firewall product and security suite when it comes to proactive protection and resistance to leaktests.

93 % is a good result but we regret that level 9 turned out to be an obstacle for the product whereas our internal testing revealed that level 10 was passed with flying colors. Nevertheless, we are positive that improving the product's performance on level 9 is not a question of whether but when. Users and security experts may rest assured we'll address this issue in the nearest future and come up with a relevant update.

Maxim Korobtsev,
CTO, Agnitum

2008-07-04 (Outpost Firewall Pro 2008 6.0.2302.264.0490 scored 99 % and took 1st place): As always, we are grateful to the Matousec team for their job! The latest tests show there is still a gap for improvements in Outpost's proactive protection, but it's just a 1% gap, and we feel fully capable to accomplish the 100% result shortly.

Pavel Goryakin,
PR Manager, Agnitum Ltd.

2008-03-18 (Outpost Firewall Pro scored 91 % and took 4th place): We'd like to thank Matousec – Transparent Security Lab for doing research on Agnitum's product – Outpost Firewall Pro 2008. Thanks to the results revealed in the latest tests (based on the lab's new methodology), we'll make related corrections and updates in the next version of our software. Those should help us resist new threat types discovered in Matousec's report – to the benefit of security products users.

Alexey Belkin,
Chief Software Architect,
Agnitum

AVG Technologies – the vendor of AVG Internet Security

2008-04-24 (AVG Internet Security 8.0.93 scored 6 %): Thank you for including our product in your test. Our firewall was never designed to be used in a stand-alone environment, but as a part of an integrated endpoint security solution that delivers our recommended layered approach to security. We will, however, review these results and implement those changes we believe to be necessary in future upgrades.

Larry Bridwell
Global Security Strategist
AVG Technologies

BitDefender – the vendor of BitDefender Internet Security

2008-06-07 (BitDefender Internet Security 2008 11.0.16 scored 4 %): Thank you for testing BitDefender Total Security 2008. Bitdefender Firewall doesn't exist as a stand-alone product because it has been designed to work and be sold with the Antivirus as a package. These tools you have been using in your test are automatically caught by our Antivirus module. We will analyze the results in this test and plan to cover this type of test in the future.

Rudi-Gabriel Bedy
BitDefender

Our response: According to VirusTotal, not even the latest version of BitDefender detects our tests. But it is a good thing that it does not detect them because they are not malware and you would scare your users with false positive alerts if they were marked as infected.

Comodo Security Solutions, Inc. – the vendor of Comodo Internet Security

2010-05-01 (Comodo Internet Security 4.0.141842.828 scored 100 % and took 1st place): With the latest set of tests, you have proven yourself as the only testing organization who is capable of assessing the prevention capabilities of the existing security suites by creating targeted dynamic tests rather than relying on the "outdated" static tests.

Egemen TAS,
Director, Desktop Security Products
COMODO

2009-10-10 (Comodo Internet Security 3.12.111745.560 scored 100 % and took 1st place): Thank you very much for testing our product. Keep up the good work!

Egemen TAS,
Project Manager,
COMODO

2009-04-08 (Comodo Internet Security 3.8.65951.477 scored 96 % and took 1st place): Thank you for testing our product and keep up the good work of testing the products frequently.

Egemen TAS,
Project Manager,
COMODO

2008-12-01 (Comodo Internet Security 3.5.55810.432 scored 90 % and took 2nd place): Thank you for testing our product. We are happy to report that the bug identified during your tests is fixed.

Egemen TAS,
Sr. Research Scientist,
COMODO

2008-05-21 (Comodo Firewall Pro 3.0.22.349 scored 95 % and took 1st place): Thank you very much for pointing out the "windows shutdown race condition" bug introduced with the tested version, which is the main reason for failing in SSS tests. We will be addressing this with the planned release on 05/20/2008. Keep up the good work.

Egemen TAS,
Sr. Research Scientist,
COMODO CP Inc

2008-03-18 (Comodo Firewall Pro 3.0.19.318 scored 98 % and took 1st place): We do appreciate the good work. We believe security researchers such as you guys, who try to poke holes in our products, are going to make them stronger and make us give our users even better products. Keep up the good work!

Egemen TAS,
Sr. Research Scientist,
COMODO Research Labs.

Emsi Software GmbH – the vendor of Mamutu

2008-11-29 (Mamutu 1.7.0.23 scored 2 %): In our opinion Mamutu is completely misplaced in this test. Mamutu is not a firewall, but a behavior blocker, designed to detect and block real malware samples only, not to pass a firewall leaktest. For us, the test results are useless because the product was tested for features that Behavior Blockers are not intended to provide in general. In oposite to Matousec, we think that Firewalls, Behavior Blockers and HIPS are not the same type of software and therefore they can not be tested and compared as they were the same.

Christian Mairoll
Managing Director
Emsi Software GmbH

Our response: We are testing a specific kind of security software which must meet some fixed criteria in order to be included to our project. The main criterion is to implement a process-based security. Proactive Security Challenge is designed to test Internet security suites, personal firewalls, HIPS products, behavior blockers and other behavior based systems. Mamutu met all the required criteria and hence there was no why not to include Mamutu to our project after we received several requests from our visitors. All the products included to our project implement similar features. These security features are tested in our project. We believe that using a set of open tests is the only objective way to compare all the products that implement the very same features. There are various tests used in Proactive Security Challenge, only a part of the used testing suite is based on leak-tests.

Filseclab Corporation – the vendor of Filseclab Personal Firewall

2008-06-18 (Filseclab Personal Firewall 3.0.0.8686 scored 1 %): Thank you for testing our firewall. Filseclab Personal Firewall 3.0.0.8686 is designed for inbound protection, it is worse for outbound protection, we recommend our users to use the Anti-virus or Anti-spyware software to protect the outbound. However, we will make a new version to improve those features in the end of this year.

Bright Chu
Filseclab

Jetico, Inc. – the vendor of Jetico Personal Firewall

2008-12-04 (Jetico Personal Firewall 2.0.2.7.2311 scored 72 %): Thank you for making SSTS available. We glad to see that the test suite is growing. Keep up your good work.

Nail Kaipov
Jetico, Inc.

2008-07-07 (Jetico Personal Firewall 2.0.2.4.2264 scored 78 %): We do appreciate your testing efforts. Your test suite reveals weaknesses in security software and helps vendors to make more reliable products. As soon as you added performance tests, please publish full technical specifications of your test system along with test results.

Nail Kaipov
Jetico, Inc.

Our response: Currently, we have only two performance tests in FWC, which are designed to compare the network performance of the products on 100 Mbit full duplex Ethernet. These tests are performed on the testing system without the tested product installed and also with the tested product installed. The measured results are compared then.

2008-03-27 (Jetico Personal Firewall 2.0.1.5.2216 scored 29 %): Thank you for testing our software. Results can be explained easily: JPF lacks self-protection. We shall add it in upcoming versions. We greatly appreciate your work on new test suite creation.

Nail Kaipov
Jetico, Inc.

Lavasoft – the vendor of Lavasoft Personal Firewall

2008-05-20 (Lavasoft Personal Firewall 3.0.2293.8822 scored 70 %): Thank you very much for your inclusion of the Lavasoft Personal Firewall 3.0 in the Matousec research. Upon review of the results, we were surprised to find that the Lavasoft Personal Firewall program received a 'good' rating and could not be a recommended firewall, whereas our firewall technology partner, Agnitum, received an 'excellent' score with recommendations for the same firewall technology. We hope that you will take this under consideration with your next round of research, and continue your good work.

Michael Helander
Vice President
Lavasoft

Our response: Outpost Firewall Pro was tested when there were 62 tests in the system, Lavasoft Personal Firewall was tested when there were 73 tests in the system. This is the first major difference. The second one is that Outpost Firewall Pro usually contains a newer version of their engine and so Outpost firewall may pass a few more tests than Lavasoft firewall. To mitigate the differences, we intend to retest Outpost Firewall Pro in the near future.

NETGATE Technologies s.r.o. – the vendor of FortKnox Personal Firewall

2008-04-08 (FortKnox Personal Firewall 2008 3.0.195.0 scored 16 %): We really appreciate testing of our product and we will try to incorporate protection for new tests in the next build releases. Thank you.

Martin Pekar
NETGATE Technologies s.r.o.

PC Tools Pty Ltd – the vendor of PC Tools Firewall Plus

2009-09-11 (PC Tools Firewall Plus 6.0.0.69 scored 99 % and took shared 1st place): Thank you for the detailed and thorough test. We are pleased with our results and are now working on resolving the issues found in the Matousec tests.

Ryan Pereira
PC Tools

2008-09-05 (PC Tools Firewall Plus 4.0.0.45 scored 85 %): We would like to thank Matousec for conducting the Proactive Security Challengetests. PC Tools has invested substantial resources to improve its support in this latest version, and will continue to do so in the future. We are pleased to hear that in this test PC Tools Firewall Plus came out as the fastest firewall. We recognize the importance of minimizing the firewalls influence on the user experience which is a testament to this latest test result. With regard to Keyloggers, PC Tools Firewall Plus does not focus on the detection of keyloggers as these types of threats fall into the Anti-Spyware category of protection. However, there are circumstances in which PC Tools Firewall Plus will detect a Keylogger, this being the point at which it attempts to access the Internet.

Hanoch Ben-David
PC Tools Firewall Plus Team Leader

PWI, Inc. – the vendor of Privatefirewall

2011-10-07 (Privatefirewall 7.0.25.4 scored 98 % and took 3rd place): We are naturally very pleased with the latest PSC test results, but will investigate and remedy the three outstanding identified flaws. Privacyware remains committed to continually strengthening Privatefirewall for our direct and OEM customers and the effort and dedication of the entire Matousec team has been instrumental in helping us do so. No other service provider possesses the level of expertise regarding leak, spy, malware, termination and so many other attack techniques - or the breadth of knowledge of the relevant software products to effectively assess these vulnerabilities. We extend our deepest gratitude and admiration for the role Matousec plays in helping vendors like Privacyware develop better products.

Greg Salvato
CEO - Privacyware: www.privacyware.com

2008-12-03 (Privatefirewall 6.0.20.9 scored 82 %): Thank you for the latest report and for continuing to push the envelope on desktop defense quality and performance. We will review the new tests and plan to modify Privatefirewall to correct the deficiencies we are able to validate and address.

Greg Salvato
CEO - Privacyware: www.privacyware.com

Soft4Ever – the vendor of Look 'n' Stop

2008-04-08 (Look 'n' Stop 2.06 scored 15 %): Look 'n' Stop Firewall is a pure firewall, which filters at TDI & NDIS levels. Look 'n' Stop is not an HIPS application and therefore is not designed to block most of the leaktests considered by the "Proactive Security Challenge" project from Matousec.

The Look 'n' Stop Team.

Sunbelt Software – the vendor of Sunbelt Personal Firewall

2008-03-24 (Sunbelt Personal Firewall 4.5.916 scored 18 %): Thanks for including us in your testing. Sunbelt Software is currently testing a new version of Sunbelt Personal Firewall (SPF), planned for release in Q2 2008, that addresses the majority of the leak tests that SPF 4.5.916 fails.

Phil Owens
Product Manager
Sunbelt Personal Firewall
Sunbelt Software