On Windows 7 (or Vista) I use
Proactive Security Challenge
- Results and comments
- Testing levels
- List of products
- Archive of results
- My leaks
- Frequently asked questions
Proactive Security Challenge has been replaced with Proactive Security Challenge 64!
- 2011-12-01: Proactive Security Challenge has been replaced with Proactive Security Challenge 64.
This project examines security software for Windows OS that implement application-based security model – i.e. most of the products called Internet security suites, personal firewalls, HIPS, behavior blockers and similar products on the market. A product must meet some fixed criteria in order to be included in this project. The list of products suitable for this project that we are aware of is available on the product list page.
Methodology and rules
Installation and configuration
The tested products are installed on Windows XP Service Pack 3 with Internet Explorer 8 set as the default browser. The products are configured to their highest usable security settings and tested with this configuration only. We define the highest security settings as settings that the user is able to set without advanced knowledge of the operating system. This means that the user, with the skills and knowledge we assume, is able to go through all forms of the graphic user interface of the product and enable or disable or choose among several therein given options, but is not able to think out names of devices, directories, files, registry entries etc. to add to some table of protected objects manually, not even if such a configuration is suggested on the product's support forum or website.
There are several testing levels in Proactive Security Challenge. Each level contains a selected set of tests and it also contains a score limit that is necessary to pass this level. All products are tested with the level 1 set of tests. Those products that reach the score limit of level 1 and thus pass this level will be tested in level 2 and so on until they reach the highest level or until they fail a limit of some level.
Testing suite and scoring
Most of the tests are part of Security Software Testing Suite, which is a set of small tests that are all available with source codes. Using this open suite makes the testing transparent as much as possible. For each test the tested product can get a score between 0 % and 100 %. The tests can be simply passed or failed only and so the product can get 0 % or 100 % score only. It should be noted that the testing programs are not perfect and in many cases they use methods, that are not reliable on 100 %, to recognize whether the tested system passes or failed the test. This means that it might happen that the testing program reports that the tested system passed the test even if it failed, this is called a false positive result. The official result of the test is always set by an experienced human tester in order to filter false positive results. The opposite situations of false negative results should be rare but are also eliminated by the tester.
To be able to make right decisions in disputable situations, we define the test types. Every test has a defined type. Tests of the same type usually attempt to achieve the same goal. Here is a list of the defined types and their goals:
- Leak-test: Leak-tests attempt to send data to the Internet server, this is called leaking. Most of the leak-tests from Security Software Testing Suite are configured to use a script on our website that logs leaks to our database by default. For such tests, you can use My leaks page to see whether the test was able to transmit the data. For leak-tests that do not use this script, we use a packet sniffer in unclear situations. In order to pass many leak-tests, the tested product has to implement some host protection features.
- Spying test: These tests attempt to spy on users' input or data. Keyloggers and packet sniffers are typical examples of spying tests. Every piece of the data they obtain is searched for a pattern, which is defined in the configuration file. These tests usually succeed if the given pattern has been found.
- Autorun test: These tests attempt to install to the system in order to ensure they will be started again. The most common goal in case of these tests is to survive the reboot. Such a system infection is typical for various kinds of malware. The tested product fails the autorun test if the test is able to ensure that it will be started in the future again.
- System integrity test: One of the roles of security suites is to protect the system integrity from malicious modifications. System integrity tests attempts to gain enough privilege in the system so that they are able to subvert the system.
- Self-defense test: This category of tests include various attacks against the security product itself. Termination tests are the first subtype of tests that belongs in this category. These tests attempt to terminate or somehow damage processes, or their parts, of the tested product. The termination test usually succeeds if at least one of the target processes, or at least one of their parts, was terminated or damaged. Besides processes and threads, the security software usually relies on various files and registry entries. Tests that attempt to remove, destroy or corrupt these critical objects for the security product also belong to this category.
- Other: Tests that do not fit any of the previously defined types are of this type. These tests, for example, may check stability, reliability or other quality of the tested product.
All tests are equal to the intent that their scores are not weighted by their level or something else. The total score of the tested product is counted as follows. For all tests in all levels that the product did not reach, the product's score is 0 %. For all other tests the score is determined by the testing. The total score of the product is a sum of the scores of all tests divided by the number of all tests and rounded to a whole number. It may happen that a new test is added to Proactive Security Challenge when some products already has their results. In such case, the result for already tested product is set to N/A for this new test, which means that it is not counted for this product and does not affect its score or level passing. Neither the number of the tests, nor the number of levels is final. We intend to create new tests in the future. We are also open to your ideas of new testing techniques or even complete tests.
All tests on the levels a tested product reaches are run at least once. If a product passes a test, this test is repeated at least once in order to mitigate false passing. For more information about the testing process see the testing guidelines.
Product's selection and vendors rights
Products for testing are selected from those that were requested for tests by their vendors or often suggested for tests by our visitors, more times than other products. If there are no such products, we will select products for tests ourselves, preferentially taking products that have not been tested at all yet and have a real chance to succeed in our tests. Every vendor has a right for its product to be tested in Proactive Security Challenge for free once in a six months period and this right is valid only for stable and publicly available versions of the products. If a vendor offers more than one product it still has a right of only one free test per six months. Moreover, next free testing of a product will be performed no sooner than three months after the last free testing. This rule should prevent vendors from using Proactive Security Challenge testing as a free beta testing service. The only exception from this rule is for vendors that offer a product with an anti-virus or an anti-malware engine and mark any of the tests of Security Software Testing Suite as a virus, an infected code, an unwanted or a malicious application, or offend any part of the suite directly using pattern recognitions or any other form of blacklisting. This approach deceives the users of such anti-virus or anti-malware engines and make the testing more difficult for us. The vendors who offend the testing suite have no right for free testing at all but can still request a paid testing.
Every vendor has a right to request a paid Proactive Security Challenge testing, in which case its product will be tested in all levels regardless the results on each of the levels. After the vendor receives the results of the paid testing, it can either keep them private or request their publishing on our website, but such a request will be satisfied only if the previously published results for the tested product, if any, are at least one month old and if the tested version is stable and publicly available. There are no limits of the frequency of the paid tests.
How you can help us
Do you enjoy Proactive Security Challenge? Do you want to help us but you do not consider yourself to be a security expert? Still you can help us! If you intend to buy a security software, you may be interested to buy one of the products we recommend in Proactive Security Challenge. Have a look at Products' ratings on the results page. The Recommendation column in the table contains links to the online stores or products' webpages of the vendors that we have affiliate agreements with. If you click on any of these links and then buy the target product or other product offered on the target webpage, we will profit from it. So, if you are going to buy a security software and you like our projects, you can help us! Even if you have the licence already, we can profit if you prolong the licence after you visit the vendor's website through our recommendation links. Thank you!
The rules for the recommended products are simple. The first condition is that the product's Protection level is at least Very good, which means that its final score is at least 80 %. The second condition is that we have an affiliate agreement with its vendor. It is important to note that if the recommended product is retested and does not reach the 80 % limit, it will not be recommended anymore, at least not until the next retesting.
We provide various services to vendors of security software. Besides the above mentioned paid Proactive Security Challenge testing, we provide commercial testing based on the original methodology for advanced analyses. We also offer consulting services and research related to Windows internals, implementation of security software, design of security software, reverse engineering and malware analyses. Get more information about the services we offer.
- 2011-12-01: Proactive Security Challenge has been replaced with Proactive Security Challenge 64.
- 2011-10-07: A response from PWI, Inc., the vendor of Privatefirewall, has been added.
- 2011-10-06: New results have been published for:
- CA Internet Security Suite Plus 2011 220.127.116.119
- Norton Internet Security 2012 18.104.22.168
- Privatefirewall 22.214.171.124
It is natural that new versions of the products score higher than the older ones. This time, however, only Privatefirewall and CA Internet Security Suite Plus improved their score.
Norton Internet Security felt down from 40% score to 20 %. Although we see that some new techniques are handled by Norton Internet Security, other techniques that were handled in the 2011 version are not handled anymore.
CA Internet Security Suite Plus went the opposite way. From 12% score it jumped up to 30 %, which is still not a good result, but it definitely is a better one and a possible promise for its future versions.
The best product from today's testing is thus the new Privatefirewall. Its score has been improved by 7 % to the Excellent, and close to perfect, 98 %. Good job.
- 2011-08-27: A single product update:
- BitDefender Internet Security 2011 126.96.36.1997
BitDefender Internet Security has been improved again. Its new 97% score suggests that there are not many tests that can bypass its protection. Although its overall protection has been improved, some of the problems with its application behavior control module, which we identified last time, were not mitigated. Many alerts displayed by BitDefender Internet Security still lack important information that would help users to make correct decisions. More information can be found in the final report.
- 2011-08-19: New results have been published for:
- PC Tools Internet Security 2011 188.8.131.525
- Rising Internet Security 2011 23.00.41.42
- ZoneAlarm Extreme Security 2012 10.0.250.000
This time PC Tools Internet Security did not confirm its previous excellent results. Because of a single imperfection in its self-protection rule set, its final score jumped down to 90 %. Although this will probably be a trivial problem to fix for its developers, the rules are uncompromising.
ZoneAlarm Extreme Security is the only product tested this time that performed better than last time. Its today's score is 72 %, quite a lot of space for improvements but still better than most.
Rising Internet Security fell down to 34 %, the Very poor protection level, far from the leaders.
- 2011-07-22: A single product update:
- Outpost Security Suite Pro 184.108.40.20691.596.1681
Outpost Security Suite Pro belongs among the best products we test in Proactive Security Challenge. The new version confirmed the Excellent performance of this security product again. It failed only 4 tests out of 148 tests, which makes 97% score.
- 2011-06-20: New results have been published for:
- Kaspersky Internet Security 2012 220.127.116.114
93% score for Kaspersky Internet Security 2012 means 1 % improvement over the previously tested 2011 version. Kaspersky Internet Security confirmed its position among the best products tested in Proactive Security Challenge. Excellent result.
- 2011-05-29: New results have been published for:
- Avira Premium Security Suite 10.0.0.608
- Malware Defender 2.7.3.0002
Malware Defender finally reached the Excellent protection level. 91 % for today, very well done, but still some space is left for improvements, especially a protection of system COM interfaces is missing.
On the other side of the result table is Avira Premium Security Suite. With its 3% score there are just few products that finished even worse.
- 2011-05-06: New results have been published for:
- F-Secure Internet Security 2011 10.51.106
- Panda Internet Security 2011 16.00.00
The 2011 versions of the products tested this time performed just as their previously tested 2010 versions. For F-Secure Internet Security this means 9% score, for Panda Internet Security it is only 2 %.
- 2011-04-14: New results have been published for:
- avast! Internet Security 6.0.1000
- Jetico Personal Firewall 18.104.22.168.2451
Both products tested this time scored much better than the last time they were tested. We can clearly see improvements in their host protection security features. Yet still the final score of avast! Internet Security is insufficient. It scored 15 % ending in level 3.
Jetico Personal Firewall jumped up from its previous 28% score to its today's 88 %. Well done! The only thing that is harming otherwise good impression is Jetico Personal Firewall's behavior in case of malware attack. It often happens with Jetico that the system must be rebooted to be usable again after the user blocks a potentially dangerous action. The real life usability of Jetico Personal Firewall is thus questionable.
- 2011-03-21: New results have been published for:
- McAfee Internet Security 2011 11.5.141
- Privatefirewall 22.214.171.124
- ZoneAlarm Extreme Security 9.3.037.000
Privatefirewall is better and better with every new version. This time it finally reached the Excellent level of protection by scoring 91 %. Very well done and we hope to see even better results next time!
ZoneAlarm Extreme Security also jumped up a little bit. We classify its today's 66% score as a Good result.
Unfortunately, McAfee Internet Security did not surprised us. Its score of 2 % places it at the end of the scoreboard.
- 2011-02-17: New results have been published for:
- Comodo Internet Security 5.3.176757.1236
- Look 'n' Stop 2.07
- Sunbelt Personal Firewall 4.6.1861.0
Comodo Internet Security confirmed its leadership in Proactive Security Challenge by scoring the perfect 100% score again. Our classification of its host protection features is nothing but Excellent.
Look 'n' Stop and Sunbelt Personal Firewall are on the opposite side of our result table. Neither of these two products passed the level one and both finished with poor 3 % score.
- 2011-01-25: A single product update:
- PC Tools Internet Security 2011 126.96.36.1993
PC Tools Internet Security replaces PC Tools Firewall Plus in our challenge. The last version of PC Tools Firewall Plus we have tested in Proactive Security Challenge scored 51 %. PC Tools Internet Security 2011 188.8.131.523 scored 99 % – great improvements in the product's security led to the Excellent level of protection. PC Tools Internet Security is now on the second place in Proactive Security Challenge. Well done!
- 2011-01-10: New results have been published for:
- Norton Internet Security 2011 184.108.40.206
- G Data InternetSecurity 2011 220.127.116.11
Norton Internet Security 2011 scored 40 % in today's testing which is similar to its previous result.
G Data InternetSecurity 2011 scored only 2 % and thus ended deeply in the red zone with the protection level None.
- 2010-12-21: New results have been published for:
- Outpost Security Suite Free 18.104.22.16818.520.1245.401
- TrustPort Internet Security 2011 22.214.171.12484
Outpost Security Suite Free returned the free version of Outpost products among the leaders of our challenge. Its today's result is Excellent 97 %, very well done and congratulations to its vendor!
On the other side of the table is TrustPort Internet Security. Its score is as low as 2 %.
- 2010-11-14: New results have been published for:
- BitDefender Internet Security 2011 126.96.36.1990
- FortKnox Personal Firewall 188.8.131.52
BitDefender Internet Security jumped from its previous 24% score to its current 84 %. The newly tested version 2011 comes with great improvements. Some users, however, might rate the protection of BitDefender Internet Security as completely useless. In case of many actions that BitDefender Internet Security controls, pop-ups it displays contain absolutely no information about the potentially dangerous action being blocked. Lack of information makes the decision whether or not to block the action extremely hard. Moreover, these actions can not be blocked without terminating the offending process. We hope to see this improved in next versions of BitDefender Internet Security.
FortKnox Personal Firewall received 7 % and joined many other products in the None Protection level group.
- 2010-11-05: New results have been published for:
- AVG Internet Security 2011 10.0.1153
- ESET Smart Security 184.108.40.206
Both products tested this time performed poorly. AVG Internet Security's final score is 3 %, ESET Smart Security received 6 %.
- 2010-11-01: Research related to real malware's use of techniques that are tested in Proactive Security Challenge has been performed. Check out the results.
- 2010-09-26: New results have been published for:
- Dr.Web Security Space Pro 6.0.2.07290
- Rising Internet Security 2010 22.33.00.01
- ZoneAlarm Free Firewall 9.2.076.000
Pleasant surprise among today's results is the score of Rising Internet Security 2010. Its previously tested version scored as low as 5 %. The latest version comes with various improvements and reached 55 %.
Dr.Web Security Space Pro is new to our challenge but it did not score very well. 14 % is far behind many of its competitors.
ZoneAlarm Free Firewall failed in the tests like its previously tested version. Its current 7% score is not good.
- 2010-09-07: New results have been published for:
- Kaspersky Internet Security 2011 220.127.116.110
- Privatefirewall 18.104.22.168
Kaspersky Internet Security 2011 comes with improvements again. Today it reaches the Excellent level of protection in our challenge with its result of 92 %. Good job!
The new version of Privatefirewall is also improved. Its today's result is 86 %, which we classify as a Very good result.
- 2010-09-07: Online Armor products have been temporarily disqualified from Proactive Security Challenge.
- 2010-07-11: New results have been published for:
- Avira Premium Security Suite 10.0.0.542
- F-Secure Internet Security 2010 10.00.246
- CA Internet Security Suite Plus 2010 22.214.171.1245
No product tested this time performed well. CA Internet Security Suite Plus's 12 % score was the best today, F-Secure Internet Security finished with 9 % and Avira Premium Security Suite ended with 3 % only. Hope to see better results next time.
- 2010-06-29: A single product update:
- Outpost Security Suite Pro 126.96.36.19976.514.1234.401
Outpost Security Suite Pro 7.0.1 reached 97% score, the Excellent level of protection. This is a great improvement compared to the performance of its previously tested version 6.7.3, which scored only 72 %. Congratulations!
- 2010-05-23: New results have been published for:
- Online Armor Premium 188.8.131.52
- Online Solutions Security Suite 1.5.14905.0
Online Armor Premium confirmed its quality and reached the Excellent 97% score.
A very pleasant surprise is the result of Online Solutions Security Suite on its first appearance in Proactive Security Challenge. It took the second place with incredible 99 %, an Excellent level of protection. Very well done!
- 2010-05-04: New results have been published for:
- Outpost Security Suite Pro 2009 184.108.40.20663.452.0726
- PC Tools Firewall Plus 220.127.116.11
Outpost Security Suite Pro performed similarly to its recently tested Free cousin. Having great problems with the new set of tests it finished with 72% score, a Good level of protection.
PC Tools Firewall Plus had even bigger problems with the new tests and its previously perfect performance against 84 tests is no longer true against 148 tests. PC Tools Firewall Plus this time finished with a Poor result of 51 %.
- 2010-05-01: A response from Comodo Security Solutions, Inc., the vendor of Comodo Internet Security, has been added.
- 2010-05-01: A single product update:
- Comodo Internet Security 4.0.141842.828
Comodo Internet Security 4.0.141842.828 reached the perfect score even against 148 tests. Comodo Internet Security is now the new leader of Proactive Security Challenge. Congratulations!
- 2010-04-12: New results have been published for:
- Outpost Firewall Free 2009 18.104.22.16825.381.0687
- Panda Internet Security 2010 15.01.00
Outpost Firewall Free 2009 had great problems with the new tests in the suite. Its new 71% score is far behind its previous 93% performance. This time it is just a Good level of protection for Outpost Firewall Free. Panda Internet Security 2010 failed in the tests completely, no more comments needed.
- 2010-03-30: New results have been published for:
- Online Armor Free 22.214.171.124
- Trend Micro Internet Security Pro 2010 17.50.1647.0000
Trend Micro Internet Security Pro 2010 finished with 9 %. Online Armor Free performed much better. We can see major improvements in Online Armor Free compared to its previously tested version. Its new score of 96 % is good enough for today to take the lead in Proactive Security Challenge. Congratulations!
- 2010-03-22: A single product update:
- Kaspersky Internet Security 2010 126.96.36.1996
Kaspersky Internet Security 2010 188.8.131.526 scored 86 % against 148 tests, a Very good result and currently the second place in our challenge, not so far behind Malware Defender.
- 2010-03-15: New results have been published for:
- BitDefender Internet Security 2010 184.108.40.2067
- Jetico Personal Firewall 220.127.116.11.2412
Jetico Personal Firewall failed most of the new tests and this why we can see the radical change in its score. It finished with the score of 28 %.
BitDefender Internet Security 2010 have a couple of improvements compared to its previously tested version and so its final score is better today – 24 %.
- 2010-03-04: New results have been published for:
- Malware Defender 2.6.0
- McAfee Internet Security 2010 11.0.378
Malware Defender did a very good job. 90% score is currently the best score we have against the full set of 148 tests. It is the first product to receive Proactive Security Challenge award, Malware Defender passed 133 out of 148 tests. Congratulations!
Not much to say about McAfee Internet Security, its 3% score says that good host protection is not something you would buy this product for.
- 2010-03-04: We have introduced awards for products that were tested against the full set of 148 tests and that reached at least Very good protection level.
- 2010-02-21: New results have been published for:
- avast! Internet Security 5.0.418.0
- Norton Internet Security 2010 18.104.22.168
- Privatefirewall 22.214.171.124
- ZoneAlarm Extreme Security 9.1.008.000
The best product tested this time, firstly against the new set of 148 tests, was Privatefirewall. It scored 74 %, which we classify as a Good level of protection.
ZoneAlarm Extreme Security and Norton Internet Security were not that good and finished with the score of 59 % and 40 % respectively.
The worst results of only 3 % was scored by avast! Internet Security.
- 2010-01-04: Several major changes have been implemented. For the full list see Another Proactive Security Challenge milestone – it now contains 148 tests and the changelog of Security Software Testing Suite. Proactive Security Challenge now contains 148 tests and newly covers file protection, registry protection and control of system autorun entries.
- 2009-12-11: New results have been published for:
- Online Armor Premium 126.96.36.199
- PC Tools Firewall Plus 188.8.131.52
PC Tools Firewall Plus finally reached the perfect 100% score. Congratullations!
Online Armor Premium also confirmed its quality and is still on 99 %, an Excellent result.
- 2009-10-29: New results have been published for:
- AVG Internet Security 9.0.697
- G DATA InternetSecurity 2010 184.108.40.206
- Norton Internet Security 2010 220.127.116.11
Norton Internet Security 2010 achieved similar results as its previously tested version 2009. Its new score is 67 %, a Good level of protection.
AVG Internet Security and G DATA InternetSecurity failed in the tests.
- 2009-10-10: A new response from Comodo Security Solutions, Inc., the vendor of Comodo Internet Security, has been added.
- 2009-10-10: A single product update:
- Comodo Internet Security 3.12.111745.560
Per its vendor's request, we have tested the latest version of Comodo Internet Security. As the first product, Comodo Internet Security reached the perfect score against all 84 tests in the testing suite. Congratulations!
The great results of the challenge leading products in the last tests force us to implement new tests to our testing suite. We are going to cover even more areas of Windows security with many new tests in the following months in order to raise the bar just a bit again.
- 2009-09-13: The testing guidelines have been published in the project's FAQ. This makes Proactive Security Challenge even more transparent and should allow more people and product vendors to reproduce the published results easily.
- 2009-09-11: A new response from PC Tools Pty Ltd, the vendor of PC Tools Firewall Plus, has been added.
- 2009-09-09: New results have been published for:
- Comodo Internet Security 3.11.108364.552
- F-Secure Internet Security 2010 10.00.246
- PC Tools Firewall Plus 18.104.22.168
- TrustPort PC Security 22.214.171.1245
PC Tools Firewall Plus performed almost perfectly against our tests and with the score of 99 % took the shared first place in our challenge, together with previously tested Online Armor Personal Firewall. Congratulations!
Comodo Internet Security also reached the Excellent level of protection scoring 97 %.
The other two products ended in the
redzone with protection level None. F-Secure Internet Security scored 8 % and TrustPort PC Security scored only 4 %.
- 2009-07-24: New results have been published for:
- FortiClient End Point Security 126.96.36.199
- Kingsoft Internet Security 9 Plus 2009.07.17.10
- Norman Security Suite 7.10
- Trend Micro Internet Security Pro 17.1.1250
- Windows Live OneCare 2.5.2900.24
Trend Micro Internet Security Pro scored 29 %, which is slightly better result than before (27 % against 73 tests). Still, Trend Micro does not belong among the better products on the market available today.
All the other products tested today also finished in the red zone of the result table but their scores are much worse. The reasons are simple. These products implement only basic network access control mechanisms. They do not implement parent process control or application hijacking control. Most of them have no self-defense features and thus can be disabled and bypassed by malicious software. Such products can not succeed in our tests, the bar is too high for them.
- 2009-06-29: New results have been published for:
- Kaspersky Internet Security 2010 188.8.131.529
- Panda Internet Security 2010 15.00.00
- Rising Internet Security 2009 21.41.21
Kaspersky Internet Security did a good job and for the first time in our challenge it reached the Excellent level of protection. It's previous result of 83 % was improved to a great 96 %.
On the other side of the table finished both other products tested today. Panda Internet Security finished with 4% score and just a little better score of 5 % was achieved by Rising Internet Security.
- 2009-06-11: New results have been published for:
- Avira Premium Security Suite 184.108.40.2067
- Lavasoft Personal Firewall 3.0.2293.8822
- Malware Defender 2.2.2
- Online Armor Personal Firewall 220.127.116.11 Free
- Outpost Security Suite Pro 2009 18.104.22.1685.381.0687
During the testing of Outpost Security Suite, we have revealed a mistake that affected testing of Outpost Firewall Free 2009 6.5.2724.381.0687.328 performed on 2nd May 2009. Outpost Firewall Free 2009 6.5.2724.381.0687.328 was retested against Kernel2 test and its result was changed and its final score modified to 93 %. Fortunately, this have not affected position of any product in the result table. We apologize for this mistake.
The best score today was achieved by Outpost Security Suite – Excellent 92 %. It may look strange that the free version of Outpost Firewall finished with the better score than the paid Outpost Security Suite. The reason is that almost all features that our project tests are implemented in the same way in both products. However, Outpost Security Suite includes antivirus and antispyware engines implemented by additional driver that are not included in Outpost Firewall Free. And because of these additional drivers Outpost Security Suite did not pass level 9 tests, which caused the final lower score.
Similar Excellent result of 92 % was also achived by Online Armor Personal Firewall Free and just a few points behind finished Malware Defender with its Very good score of 89 %.
Lavasoft Personal Firewall was tested in exactly same version as before, because no new version was published by its vendor since the last time. Against 84 tests, it achived 67% score, a Good level of protection.
Today's worst result was achieved by Avira Premium Security Suite that did not pass the level 2 and finished with 10% score.
- 2009-05-20: Internet Explorer 8 is now the default browser for Proactive Security Challenge testing. Test Kill6 has been updated to allow 50 % scoring.
- 2009-05-02: New results have been published for:
- CA Internet Security Suite Plus 2009 22.214.171.1241
- eConceal Pro for Windows 2.0.019.1
- Online Armor Personal Firewall 126.96.36.199
- Outpost Firewall Free 2009 6.5.2724.381.0687.328
- Privatefirewall 188.8.131.52
- ThreatFire Free 184.108.40.206
Online Armor Personal Firewall 220.127.116.11 scored almost perfectly, its 99% score means a clear leadership in our challenge. Congratulations!
Another Excellent score was achieved by Outpost Firewall Free 2009 6.5.2724.381.0687.328, a new free alternative on the market. It scored 94 %, which might be confusing when compared to 93% score of its commercial version, but please note that the tested commercial version is older. We intend to retest Outpost Firewall Pro in the near future.
The third remarkable result today was made by Privatefirewall 18.104.22.168. This product is improving constantly and it is now close to Excellent results with its score of 88 %.
Other three products tested today finished on the opposite side of the result table. CA Internet Security Suite Plus 2009 22.214.171.1241 and ThreatFire Free 126.96.36.199 finished with 5% score, eConceal Pro for Windows 2.0.019.1 even worse – 4% score. It should be mentioned that CA and ThreatFire are products that heavily misuse unsafe user mode hooks to implement security features. Again and again we repeat that these techniques can not be implemented safely, which our testing suite demonstrates very well.
- 2009-04-23: Firewall Challenge has been renamed to Proactive Security Challenge.
- 2009-04-08: A new response from Comodo Security Solutions, Inc., the vendor of Comodo Internet Security, has been added.
- 2009-04-07: New results have been published for:
- BitDefender Internet Security 2009 188.8.131.52
- Comodo Internet Security 3.8.65951.477
- ESET Smart Security 4.0.417.0
- ZoneAlarm Free Firewall 8.0.298.000
Among the products tested today Comodo improved the score from 93 % to 96 % which makes it the new leader of our challenge. Congratulations!
Other products did not perform so well and ended with protection level None. BitDefender implements various security features using user mode hooks which cannot be considered safe. We used internally modified tests that bypassed this protection. Despite this, BitDefender managed to pass more tests than the last time and improved its score from 7 % to 12 %. ESET Smart Security scored 4 %, just as the last time. ZoneAlarm Free, new to the challenge, scored 11 %.
- 2009-02-17: A new response from Agnitum Ltd., the vendor of Outpost Security Suite Pro, has been added.
- 2009-02-16: New results have been published for:
- Malware Defender 2.0.5
- Netchina S3 2008 184.108.40.206
- Norton Internet Security 2009 220.127.116.11
- Outpost Security Suite Pro 2009 6.5.2514.381.0685
Vendors of Malware Defender and Outpost Security Suite Pro requested testing of their products. Outpost Security Suite Pro replaces Outpost Firewall Pro in our challenge. Both these products performed well against the tests. Malware Defender scored 87 %, a Very good result, even better if we consider that it is a new product to our challenge. Outpost Security Suite Pro jumped up to the shared first place with Online Armor with the score of 93 %. Congratulations!
Netchina S3 keeped its protection level and scored 85 %. Norton Internet Security did a Good job and scored 66 %.
- 2009-02-07: Proactive Security Challenge archive has been opened.
- 2009-01-07: New results have been published for:
- Jetico Personal Firewall 18.104.22.168.2327
- Kaspersky Internet Security 2009 22.214.171.1246
- Online Armor Personal Firewall 126.96.36.199 Free
- PC Tools Firewall Plus 188.8.131.52
- ZoneAlarm Pro 8.0.059.000
The vendors of Jetico Personal Firewall and PC Tools Firewall Plus requested testing of their products after implementing several improvements into them. We can see great improvements in case of Jetico Personal Firewall. Its new version reached the score of 89 %, a Very good protection level, which is 17 % better result than its previously tested version.
The previously tested version of PC Tools Firewall Plus scored 85 % but it was not tested against the new tests. The new version scored 86 % against the new suite with 84 tests, which makes this result quite strong if we consider that most of the competitive products tested against the new tests lost significantly.
Online Armor Personal Firewall Free scored 86 %, which is an expected result since the Free version, compared to the paid version, does not contain anti-keylogger features. The previously tested Free version scored 89 % against the suite with 73 tests.
Kaspersky Internet Security lost 4 % because of the new tests and it is now on 83 %, still a Very good result.
ZoneAlarm Pro, which was previously on 86 % against 73 tests, is now on 72 % against the suite with 84 tests, this is a Good level of protection.
- 2008-12-13: The result page now supports two different views of results.
- 2008-12-04: A response from Jetico, Inc., the vendor of Jetico Personal Firewall, has been added.
- 2008-12-03: A response from PWI, Inc., the vendor of Privatefirewall, has been added.
- 2008-12-01: A response from Comodo Security Solutions, Inc., the vendor of Comodo Internet Security, has been added.
- 2008-11-30: We have received a suggestion from Comodo Security Solutions, Inc., the vendor of Comodo Internet Security, on how to improve the configuration of their product in order to pass more tests. The suggested change included switching Comodo Internet Security configuration to "COMODO - Proactive Security", which can be done via the product's tray icon. Since our rules say that the tests are performed aginst the highest usable security configuration, we retested Comodo Internet Security 3.5.55810.432 and corrected its results. Its new score is 90 %, which is much better score than previously published incorrect value of 84 %. We apologize to Comodo and all our visitors for this mistake.
- 2008-11-29: Responses from Tall Emu Pty Ltd, the vendor of Online Armor Personal Firewall, and Emsi Software GmbH, the vendor of Mamutu, have been added.
- 2008-11-28: New results have been published for:
- Comodo Internet Security 3.5.55810.432
- Jetico Personal Firewall 184.108.40.206.2311
- Mamutu 220.127.116.11
- Online Armor Personal Firewall 18.104.22.168
- Outpost Firewall Pro 2009 6.5.2358.316.0607
- Privatefirewall 22.214.171.124
- Sunbelt Personal Firewall 4.6.1861.0
- Webroot Desktop Firewall 126.96.36.199
The new tests and improvements published three weeks ago examined the tested products very well. The total score of all products went down, more or less. This is why we have removed the results of ProSecurity 1.43, which is a dead product and can not be tested against new tests, hence its excellent score achieved a very long time ago is not comparable with the score of the other top products that faced much more sophisticated tests. When a stable public version of ProSecurity's successor – Real-time Defender – is available, we will test it as soon as possible.
The power of the new versions of the tests removed the Excellent rankings from all the leading products, except for Online Armor Personal Firewall 188.8.131.52, which finished with the score of 93 % and became a new leader of Proactive Security Challenge. Online Armor lost 5 % compared to its previously tested version. The second best result today is Very good score of 89 % that belongs to Outpost Firewall Pro 2009 6.5.2358.316.0607. Outpost lost 10 % and leaved the first position but its result is still remarkable. Even more, 11 % of the total score, was lost by Comodo. Comodo Internet Security 3.5.55810.432 scored 84 %, reached a Very good level of protection and the third best score today.
Privatefirewall 184.108.40.206 finished close to Comodo with the final score of 82 %, which is 8 % lower score than of its previously tested version. The fifth best product tested today was Jetico Personal Firewall 220.127.116.11.2311. Jetico lost 6 % and finished with the score of 72 %, which is a Good protection level. A Poor level of protection and the final score of 54 % was achieved by Webroot Desktop Firewall 18.104.22.168.
Sunbelt Personal Firewall 4.6.1861.0 and Mamutu 22.214.171.124 failed the tests completely. Sunbelt finished with 5 % and Mamutu reached only 2% score.
Today's testing focused mostly on the leading products. The next update will focus on those products that are at the top of the result table but have not been tested against the new versions of tests yet.
- 2008-11-07: Several improvements have been implemented. For the full list see Major changes in Proactive Security Challenge and the changelog of Security Software Testing Suite. Proactive Security Challenge now contains 84 tests.
- 2008-10-14: New results have been published for:
- ESET Smart Security 3.0.672.0
- Trend Micro Internet Security Pro 17.0.1224
Neither of these two products can be recommended if we consider Proactive Security Challenge results. ESET Smart Security failed totally and with 4 %, which is one of the worst results we have seen so far. Trend Micro Internet Security Pro scored much better, but it finished with the same score as its previously tested version, which means 27 %.
- 2008-09-28: New results have been published for:
- BitDefender Internet Security 2009 126.96.36.199
- McAfee Internet Security 2009 10.0.209
- ZoneAlarm Pro 8.0.020.000
Among the 2009 solutions we have tested today, only ZoneAlarm Pro is worth mentioning. Its previously tested version 7 scored only 63 % (against 62 tests). The new version has been improved so that it passed all the levels with the final score of 86 %, which is a Very good result.
BitDefender Internet Security 2009 and McAfee Internet Security 2009 failed in the tests completely. BitDefender 2008 scored 4 % (against 70 tests) and its 2009 version scored 7 % (against 73 tests) remaining on None Protection level. McAfee version 2009 finished with the final score of 12 % (against 73 tests), its previously tested version scored 6 % (against 70 tests).
- 2008-09-15: A single product update:
- Norton Internet Security 2009 188.8.131.52
The new version of Norton Internet Security is out. Its improvements are noticeable. While the previous version scored Very Poor 32 % (against 62 tests), the new version comes with Good protection level and final score of 71 % (against 73 tests).
- 2008-09-11: A single product update:
- Kaspersky Internet Security 2009 184.108.40.2064
We have received many requests to test the new version of KIS. Finally, we have the results. Its previous version tested against 62 tests scored 85 %. The new version scored 87 % against 73 tests, which is slightly better result, and it is still a Very good protection level.
- 2008-09-05: A new response from PC Tools Pty Ltd, the vendor of PC Tools Firewall Plus, has been added.
- 2008-09-05: New results have been published for:
- Netchina S3 2008 220.127.116.11
- PC Tools Firewall Plus 18.104.22.168
Both products were tested on requests of their vendors. PC Tools Firewall Plus passed all testing levels and finished with 85% score. Netchina S3 is new to FWC but its results are better than of many well established products. It scored a Very good result of 86 %. Netchina is a new product that needs to mature before it is suitable for common users, but its protection is solid and future versions might be excellent products. Both these products are free.
- 2008-08-29: New results have been published for:
- Comodo Firewall Pro 22.214.171.124
- Rising Personal Firewall 2008 20.59.10
We have decided to test these two products after we received many requests to test them from our visitors. Users of Windows 2000 asked us to test the old version Comodo Firewall Pro because the new version 3 is not compatible with Windows 2000. Comodo Firewall Pro version 2 scored 55 %, which is marked as Poor level of protection, but it is not a bad result if we consider that this version is quite old. Yet still we suggest to switch to another product if you are a user of Windows 2000 or to upgrade to a newer version of Windows and use Comodo Firewall Pro version 3. Much worse is the situation with Rising Personal Firewall 2008, which scored only 11 %. Its protection is not comparable with the today's top personal firewalls.
- 2008-08-14: A single product update:
- Privatefirewall 126.96.36.199
Privatefirewall has been improved since the last time. Its previously tested version scored only 65 %, while the new version scored Excellent 90 %.
- 2008-08-01: New results have been published for:
- Dynamic Security Agent 188.8.131.52
- PC Tools Firewall Plus 184.108.40.206
- Sunbelt Personal Firewall 4.6.1839.0
Dynamic Security Agent (DSA) and PC Tools Firewall Plus are both free products and both reached the level 7. DSA scored 62 %, PC Tools Firewall Plus 74 %. DSA was tested for the first time, PC Tools Firewall Plus was already tested in the past – its previously tested version scored only 6 %, so its improvements are remarkable.
Sunbelt Personal Firewall scored worse than before because of the new tests. It did not even reach the level 2 and finished with 7% score.
- 2008-07-07: A new response from Jetico, Inc., the vendor of Jetico Personal Firewall, has been added.
- 2008-07-04: A new response from Agnitum Ltd., the vendor of Outpost Firewall Pro 2009 6.5.2355.316.0597, has been added.
- 2008-07-04: New results have been published for:
- Ashampoo FireWall FREE 1.20
- G DATA InternetSecurity 2008
- Jetico Personal Firewall 220.127.116.11.2264
- Outpost Firewall Pro 2009 6.5.2355.316.0597
Outpost Firewall Pro 2009 greatly succeeded in the tests and took a lead in the challenge. Its result is 99 %, almost a perfect score! Congratulations!
A new version of Jetico Personal Firewal can also be said to be a winner today. Compared to its previous version, which scored with 29 %, the new version has been improved dramatically, it reached level 7 and gained 78 %.
G DATA InternetSecurity 2008 did not reached level 4 and ended with 19% score. It is a very heavy product that needs a lot of system resources and its results of performance tests were the worst we have seen so far.
The worst product among today's four is Ashampoo FireWall FREE. Although it reached a perfect score in performance tests, it failed almost every other test and did not even reach level 2. Its final score is 5 %.
- 2008-06-28: New results have been published for:
- Filseclab Personal Firewall 18.104.22.16882
- Online Armor Personal Firewall 22.214.171.124
- ZoneAlarm Pro 7.0.473.000
The paid version of Online Armor Personal Firewall has been tested upon the vendor's free request. Compared to the free version, the paid version implements protection against keyloggers, hence its results are better. The paid version of Online Armor Personal Firewall takes a lead in our challenge again.
Filseclab Personal Firewall has been retested after we have received the information that we tested its very old version instead of its latest update. However, the latest update did not help this product in our challenge.
ZoneAlarm Pro 7.0.473.000 has been tested to see its performance against the new tests that were implemented to Proactive Security Challenge recently. ZoneAlarm does not protect against many of the new tests and so its final score is slightly worse than before.
We are going to publish more results soon.
- 2008-06-18: A response from Filseclab Corporation, the vendor of Filseclab Personal Firewall, has been added.
- 2008-06-07: A response from BitDefender, the vendor of BitDefender Internet Security, has been added.
- 2008-05-21: A response from Comodo Group, the vendor of Comodo Firewall Pro, has been added.
- 2008-05-20: A response from Lavasoft, the vendor of Lavasoft Personal Firewall, has been added.
- 2008-05-17: A new response from Tall Emu, the vendor of Online Armor Personal Firewall Free, has been added.
- 2008-05-17: Results for these products and versions were published:
- Comodo Firewall Pro 126.96.36.1999
- F-Secure Internet Security 2008 8.00.101
- Lavasoft Personal Firewall 3.0.2293.8822
- Online Armor Personal Firewall 188.8.131.52 Free
- Panda Internet Security 2008 12.01.00
- Trend Micro Internet Security 2008 184.108.40.2066
- Webroot Desktop Firewall 220.127.116.11
After the implementation of several new tests, we have decided to retest Comodo Firewall Pro and Online Armor Personal Firewall Free to see if they can still score 100 % in our challenge. Both these products lost the perfect score, but Comodo Firewall Pro remained on the first position in our challenge. The odd thing is that the new version of Comodo Firewall Pro did not pass two tests, namely SSS and SSS4, that its previously tested version passed. This firewall has no problems with our new keylogger tests but failed SockSnif test. It also lost a few points in performance tests but its results are still excellent and almost perfect, 95 % in total. Online Armor Personal Firewall Free lost more points, especially in keylogger tests. The classification of its protection is now Very good with 89 %, close to the Excellent protection, which starts on 90 %.
Lavasoft Personal Firewall uses the engine of Agnitum's Outpost and its score for today is 70 %. Webroot Desktop Firewall, a free product that uses the old version of the engine of Privatefirewall, scored 60 %.
F-Secure Internet Security 2008, Panda Internet Security 2008 and Trend Micro Internet Security 2008 are security suits with very high hardware requirements but their protection is noticeably worse than of other products tested today.
- 2008-05-06: Three new tests have been added to Proactive Security Challenge. Two performance tests, PerfTCP and PerfUDP, have been added to Level 1. A spying test called SockSnif has been added to Level 8.
- 2008-05-02: We have joined Tall Emu's affiliate program and so you can now buy commercial versions of Online Armor Personal Firewall through our pages and thus support this project.
- 2008-04-25: Results for these products and versions were published:
- Avira Premium Security Suite 8.1.00.206
- BitDefender Internet Security 2008 11.0.16
- BullGuard Internet Security 18.104.22.168
- Filseclab Personal Firewall 22.214.171.12486
- McAfee Internet Security Suite 2008 9.1.108
- Steganos Internet Security 2008 7.5.509
All these products scored very badly. Filseclab Personal Firewall is a very old product and so its protection against today's malware techniques is naturally insufficient. Other products tested today are anti-virus solutions with very weak personal firewall components.
- 2008-04-24: Eight new tests have been added to Proactive Security Challenge, these are Keylog1, Keylog2, Keylog3, Keylog4, Keylog5, Keylog6, Keylog7 and ShadowHook. We have 70 tests in the system now, new products and new versions of already tested products are always tested against all the tests in the system on levels that they reach.
- 2008-04-24: A response from AVG Technologies, the vendor of AVG Internet Security, has been added.
- 2008-04-20: We have joined affiliate programs of Agnitum Ltd. and Kaspersky Lab, now you can support Proactive Security Challenge by buying one of their products through our affiliate links, read more.
- 2008-04-08: Responses from Soft4Ever, the vendor of Look 'n' Stop, and NETGATE Technologies s.r.o., the vendor of FortKnox Personal Firewall, have been added.
- 2008-04-06: Results for these products and versions were published:
- AVG Internet Security 8.0.93
- FortKnox Personal Firewall 2008 126.96.36.199
- iolo Personal Firewall 188.8.131.52
- Look 'n' Stop 2.06
- Norton Internet Security 2008 184.108.40.206
- PC Tools Firewall Plus 220.127.116.11
- Rising Personal Firewall 2007 18.104.22.168
- System Safety Monitor 22.214.171.1242
The best among the products tested today was clearly System Safety Monitor 126.96.36.1992. It reached level 7 and scored 77 %. Norton Internet Security 2008 188.8.131.52 reached the level 4 with 32 % and was the second best product we tested today. All other firewalls failed in the tests completely, especially iolo Personal Firewall 184.108.40.206.
- 2008-04-01: We have added two new rules to Proactive Security Challenge to improve its quality and we have also created a FAQ page. The first change in rules is that we will publish results of stable and publicly available versions only – i.e. no public results of beta or internal versions any more. The second change is that we will not publish the results of the paid Proactive Security Challenge testing if we published some results for the given product in the last month. The vendors are thus no longer able to quickly fix their products in response to our testing, ask us then for retesting and almost silently replace the old results in one or two days. This rule should prevent the unwanted behavior of vendors that tend to focus on fighting the tests because of marketing and forget about other problems in their products.
- 2008-03-30: Comodo Firewall Pro 220.127.116.119 has been retested, its vendor ordered a paid FWC testing with a belief that it smoothly passes all the tests. We are happy to announce that since today we have two firewalls that managed to score 100 % in Proactive Security Challenge. More tests will be necessary to select the best among the today's elite personal firewalls.
- 2008-03-27: A response from Jetico, Inc., the vendor of Jetico Personal Firewall, has been added.
- 2008-03-25: We have received an email from ailef and MaratR with information about a security weakness
in Online Armor Personal Firewall 18.104.22.168 Free that was tested in our challenge recently. We have successfully
verified the information that the tested version of Online Armor automatically allows various privileged actions
if it receives no response from the user in a few minutes after the alert is shown. We would like to thank ailef and MaratR
for their findings, we would like to apologize to our visitors and other vendors for possibly wrong results in case of Online Armor.
We have contacted the vendor of Online Armor and received the information that the latest version of this product, Online Armor Personal Firewall 22.214.171.124 Free, does not suffer from the problem any more. To solve the problem with possibly wrong results, the vendor ordered a paid testing of its product. We have tested Online Armor Personal Firewall 126.96.36.199 Free and found that the security hole was fixed and also that it passes all current Proactive Security Challenge tests. Online Armor is thus the first product with the perfect result in Proactive Security Challenge tests. We are going to implement new tests to the testing system in next months and try to violate its perfect score.
- 2008-03-25: Results for these products and versions were published:
- Jetico Personal Firewall 188.8.131.52.2216
- Kaspersky Internet Security 184.108.40.2065
- Windows Live OneCare 2.0.2500.22
Very interesting were results of Jetico Personal Firewall. Its results showed the difference between the old leak-testing and Proactive Security Challenge testing. Jetico Personal Firewall passed only two levels. These results might be very disappointing for Jetico fans.
Windows Live OneCare failed the tests completely. This product was probably intended to replace the internal Windows firewall but its protection, as a personal firewall, is nearly the same.
Kaspersky Internet Security confirmed its qualities. Its result is Very good and we can expect that future versions will score even better.
- 2008-03-24: A response from Sunbelt Software, the vendor of Sunbelt Personal Firewall, has been added.
- 2008-03-18: Results for these products and versions were published:
- Comodo Firewall Pro 220.127.116.118
- ESET Smart Security 3.0.621.0
- Online Armor Personal Firewall 18.104.22.168 Free
- Outpost Firewall Pro 2008 6.0.2302.264.0490
- Privatefirewall 22.214.171.124
- ProSecurity 1.43
- Sunbelt Personal Firewall 4.5.916
- ZoneAlarm Pro 7.0.462.000
We chose some of the top products intentionally to see how strong the Proactive Security Challenge tests are. None of the tested products scored 100 % but Comodo Firewall Pro, Online Armor Personal Firewall, Outpost Firewall Pro and ProSecurity were very close. Even ZoneAlarm Pro and Privatefirewall scored pretty well too but their score is significantly worse than the score of the four, previously mentioned, excellent products. Unpleasant surprise was ESET Smart Security that did not even reach the second level, the result of Sunbelt Personal Firewall was better only a little bit.
- 2008-03-18: Proactive Security Challenge has been started. It has been called Firewall Challenge.