On Windows 7 (or Vista) I use
Proactive Security Challenge 64
- Results and comments
- Testing levels
- List of products
- Archive of results
- My leaks
- Frequently asked questions
Results and comments
The table(s) below sort(s) the tested products by their total score, which is displayed in the Product score column.
This table also shows the exact version of every tested product. The Level reached column presents the highest
level that the product reached in Proactive Security Challenge 64. If it passed all levels, this number is suffixed with a plus sign.
For products that scored at least 80 % in Proactive Security Challenge 64, the Recommendation column contains links to the online stores
or products' web pages of the vendors that we have affiliate agreements with. If you click on any of these links and then buy
the recommended product or other product offered on the target web page, we will profit from it. This is one of the ways how you can
support this project. The PDF document icon in the Report column allows you to download the testing report in PDF format for the tested product.
The last column of the table shows stars for products that received Proactive Security Challenge 64 Awards. Only products that were tested against
the latest set of tests and reached at least the Very good protection level are awarded.
You can see the award logo if you put your mouse cursor over the award star. The vendors of awarded products may ask
us for copies of award logo images to be used on their web sites or products' boxes.
Products tested against the suite with 110 tests
|Product||Product score||Level reached||Protection level||Recommendation||Report||Award|
|Comodo Internet Security Premium 7.0.317799.4142FREE||97 %||11+||Excellent||GET IT NOW!|
|Outpost Security Suite Pro 9.1.4643.690.1951||90 %||11||Excellent||GET IT NOW!|
|Kaspersky Internet Security 2015 220.127.116.113||89 %||11||Very good||GET IT NOW!|
|SpyShelter Firewall 9.2||89 %||11||Very good||GET IT NOW!|
|Privatefirewall 18.104.22.168FREE||88 %||11||Very good||N/A|
|Outpost Security Suite Free 22.214.171.12431.520.1248FREE||71 %||11||Good||Not recommended||–|
|VirusBuster Internet Security Suite 4.1||71 %||10||Good||Not recommended||–|
|ESET Smart Security 8.0.304.0||67 %||11||Good||Not recommended||–|
|Jetico Personal Firewall 126.96.36.199.2471||58 %||10||Poor||Not recommended||–|
|ZoneAlarm Extreme Security 2013 11.0.780.000||34 %||6||Very poor||Not recommended||–|
|ZoneAlarm Free Antivirus + Firewall 13.1.211.000FREE||34 %||6||Very poor||Not recommended||–|
|Total Defense Internet Security Suite 188.8.131.52||30 %||6||Very poor||Not recommended||–|
|Dr.Web Security Space 10.0.0.12011||24 %||4||None||Not recommended||–|
|Webroot SecureAnywhere IS Complete 184.108.40.206||23 %||4||None||Not recommended||–|
|Bitdefender Total Security 2014 220.127.116.111||19 %||4||None||Not recommended||–|
|BullGuard Internet Security 2014 14.0.279.6||16 %||3||None||Not recommended||–|
|eScan Internet Security Suite 14.0.1400.1381||14 %||3||None||Not recommended||–|
|Avira Internet Security 2014 18.104.22.1682||9 %||2||None||Not recommended||–|
|K7 TotalSecurity 2014 22.214.171.124||9 %||2||None||Not recommended||–|
|Norton Internet Security 2014 126.96.36.199||9 %||2||None||Not recommended||–|
|avast! Internet Security 2015.10.0.2208||8 %||2||None||Not recommended||–|
|TrustPort Total Protection 2014 188.8.131.5256||8 %||2||None||Not recommended||–|
|AVG Internet Security 2014.0.4577||7 %||2||None||Not recommended||–|
|F-Secure SAFE Internet Security 2.15.358||6 %||2||None||Not recommended||–|
|PC Tools Internet Security 2012 184.108.40.20698||6 %||2||None||Not recommended||–|
|ThreatFire 220.127.116.11FREE||5 %||2||None||Not recommended||–|
|Arcabit Internet Security 2014.0.0.282||4 %||1||None||Not recommended||–|
|FortKnox Personal Firewall 11.0.505.0||4 %||1||None||Not recommended||–|
|G Data TotalProtection 2015 18.104.22.168||4 %||1||None||Not recommended||–|
|Norman Security Suite PRO 11.00||4 %||1||None||Not recommended||–|
|Ad-Aware Total Security 11.1.5354.0||3 %||1||None||Not recommended||–|
|McAfee Total Protection 2015 13.6.1367||3 %||1||None||Not recommended||–|
|Rising Personal Firewall v16 24.00.15.64FREE||3 %||1||None||Not recommended||–|
|UnThreat Internet Security 2014 22.214.171.124892||3 %||1||None||Not recommended||–|
|AhnLab V3 Internet Security 126.96.36.199.1197||2 %||1||None||Not recommended||–|
|VIPRE Internet Security 2015 188.8.131.52||2 %||1||None||Not recommended||–|
|Zillya! Internet Security 1.1.4324.0||2 %||1||None||Not recommended||–|
|Panda Global Protection 2015 15.0.4||1 %||1||None||Not recommended||–|
The following links take you to pages with detailed products' results on each level. The level pages also contain important information about the given level and short information about its tests.
- Level 1 – Autorun12, Autorun3, Autorun9, Coat, FileDel2, Kill1, Kill2, Leaktest, Tooleaky, Yalta
- Level 2 – Autorun15, Autorun31, Autorun7, ECHOtest, FileWri1, Jumper, Kill4, Schedtest, Suspend1, Wallbreaker4
- Level 3 – Autorun10, Autorun4, AWFT4, ECHOtest2, FileDel1, HostsBlock, Keylog3, Kill6, RegDel1, Suspend2
- Level 4 – Autorun19, Autorun20, Autorun37, Crash1, FileMov1, Keylog4, Kill9, ProxyTest, SSS2, VBStest
- Level 5 – Autorun24, Autorun26, Autorun29, CopyCat, Crash2, DDEexec, FileWri2, Keylog7, RegSet1, Schedtest2
- Level 6 – Autorun25, Autorun28, Autorun36, Breakout1, Crash3, FileWri3, FireHole2, Inject2, Keylog5, SSS3
- Level 7 – Autorun17, Autorun23, Autorun41, Crash4, FileCtl1, FireHole, Keylog6, Kill8, RegDel2, Svckill
- Level 8 – Autorun38, Autorun5, Autorun8, Crash5, DDEtest, FileDel3, Flank, NewClass, Runner2, SSS4
- Level 9 – Autorun34, Autorun43, CPILSuite2, Crash6, DNStester, FileMov2, FileRep1, Keylog1, Kill12, Schedtest3
- Level 10 – Autorun39, Autorun44, Cliplog, FileOpn2, Inject1, Keylog2, Kill3e, OSfwbypass, RegAcc1, SockSnif
- Level 11 – Autorun40, Autorun42, BITStest, FileAcc1, FileRep2, Kill3f, Kill5, Screenlog, Schedtest4, WFPblock
Interpretation of results
The best product tested against the latest set of tests and the Proactive Security Challenge 64 leader is currently Comodo Internet Security Premium 7.0.317799.4142 with excellent 97% score. Outpost Security Suite Pro 9.1.4643.690.1951 is on the second place with another excellent result – 90 %. These products are closely followed by Kaspersky Internet Security 2015 184.108.40.2063 and SpyShelter Firewall 9.2 with 89% score, and Privatefirewall 220.127.116.11 with 88 %. These are the only products that have been awarded with Proactive Security Challenge 64 awards.
Proactive Security Challenge 64 makes a big difference between good products and the rest of the world. Most of the products are filtered in very low levels which means that they probably miss some critical features. There are plenty of products whose vendors claim that they can protect you against all kinds known and unknown threats. But in fact their products implement just a basic set of features that can prevent the most simple attacks only. If a malicious application uses more advanced approach to achieve its goal it is not blocked. This is very common in case of self-defense features, anti-leak protection, or system infection prevention.
Differences between quality of protection on the 64-bit platform (tested in Proactive Security Challenge 64) and the 32-bit platform (formerly tested in Proactive Security Challenge) are also very interesting. The nature of the 64-bit platform makes it uneasy for the products to protect against some of the advanced attacking techniques, but most of the techniques can be prevented by the very same mechanisms as on the 32-bit platform. This is why one would expect to see the 64-bit results of the products to be just slightly worse than the results on the 32-bit platform. This is not the case, however. The protection of many products is on a totally different level on the 64-bit platform, unfortunately much worse level.
However, it is crucial to know what does it mean if a product succeeds in our tests and what does it mean if it fails. Before you start interpreting the results, you should be familiar with the information on the index page, especially with the project's methodology and rules. You should also know what kind of products we test before you start to interpret the results. We have received a lot of reactions from people who are not familiar with that information and simply do not understand the results and misinterpret them. All the tested products have one common feature – the application-based security model. In combination with their packet filtering capabilities, the tested products attempt to block attacks from other machines on the network as well as attacks performed by malicious codes that might run inside the protected machine. This is definitely not an unusual situation. People who use email clients, instant messengers, or web browsers face attacks that exploit the vulnerabilities in this kind of software very often. It happens that a malicious code gets inside the machine. And then it may try to install itself silently to the system, to steal users' data or sniff their passwords, or to join the target machine to a botnet. This is what the products we test want to prevent. This is why they are used. The problem is that although the goal is common, not all the products implement a sufficient protection.
We require the products tested in Proactive Security Challenge 64 to prevent data and identity theft. They should also implement a packet filter functionality to prevent direct online attacks – i.e. not to let the malware get in. The products should control the software installed on the computer to prevent the malware to integrate itself into the operating system. Then the malware should not be able to get the user's private data, thus anti-sniffing, anti-keylogging and personal data protection features should be implemented too. And even if the malware succeeded to collect the information it should not be allowed to send it outside the protected system and this means an implementation of the outbound network traffic control. To achieve all these is much harder task than it seems. The protection system also has to prevent attacking trusted processes and other components in the system. Otherwise, the malware would be able to use trusted parts of the system to integrate into the operating system, to collect or steal sensitive data and/or to send the data outside the system without being noticed. So the next feature that is required here is a control of untrusted processes' activities and that is the hardest task for the tested products. It also includes an implementation of self-protection mechanisms because the malware should not be able to terminate the protection, which implies some other features to be implemented and so on.
So, what does it mean if the product fails even the most basic tests of our challenge? It means that it is unable to do what its vendor claims it can. Such a product can hardly protect you against the mentioned threats. On the other hand, if the product succeeds in all our tests, it does not mean that it is perfect. Our tests are focused on the security and stability, but there are many other aspects important for the users like performance, hardware requirements, easy to use, availability of support, price, vendor's reaction time to new threats etc. It should also be noted that although our testing suite is quite large, it is not complete and there may exist many other ways to bypass the tested products. We are working constantly on extending the suite to be able to provide more accurate information about the security of the tested products. If the tested product fails only a few tests in our challenge, it still might be a great product. This is why we can recommend, from the security point of view, the products that reached at least 80% score in the challenge. You should try them yourself and choose the one best for you, the one that you would be happy with, the one you would be able to configure and use everyday.
Note that we test the products in their highest usable security settings configuration. This means that the results reflect the best the products can do. However, out of box settings of many products may provide much less security. It is not rare that the default configuration of a product is set to a medium level of protection in order to achieve a better user experience.
We have received following responses to Proactive Security Challenge 64:
Agnitum Ltd. – the vendor of Outpost Security Suite Pro
2014-04-03 (Outpost Security Suite Pro 9.1.4643.690.1951 scored 90 % and took 2nd place): Many thanks to Matousec Lab for another testing! As revealed in the latest results, Outpost keeps up the top scores, although there are a few issues we'll need to address in the next editions. Security Challenge accepted!
2013-03-25 (Outpost Security Suite Pro 8.0.4164.639.1856 scored 90 % and took 2nd place): Agnitum has always taken 64-bit security seriously and we continue to provide full protection for 64-bit Windows and make perpetual improvements as security threats and hacker techniques evolve. Along with a high overall score, the latest Proactive Security Challenge 64 revealed a few potential issues we had already been working on for some time. There's always room for perfection, and we plan to fill the existing the gaps shortly in one of our future releases.
2012-03-27 (Outpost Security Suite Pro 18.104.22.16839.602.1809 scored 86 % and took 2nd place): Securing 32- and 64-bit systems is in many ways different due to Microsoft Kernel Patch Protection. According to the tests some products protect only 32-bit OS's. The technology behind our products lets us protect all PCs - both 32-bit and 64-bit.
Agnitum's Head of QA Department
Bitdefender – the vendor of Bitdefender Total Security
2014-06-23 (Bitdefender Total Security 2014 22.214.171.1241 scored 19 %):
As one of the world's most advanced security solutions, Bitdefender is committed to providing the best protection to our users. In the Matousec test, Bitdefender Total Security scored 19%, as it did not go through the entire testing procedure.
This unforeseeable side effect is triggered by the fact that the test only takes into account whether the product blocks individual actions that are dissociated by other features characteristic to malware, such as the ability of an application to add itself to startup via Registry. These behaviors are also specific to legitimate applications that one user may voluntarily install on their computer, and blocking them in real-world scenarios would be considered as false positives. In real life scenarios, there is no reason to prevent an application from creating its own startup Registry key unless that application is deemed malicious by a complementary Bitdefender technology.
On our customer's devices, Bitdefender uses layered technologies to ensure accurate detection by correlating inputs from antimalware engines, host-intrusion prevention systems, cloud intelligence and behavior-based inspection. For the past three years, this layered approach to security has brought us multiple No. 1 awards for detection rate in tests carried by international organizations such as AV-Test and AV-Comparatives.
Your input is very valuable to us as we are continuously working on improving our award-winning technologies to boost detection and usability in real-world usage scenarios.
Alina Dana UNGUREANU
Our response: We find it somehow strange that autorun tests are being questioned by Bitdefender. Bitdefender implements a behavior based protection that actually does prevent access (asks for application's permissions) to the most common autorun location used by legitimate applications (i.e. the technique used by Autorun3). In fact, Bitdefender also monitors many other autorun locations and prevents their infection. On the other hand, it fails to block techniques of hijacking trusted applications (e.g. technique of AWFT4), keylogging attempts (Keylog3, Keylog4), and other highly suspicious techniques.
Comodo Security Solutions, Inc. – the vendor of Comodo Internet Security Premium
2013-10-29 (Comodo Internet Security Premium 6.3.297838.2953 scored 97 % and took 1st place): We are very glad to receive these stellar results on your tests. Your testing results will reassure our users that over the years, while we have significantly improved the performance and usability, we have not sacrificed from the security and the protection we offer. Keep up the good work!
VP of Engineering,
COMODO Security Solutions, Inc.
2012-01-16 (Comodo Internet Security Premium 5.9.219863.2196 scored 94 % and took 1st place): Thank you for including our product in your latest tests. Your open-source, open methodology tests are always very well thought out and unique. The latest results indicate the fact that there is a reason for us to be able to offer a 500 USD virus free guarantee in all operating systems. Keep up the good work!
Director, Desktop Security Products,
PWI, Inc. – the vendor of Privatefirewall
2012-08-07 (Privatefirewall 126.96.36.199 scored 88 % and took 2nd place): Thank you Matousec for this latest testing effort for Privatefirewall and your continued commitment to security software vendors and their customers. Overall, we are pleased with Privatefirewall's improved score of 88 % and #2 ranking, but now have our attention focused on addressing the COM based and other vulnerabilities that remain.
CEO - Privacyware
2012-01-16 (Privatefirewall 188.8.131.52 scored 56 % and took 3rd place): While a #3 ranking is not very satisfying when combined with the Protection level earned by Privatefirewall in this initial round of the new x64 PSC, we are aware of the work required on our end to dramatically improve our performance and the effort to do so is already underway. We thank the Matousec organization for their continued commitment to helping vendors improve their products and ensuring that the consumers and businesses that rely on them are able to realize the greatest levels of system protection.
CEO - Privacyware
SpyShelter – the vendor of SpyShelter Firewall
2013-04-19 (SpyShelter Firewall 2.2 scored 85 % and took 5th place): SpyShelter Firewall is basically very young product, which already managed to reach World's top 5 in Firewall security. These great news makes us proud of SpyShelter Firewall and We will continue to develop it, so it can become the best.
2013-04-10 (SpyShelter Firewall 1.5 scored 79 % and took 5th place): Results of the test are based on SpyShelter Firewall 1.5, and the results are just great – but the 2.2 version We've just released, is even greater, as We never stop improving protection granted by using our products.