matousec.com (site map)

Poll

On Windows 7 (or Vista) I use

  unlimited administrator's account (58.04%)

  limited administrator's account (16.36%)

  common user's account (12.76%)

  nothing (I do not use Win 7/Vista) (13.72%)

more

results

Proactive Security Challenge 64

Results and comments

Contents:


Back to contents

Products' ratings

The table(s) below sort(s) the tested products by their total score, which is displayed in the Product score column. This table also shows the exact version of every tested product. The Level reached column presents the highest level that the product reached in Proactive Security Challenge 64. If it passed all levels, this number is suffixed with a plus sign. For products that scored at least 80 % in Proactive Security Challenge 64, the Recommendation column contains links to the online stores or products' web pages of the vendors that we have affiliate agreements with. If you click on any of these links and then buy the recommended product or other product offered on the target web page, we will profit from it. This is one of the ways how you can support this project. The PDF document icon in the Report column allows you to download the testing report in PDF format for the tested product. The last column of the table shows stars for products that received Proactive Security Challenge 64 Awards. Only products that were tested against the latest set of tests and reached at least the Very good protection level are awarded. You can see the award logo if you put your mouse cursor over the award star. The vendors of awarded products may ask us for copies of award logo images to be used on their web sites or products' boxes.




Products tested against the suite with 110 tests


Product Product score Level reached Protection level Recommendation Report Award
ico Comodo Internet Security Premium 7.0.317799.4142FREE 97 % 11+ Excellent GET IT NOW! pdf
ico Outpost Security Suite Pro 9.1.4643.690.1951 90 % 11 Excellent GET IT NOW! pdf
ico Kaspersky Internet Security 2015 15.0.0.463 89 % 11 Very good GET IT NOW! pdf
ico SpyShelter Firewall 9.2 89 % 11 Very good GET IT NOW! pdf
ico Privatefirewall 7.0.30.3FREE 88 % 11 Very good N/A pdf
ico Outpost Security Suite Free 7.1.1.3431.520.1248FREE 71 % 11 Good Not recommended pdf
ico VirusBuster Internet Security Suite 4.1 71 % 10 Good Not recommended pdf
ico ESET Smart Security 8.0.304.0 67 % 11 Good Not recommended pdf
ico Jetico Personal Firewall 2.1.0.13.2471 58 % 10 Poor Not recommended pdf
ico ZoneAlarm Extreme Security 2013 11.0.780.000 34 % 6 Very poor Not recommended pdf
ico ZoneAlarm Free Antivirus + Firewall 13.1.211.000FREE 34 % 6 Very poor Not recommended pdf
ico Total Defense Internet Security Suite 9.0.0.134 30 % 6 Very poor Not recommended pdf
ico Dr.Web Security Space 10.0.0.12011 24 % 4 None Not recommended pdf
ico Webroot SecureAnywhere IS Complete 8.0.4.104 23 % 4 None Not recommended pdf
ico Bitdefender Total Security 2014 17.28.0.1191 19 % 4 None Not recommended pdf
ico BullGuard Internet Security 2014 14.0.279.6 16 % 3 None Not recommended pdf
ico eScan Internet Security Suite 14.0.1400.1381 14 % 3 None Not recommended pdf
ico Avira Internet Security 2014 14.0.6.552 9 % 2 None Not recommended pdf
ico K7 TotalSecurity 2014 14.1.0.217 9 % 2 None Not recommended pdf
ico Norton Internet Security 2014 21.3.0.12 9 % 2 None Not recommended pdf
ico avast! Internet Security 2015.10.0.2208 8 % 2 None Not recommended pdf
ico TrustPort Total Protection 2014 14.0.3.5256 8 % 2 None Not recommended pdf
ico AVG Internet Security 2014.0.4577 7 % 2 None Not recommended pdf
ico F-Secure SAFE Internet Security 2.15.358 6 % 2 None Not recommended pdf
ico PC Tools Internet Security 2012 9.1.0.2898 6 % 2 None Not recommended pdf
ico ThreatFire 4.7.0.53FREE 5 % 2 None Not recommended pdf
ico Arcabit Internet Security 2014.0.0.282 4 % 1 None Not recommended pdf
ico FortKnox Personal Firewall 11.0.505.0 4 % 1 None Not recommended pdf
ico G Data TotalProtection 2015 25.0.1.2 4 % 1 None Not recommended pdf
ico Norman Security Suite PRO 11.00 4 % 1 None Not recommended pdf
ico Ad-Aware Total Security 11.1.5354.0 3 % 1 None Not recommended pdf
ico McAfee Total Protection 2015 13.6.1367 3 % 1 None Not recommended pdf
ico Rising Personal Firewall v16 24.00.15.64FREE 3 % 1 None Not recommended pdf
ico UnThreat Internet Security 2014 6.2.37.26892 3 % 1 None Not recommended pdf
ico AhnLab V3 Internet Security 8.0.6.6.1197 2 % 1 None Not recommended pdf
ico VIPRE Internet Security 2015 8.0.3.3 2 % 1 None Not recommended pdf
ico Zillya! Internet Security 1.1.4324.0 2 % 1 None Not recommended pdf
ico Panda Global Protection 2015 15.0.4 1 % 1 None Not recommended pdf
 


Detailed results

The following links take you to pages with detailed products' results on each level. The level pages also contain important information about the given level and short information about its tests.


Interpretation of results

The best product tested against the latest set of tests and the Proactive Security Challenge 64 leader is currently Comodo Internet Security Premium 7.0.317799.4142 with excellent 97% score. Outpost Security Suite Pro 9.1.4643.690.1951 is on the second place with another excellent result – 90 %. These products are closely followed by Kaspersky Internet Security 2015 15.0.0.463 and SpyShelter Firewall 9.2 with 89% score, and Privatefirewall 7.0.30.3 with 88 %. These are the only products that have been awarded with Proactive Security Challenge 64 awards.

Proactive Security Challenge 64 makes a big difference between good products and the rest of the world. Most of the products are filtered in very low levels which means that they probably miss some critical features. There are plenty of products whose vendors claim that they can protect you against all kinds known and unknown threats. But in fact their products implement just a basic set of features that can prevent the most simple attacks only. If a malicious application uses more advanced approach to achieve its goal it is not blocked. This is very common in case of self-defense features, anti-leak protection, or system infection prevention.

Differences between quality of protection on the 64-bit platform (tested in Proactive Security Challenge 64) and the 32-bit platform (formerly tested in Proactive Security Challenge) are also very interesting. The nature of the 64-bit platform makes it uneasy for the products to protect against some of the advanced attacking techniques, but most of the techniques can be prevented by the very same mechanisms as on the 32-bit platform. This is why one would expect to see the 64-bit results of the products to be just slightly worse than the results on the 32-bit platform. This is not the case, however. The protection of many products is on a totally different level on the 64-bit platform, unfortunately much worse level.

However, it is crucial to know what does it mean if a product succeeds in our tests and what does it mean if it fails. Before you start interpreting the results, you should be familiar with the information on the index page, especially with the project's methodology and rules. You should also know what kind of products we test before you start to interpret the results. We have received a lot of reactions from people who are not familiar with that information and simply do not understand the results and misinterpret them. All the tested products have one common feature – the application-based security model. In combination with their packet filtering capabilities, the tested products attempt to block attacks from other machines on the network as well as attacks performed by malicious codes that might run inside the protected machine. This is definitely not an unusual situation. People who use email clients, instant messengers, or web browsers face attacks that exploit the vulnerabilities in this kind of software very often. It happens that a malicious code gets inside the machine. And then it may try to install itself silently to the system, to steal users' data or sniff their passwords, or to join the target machine to a botnet. This is what the products we test want to prevent. This is why they are used. The problem is that although the goal is common, not all the products implement a sufficient protection.

We require the products tested in Proactive Security Challenge 64 to prevent data and identity theft. They should also implement a packet filter functionality to prevent direct online attacks – i.e. not to let the malware get in. The products should control the software installed on the computer to prevent the malware to integrate itself into the operating system. Then the malware should not be able to get the user's private data, thus anti-sniffing, anti-keylogging and personal data protection features should be implemented too. And even if the malware succeeded to collect the information it should not be allowed to send it outside the protected system and this means an implementation of the outbound network traffic control. To achieve all these is much harder task than it seems. The protection system also has to prevent attacking trusted processes and other components in the system. Otherwise, the malware would be able to use trusted parts of the system to integrate into the operating system, to collect or steal sensitive data and/or to send the data outside the system without being noticed. So the next feature that is required here is a control of untrusted processes' activities and that is the hardest task for the tested products. It also includes an implementation of self-protection mechanisms because the malware should not be able to terminate the protection, which implies some other features to be implemented and so on.

So, what does it mean if the product fails even the most basic tests of our challenge? It means that it is unable to do what its vendor claims it can. Such a product can hardly protect you against the mentioned threats. On the other hand, if the product succeeds in all our tests, it does not mean that it is perfect. Our tests are focused on the security and stability, but there are many other aspects important for the users like performance, hardware requirements, easy to use, availability of support, price, vendor's reaction time to new threats etc. It should also be noted that although our testing suite is quite large, it is not complete and there may exist many other ways to bypass the tested products. We are working constantly on extending the suite to be able to provide more accurate information about the security of the tested products. If the tested product fails only a few tests in our challenge, it still might be a great product. This is why we can recommend, from the security point of view, the products that reached at least 80% score in the challenge. You should try them yourself and choose the one best for you, the one that you would be happy with, the one you would be able to configure and use everyday.

Note that we test the products in their highest usable security settings configuration. This means that the results reflect the best the products can do. However, out of box settings of many products may provide much less security. It is not rare that the default configuration of a product is set to a medium level of protection in order to achieve a better user experience.


Vendors' responses

We have received following responses to Proactive Security Challenge 64:

Agnitum Ltd. – the vendor of Outpost Security Suite Pro

2014-04-03 (Outpost Security Suite Pro 9.1.4643.690.1951 scored 90 % and took 2nd place): Many thanks to Matousec Lab for another testing! As revealed in the latest results, Outpost keeps up the top scores, although there are a few issues we'll need to address in the next editions. Security Challenge accepted!

Mikhail Zakhryapin,
Managing Director

2013-03-25 (Outpost Security Suite Pro 8.0.4164.639.1856 scored 90 % and took 2nd place): Agnitum has always taken 64-bit security seriously and we continue to provide full protection for 64-bit Windows and make perpetual improvements as security threats and hacker techniques evolve. Along with a high overall score, the latest Proactive Security Challenge 64 revealed a few potential issues we had already been working on for some time. There's always room for perfection, and we plan to fill the existing the gaps shortly in one of our future releases.

Mikhail Zakhryapin,
Managing Director

2012-03-27 (Outpost Security Suite Pro 7.5.2.3939.602.1809 scored 86 % and took 2nd place): Securing 32- and 64-bit systems is in many ways different due to Microsoft Kernel Patch Protection. According to the tests some products protect only 32-bit OS's. The technology behind our products lets us protect all PCs - both 32-bit and 64-bit.

Konstantin Kutikov
Agnitum's Head of QA Department

Bitdefender – the vendor of Bitdefender Total Security

2014-06-23 (Bitdefender Total Security 2014 17.28.0.1191 scored 19 %): As one of the world's most advanced security solutions, Bitdefender is committed to providing the best protection to our users. In the Matousec test, Bitdefender Total Security scored 19%, as it did not go through the entire testing procedure.

This unforeseeable side effect is triggered by the fact that the test only takes into account whether the product blocks individual actions that are dissociated by other features characteristic to malware, such as the ability of an application to add itself to startup via Registry. These behaviors are also specific to legitimate applications that one user may voluntarily install on their computer, and blocking them in real-world scenarios would be considered as false positives. In real life scenarios, there is no reason to prevent an application from creating its own startup Registry key unless that application is deemed malicious by a complementary Bitdefender technology.

On our customer's devices, Bitdefender uses layered technologies to ensure accurate detection by correlating inputs from antimalware engines, host-intrusion prevention systems, cloud intelligence and behavior-based inspection. For the past three years, this layered approach to security has brought us multiple No. 1 awards for detection rate in tests carried by international organizations such as AV-Test and AV-Comparatives.

Your input is very valuable to us as we are continuously working on improving our award-winning technologies to boost detection and usability in real-world usage scenarios.

Alina Dana UNGUREANU
Product Manager

Our response: We find it somehow strange that autorun tests are being questioned by Bitdefender. Bitdefender implements a behavior based protection that actually does prevent access (asks for application's permissions) to the most common autorun location used by legitimate applications (i.e. the technique used by Autorun3). In fact, Bitdefender also monitors many other autorun locations and prevents their infection. On the other hand, it fails to block techniques of hijacking trusted applications (e.g. technique of AWFT4), keylogging attempts (Keylog3, Keylog4), and other highly suspicious techniques.

Comodo Security Solutions, Inc. – the vendor of Comodo Internet Security Premium

2013-10-29 (Comodo Internet Security Premium 6.3.297838.2953 scored 97 % and took 1st place): We are very glad to receive these stellar results on your tests. Your testing results will reassure our users that over the years, while we have significantly improved the performance and usability, we have not sacrificed from the security and the protection we offer. Keep up the good work!

Egemen TAS,
VP of Engineering,
COMODO Security Solutions, Inc.

2012-01-16 (Comodo Internet Security Premium 5.9.219863.2196 scored 94 % and took 1st place): Thank you for including our product in your latest tests. Your open-source, open methodology tests are always very well thought out and unique. The latest results indicate the fact that there is a reason for us to be able to offer a 500 USD virus free guarantee in all operating systems. Keep up the good work!

Egemen TAS,
Director, Desktop Security Products,
COMODO

PWI, Inc. – the vendor of Privatefirewall

2012-08-07 (Privatefirewall 7.0.28.1 scored 88 % and took 2nd place): Thank you Matousec for this latest testing effort for Privatefirewall and your continued commitment to security software vendors and their customers. Overall, we are pleased with Privatefirewall's improved score of 88 % and #2 ranking, but now have our attention focused on addressing the COM based and other vulnerabilities that remain.

Greg Salvato
CEO - Privacyware

2012-01-16 (Privatefirewall 7.0.25.5 scored 56 % and took 3rd place): While a #3 ranking is not very satisfying when combined with the Protection level earned by Privatefirewall in this initial round of the new x64 PSC, we are aware of the work required on our end to dramatically improve our performance and the effort to do so is already underway. We thank the Matousec organization for their continued commitment to helping vendors improve their products and ensuring that the consumers and businesses that rely on them are able to realize the greatest levels of system protection.

Greg Salvato
CEO - Privacyware

SpyShelter – the vendor of SpyShelter Firewall

2013-04-19 (SpyShelter Firewall 2.2 scored 85 % and took 5th place): SpyShelter Firewall is basically very young product, which already managed to reach World's top 5 in Firewall security. These great news makes us proud of SpyShelter Firewall and We will continue to develop it, so it can become the best.

Bogdan
SpyShelter

2013-04-10 (SpyShelter Firewall 1.5 scored 79 % and took 5th place): Results of the test are based on SpyShelter Firewall 1.5, and the results are just great – but the 2.2 version We've just released, is even greater, as We never stop improving protection granted by using our products.

Janusz
SpyShelter