Home > Projects > Proactive Security Challenge 64

Proactive Security Challenge 64

• Introduction
• Results and comments
• Testing levels
• List of products
• Archive of results
• My leaks
• Frequently asked questions

Results and comments

Contents:

• Products' ratings
• Detailed results
• Interpretation of results
• Vendors' responses

Products' ratings

The table(s) below sort(s) the tested products by their total score, which is displayed in the Product score column. There are two possible views of the total scores. The default view separates the results by the number of tests that were in the system when the products were tested, i.e. the number of tests with a valid test result value (other value than N/A). There is one table for each number of tests. The second view mixes all the results into one table. In the second view, the Product score column consists of two numbers separated by a slash – the actual score and the number of tests that were in the system when the given product was tested. You can switch between the views using the link below. This table also shows the exact version of every tested product. The Level reached column presents the highest level that the product reached in Proactive Security Challenge 64. If it passed all levels, this number is suffixed with a plus sign. For products that scored at least 80 % in Proactive Security Challenge 64, the Recommendation column contains links to the online stores or products' web pages of the vendors that we have affiliate agreements with. If you click on any of these links and then buy the recommended product or other product offered on the target web page, we will profit from it. This is one of the ways how you can support this project. The PDF document icon in the Report column allows you to download the testing report in PDF format for the tested product. The last column of the table shows stars for products that received Proactive Security Challenge 64 Awards. Only products that were tested against the latest set of tests and reached at least the Very good protection level are awarded. You can see the award logo if you put your mouse cursor over the award star. The vendors of awarded products may ask us for copies of award logo images to be used on their web sites or products' boxes.

Detailed results

The following links take you to pages with detailed products' results on each level. The level pages also contain important information about the given level and short information about its tests.

• Level 1 – Autorun12, Autorun3, Autorun9, Coat, FileDel2, Kill1, Kill2, Leaktest, Tooleaky, Yalta
• Level 2 – Autorun15, Autorun31, Autorun7, ECHOtest, FileWri1, Jumper, Kill4, Schedtest, Suspend1, Wallbreaker4
• Level 3 – Autorun10, Autorun4, AWFT4, ECHOtest2, FileDel1, HostsBlock, Keylog3, Kill6, RegDel1, Suspend2
• Level 4 – Autorun19, Autorun20, Autorun37, Crash1, FileMov1, Keylog4, Kill9, ProxyTest, SSS2, VBStest
• Level 5 – Autorun24, Autorun26, Autorun29, CopyCat, Crash2, DDEexec, FileWri2, Keylog7, RegSet1, Schedtest2
• Level 6 – Autorun25, Autorun28, Autorun36, Breakout1, Crash3, FileWri3, FireHole2, Inject2, Keylog5, SSS3
• Level 7 – Autorun17, Autorun23, Autorun41, Crash4, FileCtl1, FireHole, Keylog6, Kill8, RegDel2, Svckill
• Level 8 – Autorun38, Autorun5, Autorun8, Crash5, DDEtest, FileDel3, Flank, NewClass, Runner2, SSS4
• Level 9 – Autorun34, Autorun43, CPILSuite2, Crash6, DNStester, FileMov2, FileRep1, Keylog1, Kill12, Schedtest3
• Level 10 – Autorun39, Autorun44, Cliplog, FileOpn2, Inject1, Keylog2, Kill3e, OSfwbypass, RegAcc1, SockSnif
• Level 11 – Autorun40, Autorun42, BITStest, FileAcc1, FileRep2, Kill3f, Kill5, Screenlog, Schedtest4, WFPblock

Interpretation of results

The best product tested against the latest set of tests and the Proactive Security Challenge 64 leader is currently Comodo Internet Security Premium 5.9.219863.2196 with excellent 94% score. The difference between the leader and the second product is enormous. Comodo Internet Security Premium is the only product that reached the Excellent protection level and has been awarded with Proactive Security Challenge 64 award. Jetico Personal Firewall 2.1.0.10.2451 is on the second place with 59 %, closely followed by Privatefirewall 7.0.25.5 with 56 %.

Proactive Security Challenge 64 makes a big difference between good products and the rest of the world. Most of the products are filtered in very low levels which means that they probably miss some critical features. There are plenty of products whose vendors claim that they can protect you against all kinds known and unknown threats. But in fact their products implement just a basic set of features that can prevent the most simple attacks only. If a malicious application uses more advanced approach to achieve its goal it is not blocked. This is very common in case of self-defense features, anti-leak protection, or system infection prevention.

Differences between quality of protection on the 64-bit platform (tested in Proactive Security Challenge 64) and the 32-bit platform (formerly tested in Proactive Security Challenge) are also very interesting. The nature of the 64-bit platform makes it uneasy for the products to protect against some of the advanced attacking techniques, but most of the techniques can be prevented by the very same mechanisms as on the 32-bit platform. This is why one would expect to see the 64-bit results of the products to be just slightly worse than the results on the 32-bit platform. This is not the case, however. The protection of many products is on a totally different level on the 64-bit platform, unfortunately much worse level.

However, it is crucial to know what does it mean if a product succeeds in our tests and what does it mean if it fails. Before you start interpreting the results, you should be familiar with the information on the index page, especially with the project's methodology and rules. You should also know what kind of products we test before you start to interpret the results. We have received a lot of reactions from people who are not familiar with that information and simply do not understand the results and misinterpret them. All the tested products have one common feature – the application-based security model. In combination with their packet filtering capabilities, the tested products attempt to block attacks from other machines on the network as well as attacks performed by malicious codes that might run inside the protected machine. This is definitely not an unusual situation. People who use email clients, instant messengers, or web browsers face attacks that exploit the vulnerabilities in this kind of software very often. It happens that a malicious code gets inside the machine. And then it may try to install itself silently to the system, to steal users' data or sniff their passwords, or to join the target machine to a botnet. This is what the products we test want to prevent. This is why they are used. The problem is that although the goal is common, not all the products implement a sufficient protection.

We require the products tested in Proactive Security Challenge 64 to prevent data and identity theft. They should also implement a packet filter functionality to prevent direct online attacks – i.e. not to let the malware get in. The products should control the software installed on the computer to prevent the malware to integrate itself into the operating system. Then the malware should not be able to get the user's private data, thus anti-sniffing, anti-keylogging and personal data protection features should be implemented too. And even if the malware succeeded to collect the information it should not be allowed to send it outside the protected system and this means an implementation of the outbound network traffic control. To achieve all these is much harder task than it seems. The protection system also has to prevent attacking trusted processes and other components in the system. Otherwise, the malware would be able to use trusted parts of the system to integrate into the operating system, to collect or steal sensitive data and/or to send the data outside the system without being noticed. So the next feature that is required here is a control of untrusted processes' activities and that is the hardest task for the tested products. It also includes an implementation of self-protection mechanisms because the malware should not be able to terminate the protection, which implies some other features to be implemented and so on.

So, what does it mean if the product fails even the most basic tests of our challenge? It means that it is unable to do what its vendor claims it can. Such a product can hardly protect you against the mentioned threats. On the other hand, if the product succeeds in all our tests, it does not mean that it is perfect. Our tests are focused on the security and stability, but there are many other aspects important for the users like performance, hardware requirements, easy to use, availability of support, price, vendor's reaction time to new threats etc. It should also be noted that although our testing suite is quite large, it is not complete and there may exist many other ways to bypass the tested products. We are working constantly on extending the suite to be able to provide more accurate information about the security of the tested products. If the tested product fails only a few tests in our challenge, it still might be a great product. This is why we can recommend, from the security point of view, the products that reached at least 80% score in the challenge. You should try them yourself and choose the one best for you, the one that you would be happy with, the one you would be able to configure and use everyday.

Note that we test the products in their highest usable security settings configuration. This means that the results reflect the best the products can do. However, out of box settings of many products may provide much less security. It is not rare that the default configuration of a product is set to a medium level of protection in order to achieve a better user experience.

Vendors' responses

We have received following responses to Proactive Security Challenge 64:

Comodo Security Solutions, Inc. – the vendor of Comodo Internet Security Premium

2012-01-16 (Comodo Internet Security Premium 5.9.219863.2196 scored 94 % and took 1st place): Thank you for including our product in your latest tests. Your open-source, open methodology tests are always very well thought out and unique. The latest results indicate the fact that there is a reason for us to be able to offer a 500 USD virus free guarantee in all operating systems. Keep up the good work!

Egemen TAS,
Director, Desktop Security Products,
COMODO

PWI, Inc. – the vendor of Privatefirewall

2012-01-16 (Privatefirewall 7.0.25.5 scored 56 % and took 3rd place): While a #3 ranking is not very satisfying when combined with the Protection level earned by Privatefirewall in this initial round of the new x64 PSC, we are aware of the work required on our end to dramatically improve our performance and the effort to do so is already underway. We thank the Matousec organization for their continued commitment to helping vendors improve their products and ensuring that the consumers and businesses that rely on them are able to realize the greatest levels of system protection.

Greg Salvato
CEO - Privacyware

valid HTML 4.01 · valid CSS 2 · optimized for you · design by martina · powered by jan · generated on February 23, 2012