matousec.com (site map)

Poll

On Windows Vista I use

  → unlimited administrator's account (26.11%)

  → limited administrator's account (9.09%)

  → common user's account (3.26%)

  → nothing (I do not use Vista) (61.54%)

more

results

Firewall Challenge

Testing levels

Contents:

Back to contents

Level 1

The product has to score at least 50% in the tests on this level to pass it.

Back to contents

Tests


Breakout2
Test type: Leak-test
Scoring: Failure – 0%; Success – 100%.
Description: Breakout2 checks whether it is possible to bypass your firewall using Active Desktop COM interface.

Coat
Test type: Leak-test
Scoring: Failure – 0%; Success – 100%.
Description: Coat checks whether your firewall identifies the trusted processes properly.

ECHOtest
Test type: Leak-test
Scoring: Failure – 0%; Success – 100%.
Description: ECHOtest finds out whether your firewall filters ICMP traffic.

Kill1
Test type: Termination test
Scoring: At least one of the target processes or threads was terminated and the handle of at least one of the terminated processes or threads was obtained directly – 0%; the handles of all the terminated processes or threads were obtained using the handle stealing method – 50%. Success – 100%.
Description: Kill1 tests whether your firewall protects its processes against termination. If your firewall fails this test, it probably means that it does not implement self-defense mechanisms at all.

Kill2
Test type: Termination test
Scoring: At least one of the target processes or threads was terminated and the handle of at least one of the terminated processes or threads was obtained directly – 0%; the handles of all the terminated processes or threads were obtained using the handle stealing method – 50%. Success – 100%.
Description: Kill2 tests whether your firewall protects threads of its processes against termination.

Leaktest
Test type: Leak-test
Scoring: Failure – 0%; Success – 100%.
Description: Leaktest finds out whether your firewall filters outbound TCP traffic.

PerfTCP
Test type: Performance test
Scoring: The performance reduction is crucial for the score of this test. The product scores 100% if the network performance was not reduced below 90% of the original performance. This means that the test "tolerates" 10% performance consumption. If the performance is less than 90% of the original performance then the score is computed proportionally (e.g. 50% test score means that the performance was reduced to 45%).
Description: This test measures the network throughput. The TCP protocol is used. The products are tested on a 100 Mbit/s network. The test is launched three times on the testing machine before the installation of the tested product and three times after its installation and configuration. The final result times are averages of all three measurements. The final result time measured after the installation of the tested product is then compared with the final result time measured on the clean machine.

PerfUDP
Test type: Performance test
Scoring: The performance reduction is crucial for the score of this test. The product scores 100% if the network performance was not reduced below 90% of the original performance. This means that the test "tolerates" 10% performance consumption. If the performance is less than 90% of the original performance then the score is computed proportionally (e.g. 50% test score means that the performance was reduced to 45%).
Description: This test measures the network throughput. The UDP protocol is used. The products are tested on a 100 Mbit/s network. The test is launched three times on the testing machine before the installation of the tested product and three times after its installation and configuration. The final result times are averages of all three measurements. The final result time measured after the installation of the tested product is then compared with the final result time measured on the clean machine.

Tooleaky
Test type: Leak-test
Scoring: Failure – 0%; Success – 100%.
Description: Tooleaky checks whether your firewall controls Internet Explorer launching.

Wallbreaker1
Test type: Leak-test
Scoring: Failure – 0%; Success – 100%.
Description: Wallbreaker1 checks whether your firewall controls Windows Explorer launching.

Yalta
Test type: Leak-test
Scoring: Failure – 0%; Success – 100%.
Description: Yalta checks whether your firewall filters outbound UDP traffic.

Back to contents

Result table

In the following table 100 represents the 100% result and 0 represents the 0% result. Other values are displayed as rounded whole numbers. The last two columns summarize the product's score on this level and whether it passed this level or not.


 
Product I. ECHOtest I. Leaktest I. Tooleaky – – ScoreResult
II. Coat II. Kill2 II. PerfUDP II. Yalta
III. Breakout2 III. Kill1 III. PerfTCP III. Wallbreaker1
III. II. I. III. II. I. III. II. I. III. II. -
Ashampoo FREE 0 0 0 100 0 100 100 100 0 0 0 – 36%FAILED
AVG 0 100 100 0 0 100 N/A N/A 0 0 100 – 44%FAILED
Avira 0 100 100 0 0 100 N/A N/A 100 0 100 – 56%PASSED
BitDefender 0 100 0 0 0 100 N/A N/A 0 0 100 – 33%FAILED
BullGuard 0 100 0 0 0 100 N/A N/A 0 0 100 – 33%FAILED
Comodo 100 100 100 100 100 100 81 84 100 100 100 – 97%PASSED
DSA 100 100 N/A 100 100 100 100 98 100 100 100 – 100%PASSED
ESET 0 100 0 0 0 100 N/A N/A 0 0 100 – 33%FAILED
Filseclab 0 0 0 0 0 100 100 29 0 0 0 – 21%FAILED
FortKnox 100 100 100 0 0 100 N/A N/A 100 100 100 – 78%PASSED
F-Secure 0 100 100 100 100 100 96 85 0 0 0 – 62%PASSED
G DATA 100 100 0 0 0 100 45 54 100 0 100 – 54%PASSED
iolo 0 0 0 0 0 100 N/A N/A 100 0 0 – 22%FAILED
Jetico v2 100 100 100 100 100 100 93 92 100 100 100 – 99%PASSED
KIS 100 100 0 100 100 100 N/A N/A 100 100 100 – 89%PASSED
Lavasoft 100 100 100 100 100 100 68 71 100 100 100 – 94%PASSED
Look 'n' Stop 0 100 100 0 0 100 N/A N/A 100 0 100 – 56%PASSED
McAfee 0 100 100 0 0 100 N/A N/A 0 0 100 – 44%FAILED
Norton 0 100 0 100 100 100 N/A N/A 100 0 100 – 67%PASSED
OneCare 0 100 0 0 0 100 N/A N/A 0 0 100 – 33%FAILED
Online Armor 100 100 100 100 100 100 96 65 100 100 100 – 96%PASSED
Online Armor Free 100 100 100 100 100 100 91 66 100 100 100 – 96%PASSED
Outpost Pro 100 100 100 100 100 100 72 68 100 100 100 – 95%PASSED
Panda 0 100 0 100 0 100 76 86 0 0 100 – 51%PASSED
PC Tools 100 100 0 100 100 100 100 100 100 100 100 – 91%PASSED
Privatefirewall 100 100 0 100 100 100 100 100 100 100 100 – 91%PASSED
ProSecurity 100 100 100 100 100 100 N/A N/A 100 100 100 – 100%PASSED
Rising 0 0 100 0 0 100 N/A N/A 0 0 100 – 33%FAILED
SSM 100 100 100 100 100 100 N/A N/A 100 100 100 – 100%PASSED
Steganos 0 0 0 0 0 100 N/A N/A 0 0 100 – 22%FAILED
Sunbelt 0 100 0 0 0 100 26 68 100 100 0 – 45%FAILED
TrendMicro 100 100 0 0 0 100 83 95 100 100 0 – 62%PASSED
Webroot 100 100 0 100 100 100 100 97 100 100 100 – 91%PASSED
ZoneAlarm Pro 100 100 100 100 50 100 100 61 100 100 100 – 92%PASSED
Back to contents

Levels