matousec.com (site map)

Poll

How much system resources could your security products consume at most?

  → 0% - 1% (25.28%)

  → 2% - 5% (39.89%)

  → 6% - 10% (15.17%)

  → 11% - 15% (7.3%)

  → 16% - 20% (5.06%)

  → 21% - 25% (2.25%)

  → 25% - 30% (0.56%)

  → 31% and more (4.49%)

more

results

Firewall Challenge

Frequently asked questions

Contents:

Firewalls versus Firewall Challenge, instead of malware

Question: How is avoided a danger that FW vendors will start to focus on fighting against FW challenge instead of malware? The reason can be immediate positive business impact of successfully passed tests.

Answer: We have faced this problem since leak-testing. Some vendors really fight the tests and not their attacking techniques. Some vendors optimize against the given set of tests rather than solving the causes.

If we have a suspicion that the tested product attacks some test directly, we use internally modified versions of the tests to prove it. If we can prove such behavior, we mention this in the report and the product fails the test.

Another situation is when the vendors blindly add functionality to their software to pass some technique. In such case, their users might be confused by absurd, false, misleading or somehow bad alerts, popups and questions. In this case, such a product might get through our tests but it would be unusable for most of users. We hope that vendors will not do this for their own good.

To prevent the unwanted behaviour of the vendors, we are going to add new tests to the system and test selected products against the new tests without prior notices to their vendors. For this purpose we will select, preferentially, the prodcuts of those vendors that concentrate on fighting the tests instead of the real security of their products. This approach should give us more accurate results in a sense of their real security.

Finally, we have also set a fixed rules about the frequency of testing, this should also help. However, our original rules about paid retesting allowed vendors to make quick silent fixes and order retesting with the only intention of replacing the old results with new and better results. This is why we have added new rules that limit paid retesting too.

Termination tests' methodology

Question: The methodology for termination tests seems to indicate that termination of any of the firewall's processes results in a failure in the test. I disagree with that methodology as the main features of the firewall may be unaffected by the termination (e.g. if the process that was terminated was only the tray icon) or the firewall may have some kind of "fail-safe" (e.g. blocking all connections if the processes are not running). I think a test (e.g. "leaktest.exe") should be run after a termination to see if the protection is still working or not. If the firewall stopped the test after the termination it should receive a partial score (e.g. 50% of the normal score for the termination test).

Answer: The idea behind our scoring system is the simplicity of the tests. We can not really say how the termination of one component affects the whole protection system unless we analyse the system deeply. We do not do that in Firewall Challenge. Imagine a product that implements the GUI component which communicates with the user. Imagine that if this component is terminated, the product blocks all connections to the Internet. You say that if we run "leaktest.exe" to verify the protection, it will tell us whether the protection is weakened.

In a classic model of a driver, service and GUI component there are communications channels opened between these components. And these channels may be implemented so that only one connection is allowed to prevent malicious software to connect to the channel and send requests over it. If the GUI component is terminated, it may become possible to connect to these channels and attack the service or driver component through them. The verification you suggest does not reveal this case and there are many other situations that should be verified before we could say that the protection was not weakened.

Termination of any of the product's component is a security issue. In our scoring system it is penalized and we are not aware of any easy modification that would make the system more accurate or more fair.

Administrator's or limited account

Question: I'm just curious if these tests are carried out under an administrative or limited account? Thank you in advance!

Answer: According to our poll more than 80% of people use the full privileged account. This is why we perform our tests under administrator's account, to be as close to the real scenario as possible.