Do you know how strong your Security Suite is? Detection rates of 148 Security Suites

[Christy]

This project examines security software for Windows OS that implement application-based security model – i.e. most of the products called Internet security suites, personal firewalls, HIPS, behavior blockers and similar products on the market

There are several testing levels in Proactive Security Challenge. Each level contains a selected set of tests and it also contains a score limit that is necessary to pass this level. All products are tested with the level 1 set of tests. Those products that reach the score limit of level 1 and thus pass this level will be tested in level 2 and so on until they reach the highest level or until they fail a limit of some level.

Testing suite and scoring

Most of the tests are part of Security Software Testing Suite, which is a set of small tests that are all available with source codes. Using this open suite makes the testing transparent as much as possible. For each test the tested product can get a score between 0 % and 100 %. The tests can be simply passed or failed only and so the product can get 0 % or 100 % score only. It should be noted that the testing programs are not perfect and in many cases they use methods, that are not reliable on 100 %, to recognize whether the tested system passes or failed the test. This means that it might happen that the testing program reports that the tested system passed the test even if it failed, this is called a false positive result. The official result of the test is always set by an experienced human tester in order to filter false positive results. The opposite situations of false negative results should be rare but are also eliminated by the tester.

To be able to make right decisions in disputable situations, we define the test types. Every test has a defined type. Tests of the same type usually attempt to achieve the same goal. Here is a list of the defined types and their goals:

* Leak-test: Leak-tests attempt to send data to the Internet server, this is called leaking. Most of the leak-tests from Security Software Testing Suite are configured to use a script on our website that logs leaks to our database by default. For such tests, you can use My leaks page to see whether the test was able to transmit the data. For leak-tests that do not use this script, we use a packet sniffer in unclear situations. In order to pass many leak-tests, the tested product has to implement some host protection features.
* Spying test: These tests attempt to spy on users' input or data. Keyloggers and packet sniffers are typical examples of spying tests. Every piece of the data they obtain is searched for a pattern, which is defined in the configuration file. These tests usually succeed if the given pattern has been found.
* Autorun test: These tests attempt to install to the system in order to ensure they will be started again. The most common goal in case of these tests is to survive the reboot. Such a system infection is typical for various kinds of malware. The tested product fails the autorun test if the test is able to ensure that it will be started in the future again.
* System integrity test: One of the roles of security suites is to protect the system integrity from malicious modifications. System integrity tests attempts to gain enough privilege in the system so that they are able to subvert the system.
* Self-defense test: This category of tests include various attacks against the security product itself. Termination tests are the first subtype of tests that belongs in this category. These tests attempt to terminate or somehow damage processes, or their parts, of the tested product. The termination test usually succeeds if at least one of the target processes, or at least one of their parts, was terminated or damaged. Besides processes and threads, the security software usually relies on various files and registry entries. Tests that attempt to remove, destroy or corrupt these critical objects for the security product also belong to this category.
* Other: Tests that do not fit any of the previously defined types are of this type. These tests, for example, may check stability, reliability or other quality of the tested product.

All tests are equal to the intent that their scores are not weighted by their level or something else. The total score of the tested product is counted as follows. For all tests in all levels that the product did not reach, the product's score is 0 %. For all other tests the score is determined by the testing. The total score of the product is a sum of the scores of all tests divided by the number of all tests and rounded to a whole number. It may happen that a new test is added to Proactive Security Challenge when some products already has their results. In such case, the result for already tested product is set to N/A for this new test, which means that it is not counted for this product and does not affect its score or level passing. Neither the number of the tests, nor the number of levels is final. We intend to create new tests in the future. We are also open to your ideas of new testing techniques or even complete tests.

All tests on the levels a tested product reaches are run at least once. If a product passes a test, this test is repeated at least once in order to mitigate false passing.

Detailed results

The following links take you to pages with detailed products' results on each level. The level pages also contain important information about the given level and short information about its tests.

* Level 1 – Autorun1, Autorun3, Breakout2, Coat, ECHOtest, FileDel2, Kill1, Kill2, Leaktest, Tooleaky, Wallbreaker1, Yalta
* Level 2 – Autorun12, Autorun2, Autorun20, Autorun30, AWFT1, DNStest, FileMov2, Ghost, HostsBlock, Jumper, Kill3, Kill3b, Kill6, RegDel1, Wallbreaker3, Wallbreaker4
* Level 3 – Autorun16, Autorun24, Autorun31, Autorun4, AWFT3, AWFT4, DNStester, FileRep1, Kernel1, Kill3f, Kill4, Kill7, RegSet1, SSS2, Suspend1, Thermite, Wallbreaker2
* Level 4 – Autorun14, Autorun17, Autorun26, Autorun36, Autorun37, Autorun6, Autorun9, CopyCat, CPIL, CPILSuite1, FileRep2, Inject2, Inject3, Kernel1b, Keylog1, Kill3e, Kill8, Kill9, SSS, Suspend2
* Level 5 – Autorun15, Autorun18, Autorun21, Autorun28, Autorun5, Breakout1, CPILSuite2, Crash1, Crash2, Crash3, Crash4, FileWri1, Kernel2, Kernel3, Keylog2, Kill3c, Kill3d, RegDel2, Svckill, VBStest
* Level 6 – Autorun22, Autorun25, Autorun27, Autorun29, Autorun32, Autorun7, CPILSuite3, Crash5, Crash6, DDEtest, ECHOtest2, FileWri2, FireHole, Flank, Kernel4, Keylog3, Keylog4, Kill10, Kill11, Runner
* Level 7 – Autorun10, Autorun19, Autorun33, Autorun35, Autorun8, BITStest, Crash4b, FileDel1, FileMov1, FileWri3, FireHole2, Inject1, Keylog5, Keylog6, Kill12, OSfwbypass, RegAcc1, Runner2, Schedtest, SSS3
* Level 8 – Autorun11, Autorun13, Autorun23, Autorun34, FileDel3, FileOpn1, FileOpn2, Kernel4b, Kernel5, Kernel5b, Keylog7, Kill5, NewClass, Schedtest2, SockSnif, SSS4
* Level 9 – Crash7, Driver Verifier, FileAcc1, FileCtl1, FileWri4
* Level 10 – BSODhook, ShadowHook

Matousec is credited with KHOBE – 8.0 earthquake Vulnerability for Windows desktop security software.

Results:

http://www.matousec.com/projects/proactive-security-challenge/results.php

Capture.jpg

[INDRANIL]

Good share . I Knew that earlier, personal testing free of cost for commercial license needed . Level 10 use at your own risk .

[Boyfriend]

Thanks Christy for sharing I also already knew about their latest results.

[paf]

Thanks a lot for the info and links, Christy . I wasn't aware of this latest results, they seem interesting.

[FunkY]

Thanks for the news Christy

After Comodo there is Online Solutions Security Suite. I've heard this and have been at their homepage but never tested. Anyone know anything about that OSSS?

[Neo]

what is this , avira: 3%
gdata :4 %

nice to read but cannot agree with it
very less samples and that too 100% detection making me dubious

[Funkysourav]

what is this , avira: 3%
gdata :4 %

nice to read but cannot agree with it
very less samples and that too 100% detection making me dubious

I agree with Neo
But I think i saw a trend in the tests:
The Internet Security Suites with good Firewall scored better
I have always suspected Avira PSS and GData Firewall to be a tad weaker than Comodo Free Firewall
maybe that was its Achilles Heel in this test

if i am not terribly mistaken the point of the test was Prevention of an attack not Detection(which Avira and GData are the best at)

People who use email clients, instant messengers, or web browsers face attacks that exploit the vulnerabilities in this kind of software very often. It happens that a malicious code gets inside the machine. And then it may try to install itself silently to the system, to steal users' data or sniff their passwords, or to join the target machine to the botnet. This is what the products we test want to prevent. This is why they are used. The problem is that although the goal is common, not all the products implement a sufficient protection.
We require the products tested in Proactive Security Challenge to prevent data and identity theft. They should also implement a packet filter functionality to prevent direct online attacks – i.e. not to let the malware get in. The products should control the software installed on the computer to prevent the malware to integrate itself into the operating system. Then the malware should not be able to get the user's private data, thus anti-sniffing, anti-keylogging and personal data protection features should be implemented too. And even if the malware succeeded to collect the information it should not be allowed to send it outside the protected system and this means an implementation of the outbound network traffic control. To achieve all these is much harder task than it seems. The protection system also has to prevent attacking trusted processes and other components in the system. Otherwise, the malware would be able to use trusted parts of the system to integrate into the operating system, to collect or steal sensitive data and/or to send the data outside the system without being noticed. So the next feature that is required here is a control of untrusted processes' activities and that is the hardest task for the tested products. It also includes an implementation of self-protection mechanisms because the malware should not be able to terminate the protection, which implies some other features to be implemented and so on.

I may be wrong
But it seems to me that they are stressing on the key points of a good firewall

[safeguy]

Guys, please read this to understand what Matousec testing is all about:

Matousec Personal Firewall Tests Analyzed

I am quoting the ones that I find deserving of a quick look for all of you lazy asses

The tested products must be able to block malware from freely running on a PC, from getting to a user's private data, from sending private data to outsiders, and from attacking trusted parts of a user's system (Interpretation of results). So, as an example, it only tests software that fulfills the condition that products in the test should prevent "data and identity theft" (Interpretation of results).

However, Matousec isn't very clear about the way it presents its results or, in any case, I think there are good arguments against the validity of product scores for software that did not receive all 10 actual tests. I will argue that the tests themselves are valid and interesting (with limitations and cautions), but the scoring of the tests (as presented on its comparison table) and some of the site's claims are misleading (see sections 1-2).

So readers should take caution that the test results are basically for experienced users, users who understand warning prompts, and users who can spend time and energy knowing their software and distinguishing between probable threats and non-threats. The results might be different if novice users took the tests, but then the results would vary greatly and would become much less objective (which is always a tricky give and take with testing methodologies, except for precise measurements of fundamental phenomenon in physics!) (see sections 3-4).

If a product performs badly on a "level" of testing, however, then it is not subjected to further testing, according to rules posted on the site. Hence, it organizes products by the overall number of possible tests rather than the total number of tests actually used on products.

If a product was not designed to protect against certain threats and does not claim to protect against them, then it is incorrect to claim that the goal of testing is to hold products accountable for the level of protection they claim to provide.

The Matousec results might suggest a maximum level of security for a product. Though, it is difficult even to make this kind of claim because the challenge does not fully test products with all actual tests. So for products that did not get far up the levels of testing, the Matousec scores do not suggest a maximum level of security. Products may even provide a higher level of security for both experienced and inexperienced users than the level of security suggested by the Matousec score.

If a low scoring product is user-friendly, then it may actually provide more security for inexperienced users than a complicated firewall. Likewise if a user is more knowledgeable of a low-scoring product, then such a user may be more secure with the lower scoring product than with a higher scoring product. If a product completely confuses a user, then it might provide lower security in the real world than in the tests.

Of course, any experienced user can use the same tests used in Matousec testing since they are located on the website for a free download (http://www.matousec.com/downloads/). Therefore, money can't plausibly influence the validity of the actual tests since the tests are available to everyone (though, #1-3 point out problems with the validity of the scoring of the tests and the tests may be modified during testing); money can only influence the way the site runs -- the politics!

The test results are linked by a PDF file and anyone can see the types of tests a product fails or passes. Since the raw data is posted to the site, you can completely ignore the overall score and just look at the tests passed or failed. But some raw data results were the product of some individual tester, and they may have made an error or interpreted the tests differently than someone else.

READ the whole thing if can and then we can sit down and talk about it....otherwise, we'll be beating around the bush. Matousec testing is neither about Firewall only nor a HIPS only. It's "Proactive Security Challenge" and comprises of different levels, quite messed up for most users who are unable to understand it. And there's a very good reason why Comodo always seem to be at the top of the list with 100% - you'll figure that out yourself when you can understand what Matousec testing and what Comodo's 'prevention' ideology is all about...(Defense+ guys )

In short - don't ponder too much over at Matousec results if you're unable to comprehend on what it shows. You'd do yourself no favor but a waste of time staring blindly at figures that depress you and may 'convince' you that you're living in a dark world with naked security and about to be raped by the monsters of evil hackers that waits right outside the locked door of your house. If you locked it that is - otherwise, you'll lose your virginity within a split second before you even manage to shout "No" Oh gosh, isn't that worrying?....

Or if you can understand it, then we can have a small chit-chat over here...I hardly do any testing or read-up on Matousec results usually but it'd be great to see someone who knows what he's talking about share a thing or 2 here...