AMTSO Publishes New Guidelines for the Anti-Malware Testing Industry

[sujay]

Most tests focus only on simplistic detection rates, in some cases actually disabling part of the product’s functionality in order to isolate a single protective layer. “Whole Product Protection Testing Guidelines” advocates a more balanced look at the effectiveness of products, taking into account multiple layers of detection and protection. “This Guidelines document marks an important step in developing tests which accurately measure how an entire product actually functions when exposed to threats,” said Dr. Igor Muttik, representing McAfee. “Too many current tests focus on individual technologies, such as ‘On Demand Scans.’ Only by testing all of a product’s protection capabilities in a comprehensive test, can one provide a more realistic view of the security offered to computer users by contemporary security suites. ”

“Performance Testing Guidelines” illustrates that it is as easy to over-emphasize irrelevant metrics by implementing poorly-conceived benchmarking methodologies, as it is to bias a detection test. “The Performance Testing Guidelines examines the myriad – and often subtle – complexities in conducting speed tests,” said Mikko Hypponen, Chief Research Officer of F-Secure. “It is very tempting to take a simplistic approach to measuring speed and footprint of an anti-virus program. However, there is as much art as there is science in understanding the various elements which can skew the results for the unwary tester. This document will help testers understand these issues and allow them to take the necessary steps to minimize them and take them into account.”

Read More
http://www.amtso.org/pr-20100610-amtso-publishes-new-guidelines-for-the-anti-malware-testing-industry.html

[dredge]

just another test, I bet no different from any other tests as what important for a security software is how the security software detects malware and blocks them.

[Ceyfer √]

AMTSO = Standard.

It's fine, a good revision and I do support it.

[leofelix]

I have only a doubt: how can they consider "simplicistic" an on demand (and above all independent) test?
There are some security softwares which actually offer only on demand detection.

They decide what standard is, don't they?
Even Internet Explorer and its trident engine has been considered as a standard for years while actually was not (just an example).

Latins were used to say: "Quis custodiet custodes?" ( http://en.wikipedia.org/wiki/Quis_custodiet_ipsos_custodes%3F )

[Ceyfer √]

Good point mate, but let's hope it can do it's job now, the aim is good but always a challenging one: because there's no such thing as a perfect methodology.

[leofelix]

@ Ceyfer: I agree: let's hope

a small off topic: congrats for your new avatar, it drawed my attention immediately.
Believe me, you are always pleasently surprising not only in regards to security

[Ceyfer √]

OT:

Big thanks leo. Also, it is my pleasure to see that you're now a Mod here.

[sujay]

Yep, really an eye-candy avatar..

[Ceyfer √]

Update on Telemetry Usage in Tests, Part 1

Almost a year ago, I wrote a blog on promoting the use of telemetry when anti-malware testers compile their set of malware to run tests. I thought it might be time to give people an update. Basically, changing testers' habits is like the proverbial turning of a battleship. Testers use tried and true methodology. And it's important for the consumers of the test results to have consistent methodology to compare past results with present ones to build a pattern of progress. So, even to improve the tests, it has to be done delicately and with broad support, and sound reason.

Microsoft Malware Protection Center | Here

[safeguy]

They can do or think up of whatever changes they want....I'm more interested in reading the data and analysis once they are made