Antivirus Fight Club
Is there a conspiracy here?
Dirk Morris, the CTO of Untangle, thinks so.
Two years ago, Morris did a “bake-off” with a leading open source antivirus (AV) scanner, Clam AntiVirus. He pitted it against a bevy of proprietary AV engines to see how each one did against a bucketful of viruses he had pulled out of the wild.
Morris, whose company sells gateway appliances stocked with open source security software to small businesses, said he was astonished by the results. He figured AV engines were somewhat commoditized, but some big-name vendors, who shall remain nameless for now, had success rates as low as 30%. These were NOT zero-day viruses he was throwing at them. He was pulling stuff from the wild, viruses that had been around for a while. And still, some AV engines had no signatures to fight them.
Looking to get these results reproduced for public consumption, Morris approached a leading testing lab and asked it to test Clam against some of the commercial vendors. The lab, which he declined to name, turned him away.
“They said ‘We won’t test Clam because it’s open source.’ They refused to give us any explanation,” Morris said.
Morris cried foul play. He suggested that the lab didn’t want to show how well an open source project like Clam stacked up against the proprietary heavyweights.
“If they said the open source project was the best product, they’ll have a lot of unhappy customers and they won’t get any paying customers anymore.”
With that in mind, Morris is staging an “Anti-Virus Fight Club” at next week’s LinuxWorld. It isn’t exactly Brad Pitt versus Edward Norton. And Dirk Morris is no David Fincher, but this should be worth checking out.
Morris will take 20 viruses that are roaming about on the Internet and introduce them to machines protected by each AV product. Who will be left standing, and who will be left a bloody mess on the floor, is anyone’s guess. Morris is quite sure Clam, the open source champ, will leave most of the big guys in the dust.
AV scanners from Norton, McAfee, Fortinet, WatchGuard, SonicWall, Hauri, F-Prot, Sophos and Kaspersky Lab will all face the same 20 viruses as Clam, Morris said.
“I suspect what will happen is Clam will be the best,” Morris told me. “Kaspersky will do pretty well. They’ve got a good product. Some of the other vendors will do pretty poorly.”
Interested parties should check out the Fight Club on Aug. 8 from 6 to 7 p.m. at LinuxWorld in San Francisco. It’s a Birds of a Feather session.
Posted: August 3rd, 2007 under Uncategorized, Security.
I also would be very interested in the results of anti-virus bake off, but your missing Trendmicro products and I would like to see it added to the fight.
Trendmicro has been a growing player in the market and I have noticed it beat Norton, and McAfee with my customers and think they have one of the better products.
I hope Dirk Morris will consider adding to his list of contenders.
Comment by Scott Jenkins — August 7, 2007 @ 10:40 am
I wish I could find whatever the (*&*&^ has screwed up this XP machine I am trying to clean - none of the commercial guys found anything but it still doesn’t work. It would be nice to0 let Mr. Morris test them on finding it.
Comment by Don Thompson — August 7, 2007 @ 1:22 pm
I think Clam HAS been tested at some point or another against other AVs - and it came up poorly.
Check this article out:
http://www.isp-planet.com/equipment/2006/clam_av.html
“AV-Test.org’s Andreas Marx agrees that ClamAV’s strength lies in its speedy response to new outbreaks. He notes, though, that it has some problems with false positives and detection rates. “It only has an 85 percent detection rate in the case of WildList viruses and a 35 percent detection rate for Zoo malware, while commercial scanners usually have 100 percent WildList and 95 percent + Zoo detection rates,” Marx says.
Strength in numbers
Yankee Group Senior Analyst Andrew Jaquith says ClamAV does extremely well in competition with commercial solutions—but he advises against deploying any single anti-virus solution on its own. “We think that doubling up coverage on your AV is a good idea,” he says. “And Clam could certainly be an example of a second engine to provide that coverage.”
Jaquith notes that many companies have expressed an impressive amount of confidence in ClamAV. “Mac OS X Server, for instance, bundles ClamAV, so they think it’s good enough,” he says. “Barracuda bundles ClamAV, and a variant of IBM’s AIX has it as an option as well—so it’s enough for many uses.”…
But like Jaquith, Cornet says it’s always a good idea to deploy more than one AV product—in fact, he says ClamAV is XS4ALL’s third scanner, and for good reason. “ClamAV actually only scans for about 46,000 different viruses, where a commercial scanner detects maybe 100,000 different viruses,” he says. “But in practice it doesn’t matter, because Clam catches practically every e-mail borne virus that’s out there—so the fact that the other scanners pick up the occasional boot sector virus from 1995 that’s still floating around is irrelevant.”
I’d say Clam is probably good enough for home users, but a corporate operation is better off with a manageable AV like Kaspersky. And of course, never use Norton - that thing is a disaster that does nothing but consume system resources and interfere with other programs and frequently hoses itself and can’t even update its own virus definitions.
Comment by Richard Steven Hack — August 7, 2007 @ 2:51 pm
why will this surprise anybody?
open source is the best in almost everything.
is not conspiracy, is just plain result of most companies burocracy that goes against good results.
put the same guys (the peons)in open source and they will do miracles.
Comment by eduardo — August 7, 2007 @ 3:45 pm
[…] Antivirus Fight Club […]
Pingback by SMB Weekly Roundup for August 8, 2007 — SMB Weekly Roundup — August 9, 2007 @ 10:37 am
So where are the test results? 20 viruses is hardly a significant test. Why dont they test with 200,000? Lets see how they fare in a false positives test. Last results I saw showed Clam off the charts in false positive rates, which may be ok for e-mail but hardly idea in a server environment when your system files needlessly end up in the trash, and subsequently your server fails.
Comment by Mike Grant — August 10, 2007 @ 1:15 pm
[…] The results are in from the Antivirus Fight Club at LinuxWorld, and, as expected, Clam AntiVirus, an open source project, did quite well when measured against commercial products.The competition was much more involved than initially described to me by Dirk Morris, CTO of Untangle. There were three rounds in the fight. Each antivirus product faced three groups of viruses. […]
Pingback by And the last AV scanner standing is… — SMB Connection — August 10, 2007 @ 3:28 pm
Help, please (from author or anyone):
What happened to all the Anti-Malware Product Evaluation organizations that used to thoroughly test Anti-Virus products and similar and rate them?
If there are any trustworthy ones left, please lest their URLs.
Thanks.
Comment by Jim Harris — September 6, 2007 @ 12:27 pm