Webroot tends to stand out in these tests, one way or another. In the last on-demand test, it almost reached the "crazy many" level of false positives. Based on detection it attained the top ADVANCED+ rating, but all those false positives knocked it down to the just-passing STANDARD level. In the latest performance test, Webroot beat all the rest, with the least impact of any product measured.
Today the lab has released its latest dynamic test. This time Webroot didn't suffer false positives; it earned its place at the bottom of the heap due to poor protection. Or so it would seem.
Test MethodologyThe dynamic test is a huge endeavor, a joint effort by AV-Comparatives and the University of Innsbruck, with additional funding from the Austrian government. An impressive automated system handles testing each product with each threat, recording how well the PC was protected.
The current report averages four months of testing. Five vendors received the top rating of ADVANCED+: Bitdefender, G DATA, Kaspersky, Qihoo, and F-Secure.
Webroot and AhnLab received the uncommon rating of TESTED, meaning they didn't achieve the minimum to pass the test. In PCMag's own tests AhnLab scored very poorly, but Webroot came out near the top. Click the image below to see a chart of results.
What Happened to Webroot?First, take a good look at the chart above. Note that its Y axis begins at 85 percent. If the graph started at zero the differences wouldn't be so pronounced. The vertical black line in each bar shows the lowest and highest monthly protection levels for each product. Referring to the full report, it's clear that Webroot had a bad March and a worse April. If evaluated just on its May and June performance, it would have at least reached the STANDARD rating.
I asked Mike Malloy, Webroot's Executive VP of products and strategy, just what went wrong. As it happened, he had visited AV-Comparatives in Innsbruck to go over the testing methodology and confer with the testing team. He explained that the automated testing system doesn't take into account the way Webroot handles brand-new threats. The test script tries to launch a malware URL and records whether the antivirus prevents installation, fails to prevent it, or leaves the decision up to the user. If nothing at all happens after several minutes, the script records a failure.
According to Malloy, when Webroot's cloud detection system spots an unknown program, the local Webroot client starts journaling everything the program does. The cloud system continues evaluating the file, sometimes with human intervention. The process can take minutes or hours. If the file does prove to be dangerous, the local Webroot client uses the activity journal to roll back every single recorded action.
I pointed out that actions like capturing passwords or sending out personal information can't be rolled back. Joe Jaroch, a leading Webroot researcher, explained that the product would generically block such bad behavior even before the cloud system had made its determination.
According to Malloy, Webroot's research team analyzed the missed samples and determined that given more time Webroot would have handled all but three. That would put its detection rate on a par with the top contender, Bitdefender. A post on Webroot's threat blog details the company's position on this test.
How Do We Know?In my own testing, I've observed behavior that matches what Malloy described. In my malware blocking test, Webroot initially deleted a handful of threats, then deleted a few more, and later a few more. Eventually it wiped out the entire collection, but it took a little while. Of course I don't know for sure what was going on inside the product, but the observed results jibe with Webroot's description.
Those with a suspicious bent may wonder how we can possibly know whether Webroot's claims are true. As far as I can see, the only possibility is for the company to commission a test that takes their product's peculiarities into account. There's no way AV-Comparatives can just give them more time, not without vastly reducing the number of brand-new threats they can test. A tailored test by a respected lab would settle the matter once and for all.
0 comments:
Post a Comment