Some thoughts on the CrowdStrike vs NSS Labs debacle

Regarding the CrowdStrike vs NSS debacle, I’d like to share some thoughts with you on the subject of anti-virus tests.

Let's start with CrowdStrike. All the “next-gen” anti-virus companies are sleazebags. They are new, they are hungry, they see the tons of money made by the “established” anti-virus industry and they are envious. They want their slice of the pie.

Problem is, it is goddamn impossible for a newcomer to do really well in this field. No, it is not fair — but this is how it is. The main reason is because most users simply cannot grasp how to use any kind of anti-virus product besides a scanner.

A behavior blocker?

Process X is trying to communicate over port Y.

WTF does that mean? Is it a worm trying to infect other machines? Or is a program trying to check its update server for a new version of itself?

Program FOO is modifying program BAR.

Again, WTF does that mean? Is a virus in FOO trying to infect BAR? Or is FOO part of a compiler/IDE that is trying to create the executable BAR? The average user doesn’t have a clue.

An integrity checker?

File SNAFU was modified.

Well, WTF does this mean? Was it infected by a parasitic virus? Or was it modified by Windows Update?

As opposed to that, a scanner is easy to understand.

File X contains malware Y.


File Z is clean.

Either or both of these messages could be totally wrong (the two kinds of errors are called “false positives” and “false negatives”) but they are something the user can understand.

So, no matter how advanced an anti-virus product is, it must include a scanner (or something that behaves like a scanner) — or the users simply won’t use it. This, despite the fact that scanners are the weakest defense against viruses and malware in general.

What do scanners do? They detect either “known bad stuff” or “stuff that contains code known to do bad things”. This worked quite well quarter of a century ago when all known malicious programs could fit on a 360Kb floppy disk but nowadays there is a humongous number of known malicious programs (anything between 10 and 200 million and anybody who claims to know their number more exactly is lying) and new ones are being created (often automatically) every goddamn day — at the rate of 1-2 million new ones every day, on average.

There is absolutely no way a newcomer to the anti-virus field could produce a scanner that can detect a reasonably large amount of the known malware. (And, remember, a scanner that detects less than 90% of the known malware is considered as being mostly crap.) They would need years just to catch up with the established scanner producers (who have been at this game for quarter of a century already) — and by the time they do, the number of known malicious programs would have increased by several orders of magnitude, so they would be already obsolete anyway.

So, what is a newcomer to do? Lie, cheat and muscle your way in. Claim that the established guys are “obsolete”, “legacy”, while their product uses completely new, next-generation ideas. (There is no such thing in reality, of course. Just about every single idea in this field was already invented and tried two decades ago. Heuristics, machine learning, everything. Different techniques work to a different degree and different companies put different emphasis on them. That’s it.) Don’t allow your product to be independently tested and evaluated by competent people. Use misleading advertising. Slander the established guys and the testers who show that your product sucks compared to the established ones. Sue, if you are in the USA. That sort of thing. And hope that enough users will be fooled to buy your product for you to make money.

Now that I’ve dealt with the “next-gen anti-virus” guys, let me look at the other side of the coin — the anti-virus testers.

Testing an anti-virus product properly is an impossibly hard job. I should know — I practically invented independent competent anti-virus testing while I was working at the Virus Test Center at the University of Hamburg in the early ’90s.

To begin with, since just about every self-respecting anti-virus product contains a scanner (see above), you need a comprehensive collection of known malware. It has to contain pretty much everything known to exist, because, remember, a scanner that detects less than 90% of it is crap and you have to be able to measure this. You have to be a competent malware researcher, in order to make sure that what is in your test set is indeed malware. And if you think that you can determine that just by running a scanner on it and observing its output, you are an idiot and your place is not in the anti-virus testing business.

But scanners are only one small part of the contemporary anti-virus suites. To test just them would be hugely unfair, because they have many other ways to protect the customer from a malware infection. And you have to test in realistic conditions. In real life, nobody is attacked by a multi-terabyte hard disk full of hundreds of millions of static malicious programs. In real life people are attacked by 1-3 malicious programs that they have executed on their computer.

So, you have to execute the malicious program on the machine protected by the anti-virus suite you’re testing. This might sound simple, but it is not. Malware would often refuse to run (or to do anything meaningful) for the weirdest of reasons. It might detect that you’re using a virtual machine. It might not like your machine’s IP address. It might require weird command-line arguments (that are normally provided by another piece of malware). Or it might just do its thing only on Friday the 13th, or only if the current year is earlier than 2009. Or maybe it is simply a corrupted sample that doesn’t work at all. And if it “doesn’t do its thing” on your machine, the anti-virus you’re testing might not stop it (I am assuming here that the on-access scanner missed it and let it run in the first place). But it might stop it on a real user’s machine, if the malware decides to try to “do its thing” there. And you have to catch that. So, you have to reverse-engineer the malware and figure out why it isn’t working.

And then you have to wipe the machine and repeat the exercise with another anti-virus, for each of the anti-virus products you are testing.

And once you’re done, you realize that you have just tested how these products protect from one particular malware. Now repeat the above procedure 200 million times for all the known malicious programs in your (supposedly good) malware collection.

And once you’re done (good luck with that), you must also test all the products for false positives! Remember, the user is unable to tell whether a report from an anti-virus product is correct or not, so a wrong “this is malware” report can cause nearly as much damage as missing real malware (e.g., resulting in the anti-virus product deleting a program Windows needs to work and disabling your whole corporate network), so a good test needs to test for such things, too.

It’s an impossible task!

So, all anti-virus testing outfits generally fall into two categories — incompetent and incomplete. (Of course, some are both.) The incompetent ones often publish in popular computer magazines. They “test” irrelevant things like the user interface or whether the documentation is easy to understand. They use “malware” which somebody or something (usually — a scanner) told them is malware. They are idiots.

The incomplete ones are the ones you usually see as professional independent anti-virus testing outfits — like AV Comparatives, Virus Bulletin, and so on. They have a generally sound testing methodology, but it is necessarily very limited — because, as we saw above, proper and complete anti-virus testing is simply not humanly possible. So, they use a small (and often obsolete) test set like a few hundred malicious programs, or only test one particular aspect of the anti-virus product (e.g., the scanner) and so on.

A favorite pet peeve of mine was tests using the so-called WildList — a (very short) list of (mostly) viruses that were supposed to be “in-the-wild” — i.e., what was actually attacking users currently. Except that it was no such thing, it was full of crap, it wasn’t current, it was too short and so on and so on. So, anyone testing only with it was an idiot. (Yet many anti-virus products regularly failed even such tests.)

In addition to these two categories, some anti-virus testing outfits are both incompetent and greedy. They charge outrageous amounts of money for their tests. This is not entirely unreasonable, given how hard and time-consuming it is to get a good anti-virus test done right — but it gives the tested companies the impression that they are “buying” (paying for) tests, so it naturally annoys them when the results are not to their liking. Sometimes the situation is so bad that I’ve had one anti-virus producer tell me “we’re paying thousands for this shit, and then I have to spend days doing the tester’s job for them and explain them why some malware samples are corrupted and non-working and shouldn’t be in the test set, why some program labeled as a false positive isn’t a false positive because it does questionable stuff just as our product reports, and why they haven’t understood (or even read) the documentation”.

So, there’s that. Whenever there is a major conflict, like this CrowdStrike vs NSS Labs story, you can usually bet that both sides are in the wrong. CrowdStrike probably have a crappy product they want to sell and didn’t like the test results, while NSS Labs probably has a crappy and/or incomplete testing methodology and CrowdStrike found some legitimate flaws in it.

My personal opinion is that the companies whose products are being tested shouldn’t pay for the tests — the users of the tests should pay for the results. Also, if a company does not want its product tested by a particular testing outfit, that outfit shouldn’t test it (instead of jumping through hoops to obtain secretly a copy of the product and test it). Let the competitors use this for their advantage (“Ha-ha, company X is afraid to have their product tested independently — what are they hiding?”).

But, all in all, anti-virus testing is a thankless job, impossible to do right, and the anti-virus industry is full of snake oil peddlers.

End of rant.