On Wed, 9 Jun 2004 Valdis.Kletnieks@vt.edu wrote:

A writeup on the OpenSSL holes, the Slapper worm, and when/why users patched their systems. 17 pages, PDF.

http://www.rtfm.com/upgrade.pdf

Lots of interesting conclusions about user behavior, which we probably need to consider when planning. Some non-trivial math/stats, but they explain what the results mean in plain English too, so feel free to skip over the formulas to the "this clearly shows..."..

I've been calling this the 40/40 rule. What's interesting is how consistant it remains, regardless of the timeline, exploit or publicity. About 40% of the vulnerable population patches before the exploit. About 40% of the vulnerable population patches after the exploit. The numbers vary a little e.g. 38% or 42%, but the speed or severity or publicity doesn't change them much. If it is six months before the exploit, about 40% will be patched (60% unpatched). If it is 2 weeks, about 40% will be patched (60% unpatched). Its a strange "invisible hand" effect, as the exploits show up sooner the people who were going to patch anyway patch sooner. The ones that don't, still don't. Businesses aren't that different from consumers. A business is like a super-cluster of PCs. Don't think of individual PCs, but of clusters of sysadmins. The difference is the patching occurs in clusters. Sysadmin clusters follow the same 40/40 rule. If you have 1,000 businesses each with 10-1,000 computers, within a sysadmin cluster it tends to be a binary patched/not patched for 99% of the computers in the same cluster. But across 1,000 clusters of PCs; things don't look that different. About 40% of the clusters are patched before the exploit, about 40% are patched after the exploit. Sometimes the cluster has 1,000 patched computers, sometimes the cluster has 10 patched computers, somtimes the cluster has 1,000 unpatched computers. Don't mistake size for better managed.

Both of these papers are somewhat flawed in that they focus on the mostly-broken idea that the admin/user would even know a patch if it came by and bit them on the posterior.....

The good news is after the exploit, thanks to the invisible hand about 80% of the patching behavior occurs without a lot of extra prompting. The bad news is regardless of what actions are taken, about 60% PCs/clusters will be vulnerable when an exploit is released regardless of how long the patch has been available.

Valdis.Kletnieks@vt.edu writes:

On Wed, 09 Jun 2004 18:45:55 EDT, Sean Donelan <sean@donelan.com> said:

...

The numbers vary a little e.g. 38% or 42%, but the speed or severity or publicity doesn't change them much. If it is six months before the exploit, about 40% will be patched (60% unpatched). If it is 2 weeks, about 40% will be patched (60% unpatched). Its a strange "invisible hand" effect, as the exploits show up sooner the people who were going to patch anyway patch sooner. The ones that don't, still don't.

Remember that the black hats almost certainly had 0-days for the holes, and before the patch comes out, the 0-day is 100% effective.

What makes you think that black hats already know about your average hole?

Once the patch comes out and is widely deployed, the usefulness of the 0-day drops.

Most probably, 40% is a common value for "I might as well release this one and get some recognition". After that point, the residual value starts dropping quickly.

I don't think this assessment is likely to be correct. If you look, for instance, at the patching curve on page 1 of "Security holes... Who cares?" (http://www.rtfm.com/upgrade.pdf) theres'a pretty clear flat spot from about 25 days (roughly 60% patch adoption) to 45 days (release of the Slapper worm). So, one that 2-3 week initial period has passed, the value of an exploit is roughly constant for a long period of time. -Ekr

Valdis.Kletnieks@vt.edu writes:

On Thu, 10 Jun 2004 08:50:18 PDT, Eric Rescorla said:

...

Valdis.Kletnieks@vt.edu writes:

...

...

Remember that the black hats almost certainly had 0-days for the holes, and before the patch comes out, the 0-day is 100% effective.

What makes you think that black hats already know about your average hole?

Because unlike a role playing game, in the real world the lawful-good white hats don't have any deity-granted magic ability to spot holes that remain hidden from the chaotic-neutral/evil dark hats.

Explain to me why, given that MS03-039, MS03-041, MS03-043, MS03-044, and MS03-045 all affected systems going all the way back to NT/4, and that exploits surfaced quite quickly for all of them, there is *any* reason to think that only white hats who have been sprinkled with magic pixie dust were able to find any of those holes in all the intervening years?

Actually, I think that the persistence of vulnerabilities is an argument against the theory that the black hats in general know about vulnerabilities before they're released. I.e. given that the white hats put a substantial amount of effort into finding vulnerabilities and yet many vulnerabilities persist in software for a long period of time without being found and disclosed that suggests that the probability of white hats finding any particular vulnerability is relatively small. If we assume that the black hats aren't vastly more capable than the white hats, then it seems reasonable to believe that the probability of the black hats having found any particular vulnerability is also relatively small. For more detail on this general line of argument, see my paper "Is finding security holes a good idea?" at WEIS '04. Paper: http://www.dtc.umn.edu/weis2004/rescorla.pdf Slides: http://www.dtc.umn.edu/weis2004/weis-rescorla.pdf WRT to the relatively rapid appearance of exploits, I don't think that's much of a signal one way or the other. As I understand it, once one knows about a vulnerability it's often (though not always) quite easy to write an exploit. And as you observe, the value of an exploit is highest before people have had time to patch. -Ekr

Paul G <paul@rusko.us> wrote:

----- Original Message ----- From: "Eric Rescorla" <ekr@rtfm.com> To: <Valdis.Kletnieks@vt.edu> Cc: "Sean Donelan" <sean@donelan.com>; "'Nanog'" <nanog@merit.edu> Sent: Thursday, June 10, 2004 2:37 PM Subject: Re: AV/FW Adoption Sudies

-- snip ---

...

If we assume that the black hats aren't vastly more capable than the white hats, then it seems reasonable to believe that the probability of the black hats having found any particular vulnerability is also relatively small.

and yet, some of the most damaging vulns were kept secret for months before they got leaked and published. i won't pretend to have the answer, but fact remains fact.

I don't think that this contradicts what I was saying. My hypothesis is that the sets of bugs independently found by white hats and black hats are basically disjoint. So, you'd definitely expect that there were bugs found by the black hats and then used as zero-days and eventually leaked to the white hats. So, what you describe above is pretty much what one would expect. -Ekr

On Thu, 10 Jun 2004 11:54:31 PDT, Eric Rescorla said:

My hypothesis is that the sets of bugs independently found by white hats and black hats are basically disjoint. So, you'd definitely expect that there were bugs found by the black hats and then used as zero-days and eventually leaked to the white hats. So, what you describe above is pretty much what one would expect.

Well.. for THAT scenario to happen, two things have to be true: 1) Black hats are able to find bugs too 2) The white hats aren't as good at finding bugs as we might think, because some of their finds are leaked 0-days rather than their own work, inflating their numbers. Remember what you said:

relatively small. If we assume that the black hats aren't vastly more capable than the white hats, then it seems reasonable to believe that the probability of the black hats having found any particular vulnerability is also relatively small.

More likely, the software actually leaks like a sieve, and NEITHER group has even scratched the surface.. Remember - every single 0-day that surfaces was something the black hats found first. The only thing you're really measuring by looking at the 0-day rate is the speed at which an original black exploit gets leaked from a black hat to a very dark grey hat to a medium grey hat and so on, until it gets to somebody who's hat is close enough to white to publish openly. Data point: When did Steve Bellovin point out the issues with non-random TCP ISNs? When did Mitnick use an exploit for this against Shimomura? And now ask yourself - when did we *first* start seeing SYN flood attacks (which were *originally* used to shut the flooded machine up while and prevent it from talking while you spoofed its address to some OTHER machine?)

On Thu, 10 Jun 2004 12:23:42 PDT, Eric Rescorla said:

I'm not sure we disagree. All I was saying was that I don't think we have a good reason to believe that the average bug found independently by a white hat is already known to a black hat. Do you disagree?

Actually, yes. Non-obvious bugs (ones with a non 100% chance of being spotted on careful examination) will often be found by both groups. Let's say we have a bug that has a 0.5% chance of being found at any given attempt to find it. Now take 100 white hats and 100 black hats - compute the likelyhood that at least 1 attempt in either group finds it (I figure it as some 39% (1 - (0.995^100)). For bonus points, extend a bit further, and make multiple series of attempts, and compute the probability that for any given pair of 100 attempts, exactly one finds it, or neither finds it, or both find it. And it turns out that for that 39% chance, 16% of the time both groups will find it, 36% of the time exactly one will find it, and 48% of the time *neither* will find it. And in fact, the chance of overlap is much higher, because the two series of 100 runs *aren't* independent. Remember that for the most part, the info that suggested "Look over HERE" to the white hat was also available to the black hat.....

On Thu, 10 Jun 2004 13:30:41 PDT, Eric Rescorla said:

[0] Note that this doesn't require that the chance of finding any particular bug upon inspection of the code be very low high, but merely that there not be very deep coverage of any particular code section.

Right. However, if you hand the team of white hats and the team of black hats the same "Chatter has it there's a 0-day in Apache's mod_foo handler".... Note that the rumored 0-day doesn't even have to exist - one has to wonder how many of the bugs found in Windows by all color hats were inspired by Allchin's comment under oath that there was an API flaw in Windows so severe that publishing the API could endanger national security.....

On Thu, 10 Jun 2004 13:50:47 PDT, Eric Rescorla said:

I'm asking the question: If you find some bug in the normal course of your operations (i.e. nobody told you where to look) how likely is it that someone else has already found it?

And you're asking a question more like: Given that you hear about a bug before its release, how likely is it that some black hat alredy knows?

I think that the answer to the first question is probably "fairly low". I agree that the answer to the second question is probably "reasonably high".

Third case: Exploit in one package identified because of info from a similar exploit against some *other* package.... Back in March 2000, I spotted a rather nasty security bug in Sendmail (fixed in 8.10.1) when running under AIX or SunOS. Since the problem is a documented *feature* of the system linker, a *lot* of software had the problem - and the Sendmail release notes give enough info to make it "game over". At that point, the 3 big things left were (a) writing a general-case exploit (trivial if you use one of the another one of the basic design goals of the AIX linker against itself), (b) creating a shell one-liner to identify vulnerable programs, and (c) running the script from (b). Of the three, (c) was actually the most time-consuming. 3 years later, another package (OpenSSH) hit the same hole: http://www.securityfocus.com/archive/1/320149/2003-04-30/2003-05-06/0 And it was a known issue months before I tripped over it: http://mail.gnome.org/archives/gtk-devel-list/1999-November/msg00047.html I'd be most surprised if black hats did *not* have an exploit for the OpenSSH variant, having been pointed at the issue due to my finding a similar issue in Sendmail..... And there's *plenty* of evidence that when a novel attack is found, you see lots of people posting "So I was bored and decided to see what *else* had the same sort of bug..." (think "buffer overflow" ;)

More likely, the software actually leaks like a sieve, and NEITHER group has even scratched the surface..

How many leaks did the OpenBSD team find when they proactively audited their entire codebase for the first time a few years ago? This would be an indication of just how leaky an O/S might be expected to be.

Remember - every single 0-day that surfaces was something the black hats found first.

And 0-day exploits are only the ones that the blackhats are willing to talk about. If they keep quiet about an exploit and only use it for industrial espionage and other electronic crimes then we are unlikely to hear about it until a whitehat stumbles across the blackhat's activities. Rather like the cuckoo's egg or the recent complex exploit involving IE and the MS Help tool. Have any of your customers ever asked you for a traffic audit report showing every IP address that has ever sourced traffic to them or received traffic from them? --Michael Dillon