Declan, This is a very complex issue, and made the DDoS BoF last night even more lively. ;-) Read RFC2267. More people should be doing it, and most of these silly problems will go away. - paul At 08:31 PM 02/07/2000 -0500, Declan McCullagh wrote:

Yahoo told me on the phone that it's a malicious attack, and Global Center says the same thing. In Yahoo's words: "a coordinated distributed denial of service attack."

We've got a brief story up at: http://www.wired.com/news/business/0,1367,34178,00.html

The problem apparently originated with a router. But what kind of attack could have taken the network offline for that period of time and not affected other Global Center customers? I mean, there had to have been a gaping security hole somewhere: It looks like the routes got lost for (nearly) all of the Yahoo network, but no other non-Yahoo sites...

-Declan

I'd be one to argue that implementing egress filtering, as opposed to ingress filtering, would do more to stop DDoS attacks since one of the most crippling attacks uses forged valid source addresses to start the attack (smurf/fraggle). If you stop forged packets from leaving the offending networks (which you mention in your RFC, but only to say it's impractical to do both ingress and egress filtering and advocate ingress) and the need to track attacks goes no farther than the people in Company X's dialup pool who's causing the CPU on the router to go up. However, neither ingress or egress filtering helps stop any of the latest "seen in the wild" DDos attacks like trinoo, tribe, etc. because the floods are all unforged packets. Though they've been sketchy on details, it sounds like these or their decendants are the likey candidates for both Yahoo and Buy.com. Also, ingress filtering certainly doesn't help Tier3.net when their 4 inverse-muxed T1's are clogged with 20Mbps of traffic, forged or otherwise. Sure, the router is dropping the traffic like mad, but it's not going to help them unless their upstream will block the traffic as well once the attack starts. Egress filtering would stop the attack before it started if the traffic were forged. If it's just unforged traffic, you'd expect the attacking sites to notice the spike in bandwidth utilization and increased traffic flows from one or several machines to one destination, but that may be asking too much. Unfortunately, the rush to .COM riches has brought with it a lot of people who have only half a clue as to what they're doing if we, as the Internet community, are lucky, making the Internet landscape even more dangerous with the amount of ignorance that's out there when it comes to security issues. It should also be said that some established educational institutions seem to be having issues stopping attacks like smurf and fraggle as well. The media certainly isn't helping, classifying all DoS attacks as packet flooding attacks, which is not the case either, though all DDos attacks are (if you're a journalist, please feel free to ask what the difference is; I'll be more than happy to explain it). I wish I could have made NANOG and the DDoS BoF session, but I was unable to attend due to employment issues. -- Joseph W. Shaw - jshaw@insync.net Computer Security Consultant and Programmer Free UNIX advocate - "I hack, therefore I am." On Tue, 8 Feb 2000, Paul Ferguson wrote:

Declan,

This is a very complex issue, and made the DDoS BoF last night even more lively. ;-)

Read RFC2267. More people should be doing it, and most of these silly problems will go away.

From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Joe Shaw Sent: Tuesday, February 08, 2000 9:20 PM To: Paul Ferguson

I'd be one to argue that implementing egress filtering, as opposed to ingress filtering, would do more to stop DDoS attacks since one of the

X's dialup pool who's causing the CPU on the router to go up. However, neither ingress or egress filtering helps stop any of the latest "seen in the wild" DDos attacks like trinoo, tribe, etc. because the floods are all unforged packets. Though they've been sketchy on details, it sounds

You've nailed the heart of the problem right here and never noticed. It is significant that the packets were NOT forged. IOW, they were legitimate packets of sufficient number to cap those very large pipes. I recently performed the Platform Architect role in a large .COM deployment. As part of site evaluation I had a chance to visit the facility where eBay is hosted. In fact, that is the same facility that I wound up using. Lots of dark-fiber capacity and over 20 Gbps capacity at the facility and they support 10000baseSX back planes. I swear that I saw a few Cat 6509's in eBay's racks. This means 1 Gbps pipes, scalable in 1 Gbps increments, using gig-Ether link aggregation.

before it started if the traffic were forged. If it's just unforged traffic, you'd expect the attacking sites to notice the spike in bandwidth utilization and increased traffic flows from one or several machines to one destination, but that may be asking too much.

Gentlemen, this is a very large site, with plenty of spare capacity. It is significant that those pipes were capped, via excessive, non-forged, traffic. Although it speaks well for the infrastructure that delivered that traffic, it also scares the shit out of me. There are a very large number of very large systems, sitting behind some very large pipes, that are compromised. Think about that for a moment. These are not small machines deployed by college kids and internet newbies. No one trusts the operation of a $1.5M Sun e6500 by a group of rookies. They can probably afford to hire the best SA's that they can find and no one running equipment behind anything larger than a T1 can afford to hire the ignorant. Not at the prices charged for that size of a pipe. Just the same, those systems were compromised.

Unfortunately, the rush to .COM riches has brought with it a lot of people who have only half a clue as to what they're doing if we, as the Internet community, are lucky, making the Internet landscape even more dangerous with the amount of ignorance that's out there when it comes to security issues. It should also be said that some established educational institutions seem to be having issues stopping attacks like smurf and fraggle as well. The media certainly isn't helping, classifying all DoS attacks as packet flooding attacks, which is not the case either, though all DDos attacks are (if you're a journalist, please feel free to ask what the difference is; I'll be more than happy to explain it).

I smell denial here. The compromised systems (only 52?) had to have access to pipes at least 1 Gbps in size, in order to carry out this attack (do the math yourself). Either there were many more systems participating (in itself a scarey thought) or many of these large and professionally run systems are owned and their operators don't know it. The only other alternative is the conspiracy theory from hell. I suspect that this is not a kiddie-cracker activity. It is too well planned and carried out with too much discipline, over too long a time. I suspect that whomever is doing this has been silently "owning" systems for the past 18 months. I suggest that everyone start looking for signs of mwsh and its cousins. Because, I further suspect that the perpretrators have NOT used all of their assets. There are still a good many systems that are compromised, and not taking part in the current fracas, we just haven't found them yet.

On Tue, 8 Feb 2000, Paul Ferguson wrote:

...

Declan,

This is a very complex issue, and made the DDoS BoF last night even more lively. ;-)

Read RFC2267. More people should be doing it, and most of these silly problems will go away.

On Wed, Feb 09, 2000 at 12:02:34AM -0800, Roeland M.J. Meyer wrote:

No one trusts the operation of a $1.5M Sun e6500 by a group of rookies. They can probably afford to hire the best SA's that they can find and no one running equipment behind anything larger than a T1 can afford to hire the ignorant. Not at the prices charged for that size of a pipe.

i beg to differ. i've seen a large number of .com's which have mega gear, and minor-league SA's. the problem is that alot of .com's seem to think that the vendor will be able to solve their serious problems, or that they can just hire some "certified" SA's to do the job. -- [ Jim Mercer jim@reptiles.org +1 416 506-0654 ] [ Reptilian Research -- Longer Life through Colder Blood ] [ Don't be fooled by cheap Finnish imitations; BSD is the One True Code. ]

On Wed, 9 Feb 2000 01:09:14 -0800 "Roeland M.J. Meyer" <rmeyer@mhsc.com> wrote:

You can differ all you want, but I've never seen a large .COM deployment without at least one senior SA swinging the clue-bat, backed up by a network planner and a senior architect. How do you think they write all that code and have it work? DEsign is much more difficult than maintenance and takes more skilled personel

Only if the design was a good one. If its a bad one, maintenance can be a nightmare. [and few designers want to do post-deployment maintenance, as its traditionally an operators job]. -- Neil J. McRae - Alive and Kicking neil@DOMINO.ORG

I smell denial here. The compromised systems (only 52?) had to have access to pipes at least 1 Gbps in size, in order to carry out this attack (do the math yourself). Either there were many more systems participating (in itself a scarey thought) or many of these large and professionally run systems are owned and their operators don't know it. The only other alternative is the conspiracy theory from hell.

I've seen instances where workstations of experienced people had been compromised for considerable periods of time without their knowledge. This, to me, is not suprising. My view of security is that its all about trust. Major public servers are watched quite closely simply as a result of the attention that has to be given to the applications they support. However, those same administrators generally don't watch smaller, auxillary systems (ie, a 3rd nameserver several thousand miles away that serves no other function.) Consider the responsibility of a corporate security dude and IT guys who is trying to watch over the network used by 3 or 4 thousand employees, most of whom have desktop computers and few of which know how to do more than email 3 meg excel files to 30 or 40 people all over the corporate network several times a day. If the network is not kept absolutely tight, everything is a risk. I always work from the maxim (and those I work with have heard this at least a hundred times before) that "the easiest way to break into one computer is to break into another computer that it trusts." (eg. personal workstations... how many times have you looked at your process table this week?) ---------------------------------------------------------------------- Wayne Bouchard [Immagine Your ] web@typo.org [Company Name Here] Network Engineer ----------------------------------------------------------------------