Yahoo! Lessons Learned
As much as I enjoy finding out about Yahoo & GlobalCenter issues by reading the newswires, I wonder if there are any lessons we can learn from these events. Or was this not big enough to get attention of upper management? Was there something Yahoo!, GlobalCeneter or other providers could have done, either individually or in cooperation, to prevent the problem? Likewise, could they, individually or in cooperation with other providers, have shortened the duration or severity by doing something different? And finally, would they be more successfull in tracking the source the the problem by doing something different?
On 8 Feb 2000, Sean Donelan wrote:
Was there something Yahoo!, GlobalCeneter or other providers could have done, either individually or in cooperation, to prevent the problem?
I would say that they first have to all agree as to what the problem was...
Likewise, could they, individually or in cooperation with other providers, have shortened the duration or severity by doing something different?
And finally, would they be more successfull in tracking the source the the problem by doing something different?
Probably. However the discussion would quickly become one about the disparity in philosophies between academics and "traditional" corporate types. It's a social and cultural issue not a strictly operational one... /\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ Patrick Greenwell Earth is a single point of failure. \/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/
RFC2267. - paul At 03:25 AM 02/08/2000 -0800, Sean Donelan wrote:
As much as I enjoy finding out about Yahoo & GlobalCenter issues by reading the newswires, I wonder if there are any lessons we can learn from these events. Or was this not big enough to get attention of upper management?
Was there something Yahoo!, GlobalCeneter or other providers could have done, either individually or in cooperation, to prevent the problem?
Likewise, could they, individually or in cooperation with other providers, have shortened the duration or severity by doing something different?
And finally, would they be more successfull in tracking the source the the problem by doing something different?
From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Sean Donelan Sent: Tuesday, February 08, 2000 3:26 AM
As much as I enjoy finding out about Yahoo & GlobalCenter issues by reading the newswires, I wonder if there are any lessons we can learn from these events. Or was this not big enough to get attention of upper management?
I doubt if upper management could have done anything about it. AFAICT, Yahoo was not compromised directly. Reports that I have seen, and some of them are hearsay, indicated that this is one of the very first of a distributed DOS attack. One that CERT recent;ly warned us about. http://www.cert.org/advisories/CA-2000-01.html
Was there something Yahoo!, GlobalCeneter or other providers could have done, either individually or in cooperation, to prevent the problem?
Likewise, could they, individually or in cooperation with other providers, have shortened the duration or severity by doing something different?
highly unlikely.
And finally, would they be more successfull in tracking the source the the problem by doing something different?
The only defense I can think of is to prevent other systems from being "owned". This takes a tightly cooperative environment. We don't have this. At best, we have a loosely co-operative anarchy, with none of the big entities playing together consistently. Evenso, this may not be possible.
And finally, would they be more successfull in tracking the source the the problem by doing something different?
So thats another interesting question.. How do you go about doing a packet trace on routers passing giabits of traffic every second without killing the router/network and actually get usefull information out of it? ---------------------------------------------------------------------- Wayne Bouchard [Immagine Your ] web@typo.org [Company Name Here] Network Engineer ----------------------------------------------------------------------
And finally, would they be more successfull in tracking the source the the problem by doing something different?
So thats another interesting question.. How do you go about doing a packet trace on routers passing giabits of traffic every second without killing the router/network and actually get usefull information out of it?
passive monitoring. we don't have anything yet to run at oc-x speed (pos) but caida is working on several versions of passive monitors and at least one commercial vendor is working on one (ip capable). there was talk in the caida member meeting at nanog of doing some security bits in some of their software, and i don't remember for sure but i think someone mentioned security with respect to the passive monitors. if we installed passive monitors on IX links between providers, we might be able to do some interesting security traces. -brett
On Thu, Feb 10, 2000 at 12:17:28AM -0800, brett watson wrote:
And finally, would they be more successfull in tracking the source the the problem by doing something different?
So thats another interesting question.. How do you go about doing a packet trace on routers passing giabits of traffic every second without killing the router/network and actually get usefull information out of it?
passive monitoring. we don't have anything yet to run at oc-x speed (pos) but caida is working on several versions of passive monitors and at least one commercial vendor is working on one (ip capable).
With a Foundry BigIron you should be able to monitor high speed ports (they presently offer up to OC12 PoS as well as the usual ethernet) with no additional load on the device, just dump it directly to a gige port, get a PC with a $300 netgear gige nic, and you're off and running. Its not OC192 monitoring but if thats your customer aggregation device you're much better off, and Foundry is always up to something with bigger and better on the way... -- Richard A. Steenbergen <ras@above.net> http://users.quadrunner.com/humble PGP Key ID: 0x60AB0AD1 (E5 35 10 1D DE 7D 8C A7 09 1C 80 8B AF B9 77 BB) MFN / AboveNet Communications Inc - ISX Network Engineer, Vienna VA
At 11:11 PM 02/09/2000 -0700, Wayne Bouchard wrote:
So thats another interesting question.. How do you go about doing a packet trace on routers passing giabits of traffic every second without killing the router/network and actually get usefull information out of it?
Well, I can't _fully_ address that question, but see also: http://www.cisco.com/warp/public/707/22.html - paul
On Wed, 9 Feb 2000, Wayne Bouchard wrote:
And finally, would they be more successfull in tracking the source the the problem by doing something different?
So thats another interesting question.. How do you go about doing a packet trace on routers passing giabits of traffic every second without killing the router/network and actually get usefull information out of it?
---------------------------------------------------------------------- Wayne Bouchard [Immagine Your ] web@typo.org [Company Name Here] Network Engineer
----------------------------------------------------------------------
You bridge another device in line and have THAT device collect your data. Not as trivial for OCx connected routers but still possible. John Fraizer EnterZone, Inc
On Thu, 10 Feb 2000, NANOG Mailing List wrote:
WEB wrote: packet trace on routers passing giabits of traffic every second without killing the router/network and actually get usefull information out of it?
You bridge another device in line and have THAT device collect your data. Not as trivial for OCx connected routers but still possible.
John Fraizer
Any monetary considerations applied to this or not? OC-192c line cards cost money. The trivial answer is that DDoS attacks cost money as well, but there is a cost/benefit analysis to be done here. Would that money be better spent elsewhere? At OC-192c for typical streams and a large sized network, the data collection alone becomes a nearly insurmountable issue. Think 48 or more 192c's in a hub, think 100 hubs. Assuming you can throw out the non customer links, you're still around 2400 or so bridged OC-192c's, with data polling/netflow type stats. Not a pretty picture. Of course, given that we can get netflow type packet histories, plotting the src/dest pairs for a while and then if there is a _large_ change (some n std dev) from the norm for some particular dst (nominally the one under attack), and then raising an alarm on that router/pipe, would make it trivial to trace these type of attacks. With history storage, it would make it easier to trace back after the fact. The problem is, the amount of data storage. I think it was Dr. Li who said "you can move the bits or you can count the bits" /vijay
Sorry for the delayed message but my mailbox exploded and I'm just now catching up..
As much as I enjoy finding out about Yahoo & GlobalCenter issues by reading the newswires, I wonder if there are any lessons we can learn from these events. Or was this not big enough to get attention of upper management?
Was there something Yahoo!, GlobalCeneter or other providers could have done, either individually or in cooperation, to prevent the problem?
Likewise, could they, individually or in cooperation with other providers, have shortened the duration or severity by doing something different?
And finally, would they be more successfull in tracking the source the the problem by doing something different?
From what I understand, the traffic generated to the yahoo web servers was in the form of a SYN flood. I find it interesting that the DDOS mechanism used did this. If you try to solve the congestion problem by rate limiting, there may still be enough of the SYN packets getting through to take out the server. So it seems we had better get better at dealing with layered attacks.
---------------------------------------------------------------------- Wayne Bouchard [Immagine Your ] web@typo.org [Company Name Here] Network Engineer ----------------------------------------------------------------------
participants (9)
-
brett watson -
NANOG Mailing List -
Patrick Greenwell -
Paul Ferguson -
Richard Steenbergen -
Roeland M.J. Meyer -
Sean Donelan -
Vijay Gill -
Wayne Bouchard