Re: Yahoo offline because of attack (was: Yahoo network outage)
Roeland writes:
George wrote:
Roeland wrote:
I smell denial here. The compromised systems (only 52?) had to have access to pipes at least 1 Gbps in size, in order to carry out this attack (do the math yourself). Either there were many more systems participating (in itself a scarey thought) or many of these large and professionally run systems are owned and their operators don't know it. The only other alternative is the conspiracy theory from hell.
No, they don't. Assume there's 40k of data in the homepage. How many bytes of SYN-SYNACK-ACK-GET / HTTP/1.0\n does it take to do a TCP connect and request? I just tested, I show 160 bytes. That's a 250:1 leverage for the attacker. To fill 1 GBPS worth of outbound trunking you only need to generate 4 MBPS (32 Mbps) worth of input. 50ish systems with T-1 connectivity gets there with margins.
Okay, but you've still missed the point. Even if I stipulate everything you said here, that's still 50 largish systems that are compromised. I would almost wager that the perpetrators didn't use all of their assets either. That's a shit-load of large compromised systems on the Internet. Doesn't that thought worry you in the slightest?
50 systems across the internet with enough CPU capacity to near-fill a T-1 on a sustained basis with identical HTTP requests. Which is to say any modern multi-hundred-mhz RISC or x86 box with a reasonable OS, not really "largish". The processing needed in the OS TCP and IP stacks on the attacking system is most of the effort, and we're only talking in rough numbers 1,000 connects/sec for the attacker. Do I believe that there exist 50 or more T-1 connected hosts with that capability level or higher which still have vendor default setups and thus are vulnerable to this sort of attack, penetration, and then use as a distributed DOS attack participant? Yes, without a doubt. 50 simultaneous sites compromised by one attacker would be on the ambitious side these days, but some of the remote exploit scripts (and corresponding known holes in vendor supplied system configs) are pretty damn easy to use and it wouldn't be out of the relm of the practical for someone to do it if they worked hard, or got a small cooperating team to work on it. Of course the significance of this is highly worrysome. But the numbers have been in this rough performance range for attacker capabilities for several years now. That the tools used by attackers took that long to catch up is actually somewhat suprising to me, I was expecting this sort of thing some time ago. -george william herbert gherbert@crl.com
On Wed, 9 Feb 2000, George Herbert wrote:
50 systems across the internet with enough CPU capacity to near-fill a T-1 on a sustained basis with identical HTTP requests. Which is to say any modern multi-hundred-mhz RISC or x86 box with a reasonable OS, not really "largish". The processing needed in the OS TCP and IP stacks on the attacking system is most of the effort, and we're only talking in rough numbers 1,000 connects/sec for the attacker.
Now I haven't seen these DDoS "tools", but if you want to imagine something realy scary, imagine one exists that works like this: -attacker scans for the known OS vulns that will cough up a "#" prompt -attacker installs root kit with DDoS tool -that tool runs as a daemon that has the following features: -remote 'admin' via icmp (payload of echo-request includes password, host to attack, duration of attack -daemon launches the http "GET" flood as described earlier based on the info contained in that icmp echo-request -daemon continues this attack as prescribed with no further intervention So the attacker need only send a few packets to each compromised host to cause extreme amounts of damage. How would you track down the attacker? Sure, you could slowly find the compromised hosts and block them. You could even then look for where the icmp "control" message that starts the thing comes from, but if it's a one-way control channel, the source the attacker sends the control packet from could easily be forged and you could easily miss the one magic 'ping' that starts the thing off... The idea of such a tool is scary, and from what I've read about TFN and friends, it seems that they could be modified to work as outlined above. The worst thing about any effective DoS is, in my mind, the lack of an identifiable "attacker". Charles =-----------------= = | Charles Sprickman Internet Channel | | INCH System Administration Team (212)243-5200 | | spork@inch.com access@inch.com | = =----------------=
-george william herbert gherbert@crl.com
On Wed, Feb 09, 2000 at 10:58:00AM -0500, Charles Sprickman wrote:
So the attacker need only send a few packets to each compromised host to cause extreme amounts of damage.
How would you track down the attacker? Sure, you could slowly find the compromised hosts and block them. You could even then look for where the icmp "control" message that starts the thing comes from, but if it's a one-way control channel, the source the attacker sends the control packet from could easily be forged and you could easily miss the one magic 'ping' that starts the thing off...
The idea of such a tool is scary, and from what I've read about TFN and friends, it seems that they could be modified to work as outlined above. The worst thing about any effective DoS is, in my mind, the lack of an identifiable "attacker".
They do work as above, with encrypted control messages. If you look at some of the code (and then manage to stop laughing) you will find some interesting ways to counteract, trace to the control nodes, and in some cases even immediately kill the daemon on every attacking node. Keep in mind that the people writing these things are doing it with often very little clue, experience, or thought. Most are blindly stabbing at things they do not understand trying to tweak things and test them out to see if it makes their victim "die any faster", ripping mismatched code from various places (like blowfish code from eggdrop), and creating what will quite possibly be one of the quickest ways to spend a long long long LONG time in jail when they get caught and lawyers and accountants start adding up the "cost" of their distributed fun and games... -- Richard A. Steenbergen <ras@above.net> http://users.quadrunner.com/humble PGP Key ID: 0x60AB0AD1 (E5 35 10 1D DE 7D 8C A7 09 1C 80 8B AF B9 77 BB) MFN / AboveNet Communications Inc - ISX Network Engineer, Vienna VA
Now I haven't seen these DDoS "tools", but if you want to imagine something realy scary, imagine one exists that works like this:
-attacker scans for the known OS vulns that will cough up a "#" prompt -attacker installs root kit with DDoS tool -that tool runs as a daemon that has the following features: -remote 'admin' via icmp (payload of echo-request includes password, host to attack, duration of attack -daemon launches the http "GET" flood as described earlier based on the info contained in that icmp echo-request -daemon continues this attack as prescribed with no further intervention
So the attacker need only send a few packets to each compromised host to cause extreme amounts of damage.
How would you track down the attacker?
You've just described stacheldraht (http://staff.washington.edu/dittrich/misc/stacheldraht.analysis). It wasn't built with forged IP sources on the ICMP "trigger messages", but we did just catch such an attack here recently, and the presumed ICMP trigger message had a forged source IP address. The bitch about it is that the DoS floods used forged source addresses, but only among the last octet of the source IP address; the first three octets are "valid" for the machine that's doing the flooding. This means that flood packets will get out even with ingress filtering down to the subnet level, one has to catch the attack in progress and stick in an access-list with log-input to snag the hardware address of the attack packets in order to be able track down the actual machine involved. It's very, very nasty. Often we can go back through our netflow logs and find the original breakin to the machine once we know its IP address, which of course leads us back to a valid IP address and possibly the perpetrator. /cvk
I'll once again suggest adopting an extended router-to-router record route option which holds more info, more addresses than the current IP spec calls for that is, and is only used between routers trunc'd off on exit (well, configurable, of course.) Obviously then one has to get it into router software and turned on but that's nothing new as a problem whenever new technology is being adopted. One useful feature is that it'd probably be difficult in most environments for the villain to know which sites support this tracing and which don't, since they probably can't see traffic on the router, and the info is removed when it leaves (e.g. hits their PPP session.) Then if there's an attack one would only have to get the extended RR info from the router or routers the attack is coming through on your side to trace it back to a source router, and with luck could do something with that info. Even partial info, such as when it goes back through to a router which doesn't support this, should often be of some use. -- -Barry Shein Software Tool & Die | bzs@world.std.com | http://www.world.com Purveyors to the Trade | Voice: 617-739-0202 | Login: 617-739-WRLD The World | Public Access Internet | Since 1989 *oo*
At 03:11 AM 2/9/2000 -0800, you wrote:
50 systems across the internet with enough CPU capacity to near-fill a T-1 on a sustained basis with identical HTTP requests. Which is to say any modern multi-hundred-mhz RISC or x86 box with a reasonable OS, not really "largish".
Multi-hundred-mhz, nothing; a 486/33 can do that. 50 cast-off 486 motherboards with $50 AMD 5x86 processors could saturate those T1s and still get good GUI response. 50 Pentium IIs could do that, running even Windows 95, and probably have enough CPU left to get good RC5 cracking rates. :-) I think we're leaping to majorly unwarranted conclusions here.
From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Shawn McMahon Sent: Wednesday, February 09, 2000 8:01 AM
At 03:11 AM 2/9/2000 -0800, you wrote:
50 systems across the internet with enough CPU capacity to near-fill a T-1 on a sustained basis with identical HTTP requests. Which is to say any modern multi-hundred-mhz RISC or x86 box with a reasonable OS, not really "largish".
Multi-hundred-mhz, nothing; a 486/33 can do that.
50 cast-off 486 motherboards with $50 AMD 5x86 processors could saturate those T1s and still get good GUI response.
50 Pentium IIs could do that, running even Windows 95, and probably have enough CPU left to get good RC5 cracking rates. :-)
I think we're leaping to majorly unwarranted conclusions here.
A simple case of denial here, T1's are not cheap. It isn't the CPU horsepower that is significant here. It is the access to the required bandwidth that makes this so worrisome. In order to operate stealth-mode in a system, one must be on a box that has sufficient power such that the operation of your code consumes less than 3% of the box's available capacity. In addition, your network should consume less than 5% of the site's pipe, even during an attack. Remember, it appears that these hosts have been compromised for some time. Further, Sean indicates that the entire attack system was tested at least once and no one noticed. These guys have to be frugal with the assets if they want to contnue using them undetected. This indicates planning and discipline. These are NOT ignorant cracker-kiddies. This indicates one or two compromised hosts per site with 50-ish sites penetrated, at minimum (probably, 100's). I would wager that even the 50-ish sites actually used in the attacks had no idea that they were participating. This indicates low resource usage on part of the attacking code, since the first indicator SA's usually look for is abnormally high usage of resources. Let's quit assuming that all other operators are incompetent and start assuming the worst, that crackers got this one by "competent" SAs, shall we? If this is the case, then any of us are vulnerable. I find it difficult to believe that there are 50 sites, with T3 connectivity or better, that are all staffed exclusively by incompetent operators, let alone 100's or 1000's.
If we assume that the attacks are being lead by competent attackers, we must also assume that their motive could be more complex than just "hah hah, let's see if we can make Yahoo disappear." In fact, it could be far more interesting than just a technical display of capabilities. In light of Yahoo, Exodus and UUNET's issues over the last three days, anyone who doesn't consider this a mandate to improve the accountability of net-connected sites is seriously missing the boat. Just my opinion, Deepak Jain AiNET On Wed, 9 Feb 2000, Roeland M.J. Meyer wrote:
From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Shawn McMahon Sent: Wednesday, February 09, 2000 8:01 AM
At 03:11 AM 2/9/2000 -0800, you wrote:
50 systems across the internet with enough CPU capacity to near-fill a T-1 on a sustained basis with identical HTTP requests. Which is to say any modern multi-hundred-mhz RISC or x86 box with a reasonable OS, not really "largish".
Multi-hundred-mhz, nothing; a 486/33 can do that.
50 cast-off 486 motherboards with $50 AMD 5x86 processors could saturate those T1s and still get good GUI response.
50 Pentium IIs could do that, running even Windows 95, and probably have enough CPU left to get good RC5 cracking rates. :-)
I think we're leaping to majorly unwarranted conclusions here.
A simple case of denial here, T1's are not cheap. It isn't the CPU horsepower that is significant here. It is the access to the required bandwidth that makes this so worrisome.
In order to operate stealth-mode in a system, one must be on a box that has sufficient power such that the operation of your code consumes less than 3% of the box's available capacity. In addition, your network should consume less than 5% of the site's pipe, even during an attack. Remember, it appears that these hosts have been compromised for some time. Further, Sean indicates that the entire attack system was tested at least once and no one noticed. These guys have to be frugal with the assets if they want to contnue using them undetected. This indicates planning and discipline. These are NOT ignorant cracker-kiddies.
This indicates one or two compromised hosts per site with 50-ish sites penetrated, at minimum (probably, 100's). I would wager that even the 50-ish sites actually used in the attacks had no idea that they were participating. This indicates low resource usage on part of the attacking code, since the first indicator SA's usually look for is abnormally high usage of resources.
Let's quit assuming that all other operators are incompetent and start assuming the worst, that crackers got this one by "competent" SAs, shall we? If this is the case, then any of us are vulnerable. I find it difficult to believe that there are 50 sites, with T3 connectivity or better, that are all staffed exclusively by incompetent operators, let alone 100's or 1000's.
You mean, like the guy that threatened to publish 50,000 credit card numbers, with x-dates, if he wasn't paid off?
-----Original Message----- From: Deepak Jain [mailto:deepak@ai.net] Sent: Wednesday, February 09, 2000 9:34 AM To: Roeland M.J. Meyer Cc: Shawn McMahon; nanog@merit.edu Subject: RE: Yahoo offline because of attack (was: Yahoo network outage)
If we assume that the attacks are being lead by competent attackers, we must also assume that their motive could be more complex than just "hah hah, let's see if we can make Yahoo disappear." In fact, it could be far more interesting than just a technical display of capabilities.
In light of Yahoo, Exodus and UUNET's issues over the last three days, anyone who doesn't consider this a mandate to improve the accountability of net-connected sites is seriously missing the boat.
Just my opinion,
Deepak Jain AiNET
On Wed, 9 Feb 2000, Roeland M.J. Meyer wrote:
From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Shawn McMahon Sent: Wednesday, February 09, 2000 8:01 AM
At 03:11 AM 2/9/2000 -0800, you wrote:
50 systems across the internet with enough CPU capacity to
T-1 on a sustained basis with identical HTTP requests. Which is to say any modern multi-hundred-mhz RISC or x86 box with a reasonable OS, not really "largish".
Multi-hundred-mhz, nothing; a 486/33 can do that.
50 cast-off 486 motherboards with $50 AMD 5x86 processors could saturate those T1s and still get good GUI response.
50 Pentium IIs could do that, running even Windows 95, and
enough CPU left to get good RC5 cracking rates. :-)
I think we're leaping to majorly unwarranted conclusions here.
A simple case of denial here, T1's are not cheap. It isn't the CPU horsepower that is significant here. It is the access to the required bandwidth that makes this so worrisome.
In order to operate stealth-mode in a system, one must be on a box that has sufficient power such that the operation of your code consumes less than 3% of the box's available capacity. In addition, your network should consume less than 5% of the site's pipe, even during an attack. Remember, it appears that these hosts have been compromised for some time. Further, Sean indicates that the entire attack system was tested at least once and no one noticed. These guys have to be frugal with the assets if they want to contnue using them undetected. This indicates planning and discipline. These are NOT ignorant cracker-kiddies.
This indicates one or two compromised hosts per site with 50-ish sites penetrated, at minimum (probably, 100's). I would wager that even the 50-ish sites actually used in the attacks had no idea that they were
near-fill a probably have participating.
This indicates low resource usage on part of the attacking code, since the first indicator SA's usually look for is abnormally high usage of resources.
Let's quit assuming that all other operators are incompetent and start assuming the worst, that crackers got this one by "competent" SAs, shall we? If this is the case, then any of us are vulnerable. I find it difficult to believe that there are 50 sites, with T3 connectivity or better, that are all staffed exclusively by incompetent operators, let alone 100's or 1000's.
Extortion is a very sloppy motivation. How about something like "Our website stays up, our competitor doesn't." And the investors make out (either by shorting one, or going long on the other)... No threats, just marketing. My cup of tea may have been sour this morning. If am offending anyone's sensibilities, please disregard me. Deepak Jain AiNET On Wed, 9 Feb 2000, Roeland M.J. Meyer wrote:
You mean, like the guy that threatened to publish 50,000 credit card numbers, with x-dates, if he wasn't paid off?
-----Original Message----- From: Deepak Jain [mailto:deepak@ai.net] Sent: Wednesday, February 09, 2000 9:34 AM To: Roeland M.J. Meyer Cc: Shawn McMahon; nanog@merit.edu Subject: RE: Yahoo offline because of attack (was: Yahoo network outage)
If we assume that the attacks are being lead by competent attackers, we must also assume that their motive could be more complex than just "hah hah, let's see if we can make Yahoo disappear." In fact, it could be far more interesting than just a technical display of capabilities.
In light of Yahoo, Exodus and UUNET's issues over the last three days, anyone who doesn't consider this a mandate to improve the accountability of net-connected sites is seriously missing the boat.
Just my opinion,
Deepak Jain AiNET
On Wed, 9 Feb 2000, Roeland M.J. Meyer wrote:
From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Shawn McMahon Sent: Wednesday, February 09, 2000 8:01 AM
At 03:11 AM 2/9/2000 -0800, you wrote:
50 systems across the internet with enough CPU capacity to
T-1 on a sustained basis with identical HTTP requests. Which is to say any modern multi-hundred-mhz RISC or x86 box with a reasonable OS, not really "largish".
Multi-hundred-mhz, nothing; a 486/33 can do that.
50 cast-off 486 motherboards with $50 AMD 5x86 processors could saturate those T1s and still get good GUI response.
50 Pentium IIs could do that, running even Windows 95, and
enough CPU left to get good RC5 cracking rates. :-)
I think we're leaping to majorly unwarranted conclusions here.
A simple case of denial here, T1's are not cheap. It isn't the CPU horsepower that is significant here. It is the access to the required bandwidth that makes this so worrisome.
In order to operate stealth-mode in a system, one must be on a box that has sufficient power such that the operation of your code consumes less than 3% of the box's available capacity. In addition, your network should consume less than 5% of the site's pipe, even during an attack. Remember, it appears that these hosts have been compromised for some time. Further, Sean indicates that the entire attack system was tested at least once and no one noticed. These guys have to be frugal with the assets if they want to contnue using them undetected. This indicates planning and discipline. These are NOT ignorant cracker-kiddies.
This indicates one or two compromised hosts per site with 50-ish sites penetrated, at minimum (probably, 100's). I would wager that even the 50-ish sites actually used in the attacks had no idea that they were
near-fill a probably have participating.
This indicates low resource usage on part of the attacking code, since the first indicator SA's usually look for is abnormally high usage of resources.
Let's quit assuming that all other operators are incompetent and start assuming the worst, that crackers got this one by "competent" SAs, shall we? If this is the case, then any of us are vulnerable. I find it difficult to believe that there are 50 sites, with T3 connectivity or better, that are all staffed exclusively by incompetent operators, let alone 100's or 1000's.
On Wed, 9 Feb 2000, Deepak Jain wrote:
If we assume that the attacks are being lead by competent attackers, we must also assume that their motive could be more complex than just "hah hah, let's see if we can make Yahoo disappear." In fact, it could be far more interesting than just a technical display of capabilities.
It would make one hell of an excuse for those wishing more government(s) involvement/control.... /\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ Patrick Greenwell Earth is a single point of failure. \/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/
On Wed, Feb 09, 2000 at 09:25:43AM -0800, Roeland M.J. Meyer wrote:
A simple case of denial here, T1's are not cheap. It isn't the CPU horsepower that is significant here. It is the access to the required bandwidth that makes this so worrisome.
In order to operate stealth-mode in a system, one must be on a box that has sufficient power such that the operation of your code consumes less than 3% of the box's available capacity. In addition, your network should consume less than 5% of the site's pipe, even during an attack. Remember, it appears that these hosts have been compromised for some time. Further, Sean indicates that the entire attack system was tested at least once and no one noticed. These guys have to be frugal with the assets if they want to contnue using them undetected. This indicates planning and discipline. These are NOT ignorant cracker-kiddies.
This indicates one or two compromised hosts per site with 50-ish sites penetrated, at minimum (probably, 100's). I would wager that even the 50-ish sites actually used in the attacks had no idea that they were participating. This indicates low resource usage on part of the attacking code, since the first indicator SA's usually look for is abnormally high usage of resources.
Let's quit assuming that all other operators are incompetent and start assuming the worst, that crackers got this one by "competent" SAs, shall we? If this is the case, then any of us are vulnerable. I find it difficult to believe that there are 50 sites, with T3 connectivity or better, that are all staffed exclusively by incompetent operators, let alone 100's or 1000's.
You are quite confused. T1's are cheap, OC12s are not cheap. CPU is the limiting factor in many of these attacks, but not because of the ability to saturate a T1 with HTTP GETs or any other such nonsense. These attacks often taken down the attacking-victim as much as the attacked-victim, infact often times they run their attacks so strongly that they are unable to access the systems to stop them, which is why all the distributed attack programs have a built in length of time for the attack to run, any signal to "stop" would often never be received. The belief that previously seen problem were some kind of "test" is totally ubsubstantiated guesswork, of little quality. Your numbers are totally random with no basis in reality. You are correct that most sites do not realize they are participating even after a huge attack that cripples BOTH networks. It has not so much to do with "competency" as attention to detail and careful network monitoring, though you could easily make the arguement that operators who do not do such are incompetent. If you find this difficult to imagine you need a better imagination. -- Richard A. Steenbergen <ras@above.net> http://users.quadrunner.com/humble PGP Key ID: 0x60AB0AD1 (E5 35 10 1D DE 7D 8C A7 09 1C 80 8B AF B9 77 BB) MFN / AboveNet Communications Inc - ISX Network Engineer, Vienna VA
Richard Steenbergen wrote:
On Wed, Feb 09, 2000 at 09:25:43AM -0800, Roeland M.J. Meyer wrote:
A simple case of denial here, T1's are not cheap. It isn't the CPU horsepower that is significant here. It is the access to the required bandwidth that makes this so worrisome.
In order to operate stealth-mode in a system, one must be on a box that has sufficient power such that the operation of your code consumes less than 3% of the box's available capacity. In addition, your network should consume less than 5% of the site's pipe, even during an attack. Remember, it appears that these hosts have been compromised for some time. Further, Sean indicates that the entire attack system was tested at least once and no one noticed. These guys have to be frugal with the assets if they want to contnue using them undetected. This indicates planning and discipline. These are NOT ignorant cracker-kiddies.
This indicates one or two compromised hosts per site with 50-ish sites penetrated, at minimum (probably, 100's). I would wager that even the 50-ish sites actually used in the attacks had no idea that they were participating. This indicates low resource usage on part of the attacking code, since the first indicator SA's usually look for is abnormally high usage of resources.
Let's quit assuming that all other operators are incompetent and start assuming the worst, that crackers got this one by "competent" SAs, shall we? If this is the case, then any of us are vulnerable. I find it difficult to believe that there are 50 sites, with T3 connectivity or better, that are all staffed exclusively by incompetent operators, let alone 100's or 1000's.
You are quite confused.
T1's are cheap, OC12s are not cheap.
CPU is the limiting factor in many of these attacks, but not because of the ability to saturate a T1 with HTTP GETs or any other such nonsense.
These attacks often taken down the attacking-victim as much as the attacked-victim, infact often times they run their attacks so strongly that they are unable to access the systems to stop them, which is why all the distributed attack programs have a built in length of time for the attack to run, any signal to "stop" would often never be received.
The belief that previously seen problem were some kind of "test" is totally ubsubstantiated guesswork, of little quality.
Your numbers are totally random with no basis in reality.
You are correct that most sites do not realize they are participating even after a huge attack that cripples BOTH networks.
It has not so much to do with "competency" as attention to detail and careful network monitoring, though you could easily make the arguement that operators who do not do such are incompetent. If you find this difficult to imagine you need a better imagination.
One hard, solid data point: I was talking to a friend who is a part-time SA on a box colocated at his place of business (behind a 2xT1) which he found out was participating in the attack. He found this out when the links suddenly spiked through the roof and his ethernet switch lit up with a nice, solid traffic light. The only reason he spotted it? He was at work at the time. Had it occured at night, it's quite probably that nobody would have noticed, given how rarely they check the traffic stats (since it doesn't really matter to them until the traffic is pushing their ability to carry it). The box? RedHat 6.0 without the security patches; from logs, it appears to have been taken by an automated attack, via the old NFS bug. Nothing at all suprising there, of course. This sort of thing is not exactly rare. Compromised boxes at .edu sites have been a thorn in many operator's sides for a long time now, and other sites happen as well; the difference is that the attackers are now biding their time (which may not be all that long) before launching an attack, so that they have enough points to fire it off from. While this hardly rules out a more "professional" attack, it's quite possible for this sort of thing to be accomplished by a bored or angry kid with nothing better to do. Or more likely, a group of half a dozen of them doing it for kicks, scanning for hosts for an hour while doing homework, all week, until they have a sizeable list. If you think that's bad, wait until they find a way to compromise Windows hosts on DSL lines. That... will be deep pain. (BTW, the box in question was taken off-net, and is currently being given the scorched-earth treatment; the person in question will be undergoing education in security principles from a veteran operator, and realizes that the compromise was made possible by their own negligence... now.) -- *************************************************************************** Joel Baker System Administrator - lightbearer.com lucifer@lightbearer.com http://www.lightbearer.com/~lucifer KF6WAY (Tech) - 146.475 MHz (FM/Phone)
On Wed, 9 Feb 2000 lucifer@lightbearer.com wrote:
If you think that's bad, wait until they find a way to compromise Windows hosts on DSL lines. That... will be deep pain.
Its already happening. It's called backorifice and netbus (and a legion of other trojans) -Dan
here is a little trojan list http://nethog.com/feeds/niteryder/trojans.htm Dan Hollis wrote:
On Wed, 9 Feb 2000 lucifer@lightbearer.com wrote:
If you think that's bad, wait until they find a way to compromise Windows hosts on DSL lines. That... will be deep pain.
Its already happening. It's called backorifice and netbus (and a legion of other trojans)
-Dan
-- Thank you; |--------------------------------------------| | Thinking is a learned process so is UNIX | |--------------------------------------------| Henry R. Linneweh
Lots of NSPs and ISPs are tracking customer utilization of links, either by MRTG or RRD ... .and many of them bill by utilization using these or other SNMP-based tools. It should be trivial, during a DDoS attack of the scale that took down Yahoo, to find participating sites. A jump from normal utilization to 100% link utilization should be easily noticible if it lasts more than 15 minutes (3 polling intervals, if you are doing it at 5 minutes). It seems to me that a customer would be more than willing to have a rate-limit or filter installed on their routers during this kind of event, especially if it helps them track down the compromised machine. Host-by-host prevention, during an attack, should be very easy ... assuming a minimal amount of cooperation between upstream provider and compromised network, if link utilization is tracked and the spike is noticible. Perhaps we should be notifying operations staff to be on the lookout for suddenly saturated circuits, and to be prepared to help out owners of compromised hosts with filter configuration? Just a thought. -------------------------------------------- Travis Pugh Sr. Network Engineer tpugh@shore.net Shore.net -------------------------------------------- On Wed, 9 Feb 2000 lucifer@lightbearer.com wrote:
One hard, solid data point:
I was talking to a friend who is a part-time SA on a box colocated at his place of business (behind a 2xT1) which he found out was participating in the attack.
He found this out when the links suddenly spiked through the roof and his ethernet switch lit up with a nice, solid traffic light. The only reason he spotted it? He was at work at the time. Had it occured at night, it's quite probably that nobody would have noticed, given how rarely they check the traffic stats (since it doesn't really matter to them until the traffic is pushing their ability to carry it).
*************************************************************************** Joel Baker System Administrator - lightbearer.com lucifer@lightbearer.com http://www.lightbearer.com/~lucifer KF6WAY (Tech) - 146.475 MHz (FM/Phone)
I don't know how much cooperation is expected between Tier 1s and their dedicated line customers ... last time I was working on an OC48 backbone we had no problem helping customers who were being used as bounce sites for smurfs. Cooperation between backbones may not happen, but it is really an NSP's job (if customer service is anywhere in their philosophy) to help their customers out ... if the attack is causing disruption on the compromised network, any NOC worth it's salt should be able to aid their customer with ACLs and technical assistance. -travis On Wed, 9 Feb 2000, Dan Hollis wrote:
On Wed, 9 Feb 2000, Travis Pugh wrote:
... assuming a minimal amount of cooperation between upstream provider and
Theres that "c" word again. It just isnt going to happen.
-Dan
I firmly believe that the security groups will be working closer together after this week than ever before. I spent some time talking to the C&W security group today about a problem related to the stream.c exploit being used. Is anyone dropping traffic from src/dst ips that are currently reserved and do not have any allocations out of them (such as 60/8 for example) anywhere inside their network? I know that MAPS has gone and done something similar to this as it relates to their rbl, but I am not aware of any providers doing anything but route filtering on these prefixes, not packet filtering. - jared On Wed, Feb 09, 2000 at 01:09:29PM -0800, Dan Hollis wrote:
On Wed, 9 Feb 2000, Travis Pugh wrote:
... assuming a minimal amount of cooperation between upstream provider and
Theres that "c" word again. It just isnt going to happen.
-Dan
-- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine. END OF LINE |
On Wed, Feb 09, 2000 at 03:51:45PM -0500, Travis Pugh wrote:
Host-by-host prevention, during an attack, should be very easy ... assuming a minimal amount of cooperation between upstream provider and compromised network, if link utilization is tracked and the spike is noticible. Perhaps we should be notifying operations staff to be on the lookout for suddenly saturated circuits, and to be prepared to help out owners of compromised hosts with filter configuration?
This sort of alarming is fairly trivial. Just about any network management system can be configured to poll interface counters on a regular basis and alarm when some threshold is reached. The difficult question to answer is "How long should the link be saturated before sending an alarm". With high speed links this is a lot easier. It's relatively easy to saturate a T1 with a file transfer, however the same would not be true for an OC-3c. This type of alarming should be based upon deviation from the established mean as well. (For example, if a circuit sees around 50mbit/sec worth of usage on a regular basis, and then spikes to 130mbit/sec and stays there, something is clearly wrong) /cbz
Anyone find it interesting that all the big name sites are getting hit except AOL? Makes you wonder.... Jim Williams Ntrnet Systems, Inc. President/CEO Research Triangle Park, NC jaw12@ntrnet.net (919)484-0504 fax(919)484-0782 On Thu, 10 Feb 2000, Christopher B. Zydel wrote:
On Wed, Feb 09, 2000 at 03:51:45PM -0500, Travis Pugh wrote:
Host-by-host prevention, during an attack, should be very easy ... assuming a minimal amount of cooperation between upstream provider and compromised network, if link utilization is tracked and the spike is noticible. Perhaps we should be notifying operations staff to be on the lookout for suddenly saturated circuits, and to be prepared to help out owners of compromised hosts with filter configuration?
This sort of alarming is fairly trivial. Just about any network management system can be configured to poll interface counters on a regular basis and alarm when some threshold is reached. The difficult question to answer is "How long should the link be saturated before sending an alarm". With high speed links this is a lot easier. It's relatively easy to saturate a T1 with a file transfer, however the same would not be true for an OC-3c. This type of alarming should be based upon deviation from the established mean as well. (For example, if a circuit sees around 50mbit/sec worth of usage on a regular basis, and then spikes to 130mbit/sec and stays there, something is clearly wrong)
/cbz
On Thu, 10 Feb 2000, Jim Williams wrote:
Anyone find it interesting that all the big name sites are getting hit except AOL? Makes you wonder....
I'd expect them to be a target, considering how high profile they are, but the week isn't out yet. We could end up with a few more high-profile sites getting the treatment. Sadly, since there's nothing that can really be done to prevent being the victim, it only means there will be more evidence after the fact. -- Joseph W. Shaw - jshaw@insync.net Computer Security Consultant and Programmer Free UNIX advocate - "I hack, therefore I am."
Anyone find it interesting that all the big name sites are getting hit except AOL? Makes you wonder....
...if they'll be next. ---------------------------------------------------------------------- Wayne Bouchard [Immagine Your ] web@typo.org [Company Name Here] Network Engineer ----------------------------------------------------------------------
On Wed, 9 Feb 2000 lucifer@lightbearer.com wrote:
If you think that's bad, wait until they find a way to compromise Windows hosts on DSL lines. That... will be deep pain.
It's very similar to your friends RedHat box with NFS holes. It's called File/Printer Sharing and any of the available trojan packages out there. You'd be amazed at how many unprotected Windows file shares are open with full access enabled. -- Joseph W. Shaw - jshaw@insync.net Computer Security Consultant and Programmer Free UNIX advocate - "I hack, therefore I am."
T1's are cheap, OC12s are not cheap.
That may be the case, but I think that Kim hit the nail on the head earlier. With the number of multi-megabit connected homes growing rapidly, there is a rapidly growing number of exploitable hosts for those perpetrating DDoS attacks to take advtange of. /cbz
Please remember that cable-modems are asymetric and the aggregate upstream pipe is shared.
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Christopher B. Zydel Sent: Wednesday, February 09, 2000 3:02 PM To: Richard Steenbergen Cc: nanog@merit.edu Subject: Re: Yahoo offline because of attack (was: Yahoo network outage)
T1's are cheap, OC12s are not cheap.
That may be the case, but I think that Kim hit the nail on the head earlier. With the number of multi-megabit connected homes growing rapidly, there is a rapidly growing number of exploitable hosts for those perpetrating DDoS attacks to take advtange of.
/cbz
T1's are cheap, OC12s are not cheap.
That may be the case, but I think that Kim hit the nail on the head earlier. With the number of multi-megabit connected homes growing rapidly, there is a rapidly growing number of exploitable hosts for those perpetrating DDoS attacks to take advtange of.
On Wed, Feb 09, 2000 at 05:37:49PM -0800, Roeland M.J. Meyer wrote:
Please remember that cable-modems are asymetric and the aggregate upstream pipe is shared.
Some MSOs choose to rate limit their user's upstreams as low as 128kbit/sec, others do not. For example, we limit our users to 1mbit/sec currently. As for the upstream communications channel, this is not much of a limitation. Typical DOCSIS configurations include multiple upstream ports tied to a single downstream. It is typical to combine a small number of optical receivers to a given upstream port (1 or 2). Each optical receiver typically carries 500 homes passed. Operating a 16 QAM carrier with a channel width of 3.2MHz yields ~10.24mbit/sec of bandwidth. Subtract a little for overhead, and figure you're doing pretty well and subscribe 10% of your passed homes, or roughly 100 users per upstream port. Your average user isn't pounding on the upstream too hard, so figure less than a quarter of these users really hit it hard, and they're not likely to all be doing it at the same time. I'd consider a few cable or DSL networks with handfuls of compromised hosts sitting on them a large threat given that it doesn't take a huge amount of bandwidth to create a rather damaging TCP flood. I realize that these users are not as threatening as a dorm network attached to a T3/OC-3c, but the CM/DSL population is growing a lot faster than the dorm population. /cbz
From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Richard Steenbergen Sent: Wednesday, February 09, 2000 9:42 AM
On Wed, Feb 09, 2000 at 09:25:43AM -0800, Roeland M.J. Meyer wrote:
A simple case of denial here, T1's are not cheap. It isn't the CPU horsepower that is significant here. It is the access to the required bandwidth that makes this so worrisome.
In order to operate stealth-mode in a system, one must be on a box that has sufficient power such that the operation of your code consumes less than 3% of the box's available capacity. In addition, your network should consume less than 5% of the site's pipe, even during an attack.
This indicates one or two compromised hosts per site with 50-ish sites penetrated, at minimum (probably, 100's). I would wager that even the 50-ish
Let's quit assuming that all other operators are incompetent and start assuming the worst, that crackers got this one by "competent" SAs, shall we?
You are quite confused.
T1's are cheap, OC12s are not cheap.
From the POV of a start-up, OC12's are outragously expensive, and the prime reason to co-lo.
These attacks often taken down the attacking-victim as much as the attacked-victim, infact often times they run their attacks so strongly that they are unable to access the systems to stop them, which is why all the distributed attack programs have a built in length of time for the attack to run, any signal to "stop" would often never be received.
Your numbers are totally random with no basis in reality.
About a year and a half ago (ancient times) I had a client where three of their names servers were penetrated by the MWSH program (Millennium Worm Shell). The first one exhibited just the behaviour you describe here, the second one operated at the 3% level that I indicated, and the third one stayed dormant until I provoked it. The client was all set to believe that only the one name server was compromised. Whereas all three systems were completely "owned" by MWSH. We wound up scrubbing all the DASD down to bare magnetic particles (format with 0xe5 in all sectors) and rebuilding all three systems from known good sources. We also upgraded them to BIND8 and placed specific blocks in "/tmp/..." and "/...." (fs level 0000).
You are correct that most sites do not realize they are participating even after a huge attack that cripples BOTH networks.
How could this be? If a host goes into overload and the network is congested, one would think that there is something screwy going on ...
Nonsense. Fire it up on all the Windows 95 workstations at a few public libraries around the country, throw in a hacked screensaver running on lots of RoadRunner PCs, and you've got more bandwidth than you can shake a stick at. The whole thing could be fired off by one trip to a public library, or from a high school. You cannot draw any conclusions about this attack from the amount of bandwidth used. You will have to track down source sites, track down who hacked them, and follow the chain. Either that, or get the big providers to tcpdump their user's IRC traffic and grep for keywords; somebody will shoot his mouth off about this. This could be anybody of any level of ability, but I'm telling you that this is not in any way beyond the ability of script kiddies. At 09:25 AM 2/9/2000 -0800, you wrote:
A simple case of denial here, T1's are not cheap. It isn't the CPU horsepower that is significant here. It is the access to the required bandwidth that makes this so worrisome.
In order to operate stealth-mode in a system, one must be on a box that has sufficient power such that the operation of your code consumes less than 3% of the box's available capacity. In addition, your network should consume less than 5% of the site's pipe, even during an attack. Remember, it appears that these hosts have been compromised for some time. Further, Sean indicates that the entire attack system was tested at least once and no one noticed. These guys have to be frugal with the assets if they want to contnue using them undetected. This indicates planning and discipline. These are NOT ignorant cracker-kiddies.
A simple case of denial here, T1's are not cheap. It isn't the CPU horsepower that is significant here. It is the access to the required bandwidth that makes this so worrisome.
No, but cable modems and DSL are. ---------------------------------------------------------------------- Wayne Bouchard [Immagine Your ] web@typo.org [Company Name Here] Network Engineer ----------------------------------------------------------------------
participants (18)
-
Barry Shein -
Charles Sprickman -
Charley Kline -
Christopher B. Zydel -
Dan Hollis -
Deepak Jain -
George Herbert -
Henry R. Linneweh -
Jared Mauch -
Jim Williams -
Joe Shaw -
lucifer＠lightbearer.com -
Patrick Greenwell -
Richard Steenbergen -
Roeland M.J. Meyer -
Shawn McMahon -
Travis Pugh -
Wayne Bouchard