Folks, We've been handling a multi-vector DDoS - 40-byte spoofed SYN-flooding towards www.cisco.com (198.133.219.25/32) as well as an HTTP-AUTH resource-exhaustion attack, and working these issues with our upstreams. Our apologies for any inconveniences, and our thanks to those who've assisted in tracing and blocking the spoofed traffic. We're continuing the work the issue, and would be grateful if operators would check for 40-byte spoofed TCP headed towards 198.133.219.25/32 and trace/block it as warranted. Your patience and understanding are greatly appreciated. Thanks! ------------------------------------------------------------- Roland Dobbins <rdobbins@cisco.com> // 408.527.6376 voice
On Mon, 6 Oct 2003 14:01:31 -0700, Roland Dobbins wrote
Folks,
We've been handling a multi-vector DDoS - 40-byte spoofed SYN-flooding towards www.cisco.com (198.133.219.25/32) as well as an HTTP-AUTH resource-exhaustion attack, and working these issues with our upstreams. Our apologies for any inconveniences, and our thanks to those who've assisted in tracing and blocking the spoofed traffic.
We're continuing the work the issue, and would be grateful if operators would check for 40-byte spoofed TCP headed towards 198.133.219.25/32 and trace/block it as warranted. Your patience and understanding are greatly appreciated.
Thanks!
------------------------------------------------------------- Roland Dobbins <rdobbins@cisco.com> // 408.527.6376 voice
My mailbox has filled quite a bit (to the tune of a dozen-plus mails) with comments along the lines of "don't quote me, NANOG is too important for my work, I don't want to get on Sue Harris' bad side" since my last so-called "off-topic" NANOG post (which all but *one* person, other than Sue Harris, found to be "within range and reason"). The spammers, the DDoS'ers, the proxy scanners and rapists, the SMTP auth crackers. the trojan spreaders, the DNSBL-DOS'ers, the hardcore computer criminals are the evil army of one? The following well-remembered lines come to mind here, and excuse me if you hear a slight hysterical laughter from my direction: "First They Came for the Jews First they came for the Jews and I did not speak out because I was not a Jew. Then they came for the Communists and I did not speak out because I was not a Communist. Then they came for the trade unionists and I did not speak out because I was not a trade unionist. Then they came for me and there was no one left to speak out for me." Pastor Martin Niemöller
-----BEGIN PGP SIGNED MESSAGE----- Hash: MD5 Hello Kai, Monday, October 6, 2003, 6:39:49 PM, you wrote: KS> The following well-remembered lines come to mind here, and excuse me if KS> you hear a slight hysterical laughter from my direction: I don't know what your post has to do with the original topic, but if you don't like the way NONOG is moderated, please feel free to start your own Network Operators mailing list. As far as comparing NANOG moderation to Nazi Germany that is disgusting and beneath contempt. allan - -- Allan Liska allan@allan.org http://www.allan.org http://www.hosthideout.com -----BEGIN PGP SIGNATURE----- Version: 2.6 iQCVAwUAP4H5OSkg6TAvIBeFAQH71gP/XLt+Z9O+VHTUJQTNIZpyOI8ijA+HYYI+ Gbji4Z6W9KJcuUKpv5fM6Ud5PbD79yOcGpl4fMndoQnCQsT42CnXAeg9v+mj49/e e0WXAStNfxl+DC/Arr7vLi4/SkxRqHjdSEPulDTeJeHEWykDm8On/nSUyXinRsrS gxIGYyOSEYE= =PMx8 -----END PGP SIGNATURE-----
On Mon, 6 Oct 2003, Allan Liska wrote:
KS> The following well-remembered lines come to mind here, and excuse me if KS> you hear a slight hysterical laughter from my direction:
I don't know what your post has to do with the original topic, but if you don't like the way NONOG is moderated, please feel free to start your own Network Operators mailing list.
I'm only guessing here, but I think what he may have meant was: First They Came for the IRC bots and I did not speak out because I did not run a bot. Then They Came for the IRC servers and I did not speak out because I did not run an IRC server. ...skip a few years... Then They Came for the DNSBLs and I did not speak out because I did not run a DNSBL. Now that they've come for cisco, maybe law enforcement, network operators, and router vendors will all get their $h!t together and do something to put a stop to these DDoS attacks that have been going on in various forms for several years. A handful of people (an assumption on my part) have the power / distributed bandwidth to bring just about any internet site/network to its knees using the distributed.net meets DoS tools they've created and distributed to thousands, perhaps millions of internet connected windows boxes. Anyone who doesn't think that's an operational issue, just wait until it bites you on the ass. ---------------------------------------------------------------------- Jon Lewis *jlewis@lewis.org*| I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
jlewis@lewis.org wrote:
On Mon, 6 Oct 2003, Allan Liska wrote:
KS> The following well-remembered lines come to mind here, and excuse me if KS> you hear a slight hysterical laughter from my direction:
I don't know what your post has to do with the original topic, but if you don't like the way NONOG is moderated, please feel free to start your own Network Operators mailing list.
I'm only guessing here, but I think what he may have meant was:
First They Came for the IRC bots and I did not speak out because I did not run a bot. Then They Came for the IRC servers and I did not speak out because I did not run an IRC server. ...skip a few years... Then They Came for the DNSBLs and I did not speak out because I did not run a DNSBL.
Now that they've come for cisco, maybe law enforcement, network operators, and router vendors will all get their $h!t together and do something to put a stop to these DDoS attacks that have been going on in various forms for several years.
A handful of people (an assumption on my part) have the power / distributed bandwidth to bring just about any internet site/network to its knees using the distributed.net meets DoS tools they've created and distributed to thousands, perhaps millions of internet connected windows boxes.
Anyone who doesn't think that's an operational issue, just wait until it bites you on the ass.
Now we have clear evidence that there are no less than three who understand the threat.
First They Came for the IRC bots and I did not speak out because I did not run a bot. Then They Came for the IRC servers and I did not speak out because I did not run an IRC server. ...skip a few years... Then They Came for the DNSBLs and I did not speak out because I did not run a DNSBL.
Anyone who doesn't think that's an operational issue, just wait until it bites you on the ass.
Let's add a very important line: "Then They Came for the OC-3 or smaller connections and I did not speak out because I run fat OC-12 - OC-48 pipes" It's my guess that the "top providers" that ignore cries for help because they can sink the traffic (and bill for it) without breaking a sweat will one day find themselves without a plan and without a clue when the Kiddiez come pounding down the door with something that can saturate their pipes and bring a major customer down. I hope we don't have to wait until that time comes around to figure out how to cooperate.
Now we have clear evidence that there are no less than three who understand the threat.
Heh. Why things like this don't scare everyone on this list sh*tless is beyond me. If anyone ever sees garbage coming out of 8059, hit the abuse@ address or peek at Jared's list for phone info. Charles
Charles,
Let's add a very important line:
"Then They Came for the OC-3 or smaller connections and I did not speak out because I run fat OC-12 - OC-48 pipes"
which doesn't help you much today. I've seen attacks of around a Gbit/s bandwidth. So a OC-48 is already in danger. The OC-12 is useless. And _of course_ the top providers have OC-192 "everywhere" ... .
It's my guess that the "top providers" that ignore cries for help because they can sink the traffic (and bill for it)
and get complains from customers because the Internet access doesn't work as promised. Ignoring this in a competitive market is no option. A least not for a longer time. What is underestimated is the difficulty to detect an attack and the details of it. Fortunately tools like Arbor or Riverhead exist meanwhile but even then it's often reactive for smaller customers. From my impression "top providers" spent the money for such tools although there is no direct/obvious revenue impact (read: gain). I would name this a responsible behavior for commercial companies.
I hope we don't have to wait until that time comes around to figure out how to cooperate.
There is cooperation. Maybe not that much on list like NANOG but Hank mentioned already a non-public list which succeeded in building the trust to cooperate with other providers. Without the risk to see your issues on news.com the next day. Just because it doesn't appear on NANOG doesn't mean nobody takes care :-) Regards, Marc -- Marc Binderberger <marc@sniff.de> Powered by *BSD ;-)
On Mon, 06 Oct 2003 18:45:15 -0500 "Laurence F. Sheldon, Jr." <larrysheldon@cox.net> wrote: | Now we have clear evidence that there are no less than three who | understand the threat. If you mean the threat from those who will attack and disable sites because they don't like what people at those sites say or do, then I assure you there are many who do understand that threat; some of whom can see little difference in terms of effect between DDoS attacks run by individuals, and the null-routing by a backbone network of IPs (or ranges of IPs) for which they make BGP announcements. Both are actions designed to interfere with individual freedoms; both are serious operational issues, and need to be discussed here. Or was it a different kind of threat that you were referring to, which might have discouraged some who understand the real threat from talking about it? -- Richard Cox
Anyone who doesn't think that's an operational issue, just wait until it bites you on the ass.
Now we have clear evidence that there are no less than three who understand the threat.
My first thought was that the DDoS was a means of obscuring access to patches for other vulnerabilities that might be simulatenously exploited. I'm assuming, though not certain, that Cisco would have alternative distribution/communication/update channels in such an event, but is that indeed the case? -ed ----------------- ed@the7thbeer.com
On Mon, 6 Oct 2003, ed@the7thbeer.com wrote:
Anyone who doesn't think that's an operational issue, just wait until it bites you on the ass.
Now we have clear evidence that there are no less than three who understand the threat.
My first thought was that the DDoS was a means of obscuring access to patches for other vulnerabilities that might be simulatenously exploited. I'm assuming, though not certain, that Cisco would have alternative distribution/communication/update channels in such an event, but is that indeed the case?
My access to ftp.cisco.com is working fine whilst the website remains down.. Steve
I'm assuming, though not certain, that Cisco would have alternative distribution/communication/update channels in such an event, but is that indeed the case?
My access to ftp.cisco.com is working fine whilst the website remains down..
Hi Steve, No I do realize that what I suggested in my email was just a scenario removed from the case at hand. What I was suggesting though is that 1) if the portals of distribution (http and ftp) were DDoS'd, say as a precursor to exploitation of some other vulnerability. I was not trying to suggest that all means of communication were blocked and that this particular instance was one of an opportunistic DDoS. Sorry if I was unclear. -ed ----------------- ed@the7thbeer.com
On Mon, 06 Oct 2003 19:38:38 EDT, jlewis@lewis.org said:
A handful of people (an assumption on my part) have the power / distributed bandwidth to bring just about any internet site/network to its knees using the distributed.net meets DoS tools they've created and distributed to thousands, perhaps millions of internet connected windows boxes.
Zombie networks of 10K or 20K machines all controlled by *one* black hat are not uncommon now, and I've seen a citation for a single net of 140K. Let's assume the interesting hosts are on cablemodem, that they have 2Mbit/sec connectivity, and that one black hat has 10K (if you prefer, he's got 20K but the rest are on slow links). Now tell me - how many of you have enough *excess* bandwidth that you can afford not to worry about suddenly being handed a 200Gbit/sec inbound stream? And if you don't have enough spare capacity, are you set up to deal with 10K machines attacking, quite possibly with spoofed addresses because your peers don't ingress filter? Remember guys - Yahoo got whacked by MafiaBoy using only several hundred machines. You could be the recipient of a flood 200 times bigger. And if you're not ready, it won't be an operational issue - it will be a NON-operational issue, because that's what your network will be....
We've been handling a multi-vector DDoS - 40-byte spoofed SYN-flooding towards www.cisco.com
Now that they've come for cisco, maybe law enforcement, network operators, and router vendors will all get their $h!t together and do something to put a stop to these DDoS attacks that have been going on in various forms for several years.
Maybe this will have the positive effect of motivating Cisco to do more to encourage best practices such as edge anti-spoof filtering. To begin with, Barry Green's presentations on these issues are hidden away on his/Cisco's FTP server (ftp://ftp-eng.cisco.com/cons/) -- maybe it would be beneficial to put them (along with write-ups) in an easily-accessible and often-visited area of the main site where people will see them. These issues aren't just for ISPs: if edge networks would filter their borders, ISPs wouldn't have to do it for them. (Or in most cases, fail to do it for them.) -Terry
Terry Baranski [10/7/2003 6:05 AM] :
Maybe this will have the positive effect of motivating Cisco to do more to encourage best practices such as edge anti-spoof filtering. To begin with, Barry Green's presentations on these issues are hidden away on his/Cisco's FTP server (ftp://ftp-eng.cisco.com/cons/) -- maybe it would be beneficial to put them (along with write-ups) in an easily-accessible and often-visited area of the main site where people will see them.
There is of course BCP 38 for starters - http://www.armware.dk/RFC/bcp/bcp38.html srs -- Suresh Ramasubramanian <suresh@outblaze.com> gpg# EDEDEFB9 Security and Antispam Operations Manager, Outblaze Limited
On Tue, 7 Oct 2003, Suresh Ramasubramanian wrote:
Terry Baranski [10/7/2003 6:05 AM] :
Maybe this will have the positive effect of motivating Cisco to do more to encourage best practices such as edge anti-spoof filtering. To begin with, Barry Green's presentations on these issues are hidden away on his/Cisco's FTP server (ftp://ftp-eng.cisco.com/cons/) -- maybe it would be beneficial to put them (along with write-ups) in an easily-accessible and often-visited area of the main site where people will see them.
There is of course BCP 38 for starters - http://www.armware.dk/RFC/bcp/bcp38.html
You are making assumptions.. Cisco havent said if the source was spoofed or not, as a recent nanog thread indicated a lot of attacks do not use spoofed addresses any more simply because the controllers have access to enough legitimate windows boxes to not care about discovery of source. Even with all your BCPs in place if someone can get control of enough machines across enough networks collectively they can produce enough traffic to overwhelm absolutely any single system on the Internet. I am increasingly sharing the opinion that many of these high profile attacks are carried out by a small group.. spammers or whoever they are, the only way to tackle them is directly by hunting them down and prosecuting them. Assuming that there is a cash motivation somewhere (eg spam) this also means that there is a very high probability the attackers reside in a country where prosecution would be possible eg US/Europe Steve
Stephen J. Wilcox [10/7/2003 6:06 PM] :
You are making assumptions.. Cisco havent said if the source was spoofed or not, as a recent nanog thread indicated a lot of attacks do not use spoofed addresses any more simply because the controllers have access to enough legitimate windows boxes to not care about discovery of source.
I did say "for starters". I put it to you that there is still a non trivial amount of attacking going on that does use spoofed traffic. Yes, there are lots of IRC controlled zombies, and yes, there are pissed off teenage skript kiddies who shut down the port of houston's servers trying to bomb someone they had a pissing match with on IRC (don't have more details than what I read on Dave Farber's IP list today).
I am increasingly sharing the opinion that many of these high profile attacks are carried out by a small group.. spammers or whoever they are, the only way to tackle them is directly by hunting them down and prosecuting them. Assuming that there is a cash motivation somewhere (eg spam) this also means that there is a very high probability the attackers reside in a country where prosecution would be possible eg US/Europe
Easier said than done. First - prove that the guy did it (or hired a kiddie in china or eastern europe or wherever to do it) Next, prove to the Feds that damage > [what, USD 25K?] was caused. And that is for starters. srs -- Suresh Ramasubramanian <suresh@outblaze.com> gpg# EDEDEFB9 Security and Antispam Operations Manager, Outblaze Limited
"Stephen J. Wilcox" wrote:
You are making assumptions.. Cisco havent said if the source was spoofed or not, as a recent nanog thread indicated a lot of attacks do not use spoofed addresses any more simply because the controllers have access to enough legitimate windows boxes to not care about discovery of source.
Interesting. I read (and just now reread) Mr. dobbins posting and made the same assumptions, based on the part where he said: We've been handling a multi-vector DDoS - 40-byte spoofed SYN- ~~~~~~~ flooding towards www.cisco.com (198.133.219.25/32) as well as an HTTP-AUTH resource-exhaustion attack, and working these issues with our upstreams. I made the assupmtion that if the upstreams had an interest in cisco's survival beyond the end-of-quarter numbers they would do something useful. Strange how we leap to these shaky conclusions.
As the bandwidth ramps up on the access side, this problem is only going to become more and more prevalent (as if it's not already enough of a problem). While I don't think filtering is the silver bullet, it can certainly help at times. I think some of the larger watch sites (eg SANS, etc.) out there have the right idea - even though reactive in nature, almost real-time dissemination of attack vectors and trending of methods goes a long way towards slowing down some of these attacks. Unfortunately, these single target attacks, such as attacks on Cisco, Ebay, Yahoo, etc. cannot be entirely thwarted if the attacker(s) is/are determined enough. We could go down the client side discussion (you know, the one about certain software vendors, etc.) but that topic has already been covered in great length.
On 6 Oct 2003 at 19:22, Allan Liska wrote:
I don't know what your post has to do with the original topic, but if you don't like the way NONOG is moderated, please feel free to start your own Network Operators mailing list.
As far as comparing NANOG moderation to Nazi Germany that is disgusting and beneath contempt.
Read it again. He has a point (not yours). Perhaps this should be an agenda topic for the upcoming get- together: A common strategy for dealing with Internet crime. Much of it does appear to have common roots. (And I'm not even a conspiracy buff.) Hm. Oddly enough there's a blurb on <overclockers.com> that follows this somewhat: <http://www.overclockers.com/articles843/>. Peter E. Fry
On Mon, 6 Oct 2003, Peter E. Fry wrote: Hi, As a jew, I must admit that I also understood the point, and didn't think of Nazi Germany, although you'd think it would evoke an immediate emotional reaction (which it admitedly did), but that reaction did not cloud my judgement. I think it's safe to assume that most people on this list have a reason for being on it. Although I am not trying to say that sometimes we get to see posts that are ... well, that shouldn't be sent before thinking, it would be wise to read an e-mail twice, even three times, before assuming mal-intent from its originator. peace, --Ariel
Read it again. He has a point (not yours). Perhaps this should be an agenda topic for the upcoming get- together: A common strategy for dealing with Internet crime.Much of it does appear to have common roots.(And I'm not even a conspiracy buff.) Hm. Oddly enough there's a blurb on <overclockers.com> that follows this somewhat: <http://www.overclockers.com/articles843/>.
Peter E. Fry
-- Ariel Biener e-mail: ariel@post.tau.ac.il PGP(6.5.8) public key http://www.tau.ac.il/~ariel/pgp.html
-----BEGIN PGP SIGNED MESSAGE----- Hash: MD5 Hello Allan, Monday, October 6, 2003, 7:22:30 PM, you wrote: AL> As far as comparing NANOG moderation to Nazi Germany that is AL> disgusting and beneath contempt. My apologies to Kai and the list, I misread -- to some extent -- the original meaning of the post. My comments were certainly harsher than warranted. allan - -- Allan Liska allan@allan.org http://www.allan.org http://www.hosthideout.com -----BEGIN PGP SIGNATURE----- Version: 2.6 iQCVAwUAP4IEiSkg6TAvIBeFAQG7mAQAsiNj+6O0K7LXpnXFgLDI/0135zCoSgW1 qQXXQLJ55VxofXl68YcATV6ANCNzmsOXVcztnO3u8k2WXfxWhpXqXTItdf2JMTCH i0T/VfjXDl7GTSwhBKGh2JF07qlO9r8J94qOaegvIsz9bnNpuKrd4PyUzofcWRSx W9k/4C5v23k= =ALrZ -----END PGP SIGNATURE-----
participants (16)
-
Allan Liska -
Ariel Biener -
Charles Sprickman -
ed＠the7thbeer.com -
jlewis＠lewis.org -
Kai Schlichting -
Laurence F. Sheldon, Jr. -
Marc Binderberger -
Matt -
Peter E. Fry -
Richard Cox -
Roland Dobbins -
Stephen J. Wilcox -
Suresh Ramasubramanian -
Terry Baranski -
Valdis.Kletnieks＠vt.edu