What you say here makes sense to me. But, out of a 500 ping sample over the course of half a day, I was getting well below one percent loss -- six months ago. Now, my same sample group is going for more than three percent, with many ten percent loss routes showing. This has changed and has changed commensurately with the throughput problems shown in TCP and name lookup problems via UDP. It may not be scientific proof, but it could very well be a good scientific indicator. An indicator is all I really need for the most part.

There is considerable difference between forwarding a packet that happens to contain ICMP data (destination not the router) and responding to a packet that contains ICMP data (destination is the router). In the former, priority in a Cisco is the same for ICMP as it is for UDP or TCP, since this part of the packet is not even being examined. In the later, priority is lower and can be ignored altogether. I treat ignored (link good, but no response received) ICMP echo requests as an indicator that a router is too loaded. If the router has been pushed to the point of not being able to respond to an ICMP, how well is it going to do when a bunch BGP updates occur? (rhetorical) Both are CPU intensive operations. Chris

Would someone please tell me just why icmp echos are "cpu intensive"? Yes, I know they're in software. So what? A PC can respond to an ethernet loaded with them with a trivial percentage of it's CPU cycles. Recall that the high end routers do most of their forwarding in hardware. So echos are "cpu intensive" in that they take many, many, many more CPU cycles to process than packets which are forwarded by the hardware. For example, in a cisco 7000, the main traffic flow is switched by the silicon switch engine. Pings of the router themselves cause the packet to leave the silicon switch engine and sit on a queue (consuming buffer space), waiting for the CPU. When the CPU gets around to it, the packet is then copied out of the hardware up to the CPU. The CPU then makes a normal forwarding decision, and decides to receive the packet. It then decides to format the reply. Now it makes another forwarding decision, copies the packet back down into a hardware buffer and squirts it out. This sounds to me a whole lot more like a solution to an imagined problem, but I'm prepared to be convinced that responding to pings actually takes a great enough percentage of CPU cycles to slow down packet delivery..... In fact, responding to pings does not directly affect the speed of the hardware forwarding engine. However, it does increase buffer utilization in the hardware. It also deters routing protocol processing from happening, as it's consuming CPU. Tony

The fact remains that a ping packet stream a Linux 386SX would barely notice maxes out a 7010 (far more powerful CPU) Bzzzt. That's a 30Mhz 68040 you're talking about. You're 386SX is on par if not ahead. And you might recall that it's handled at process level, whereas Linux does it at kernel level (or at least other Unixen do). Rather and obvious DoS attack, and one which even MS were red faced enough to fix in their NT s/w pretty sharpish. You can DoS attack anything with echos. Trying to make echo handling "fast enough" is an untenable problem. So you should simply drop them on the floor... Tony

The fact remains that a ping packet stream a Linux 386SX would barely notice maxes out a 7010 (far more powerful CPU)

Bzzzt. That's a 30Mhz 68040 you're talking about. You're 386SX is on par if not ahead.

Well I'd have said 68030~=386SX but anyway...

And you might recall that it's handled at process level, whereas Linux does it at kernel level (or at least other Unixen do).

Well this is the real point isn't it. Though Cisco has obviously paid an awful lot of attention to switching packets fast its host IP stack seems to be particularly bad compared to something built with a decent TCP/IP stack. If Cisco don't implement ICMP response in some sort of kernel layer below "application" layers handling other such functions the obvious question is "why not?". It must be possible to do this - just take your Netstar and downgrade the processor to a 386SX and see if it still works (fast external switching engine, slow processor).

Rather and obvious DoS attack, and one which even MS were red faced enough to fix in their NT s/w pretty sharpish.

You can DoS attack anything with echos. Trying to make echo handling "fast enough" is an untenable problem. So you should simply drop them on the floor...

I believe Microsoft's response was (a) to handle ICMP echos further down in the kernel or at least without breaking upper layers quite so much and (b) to drop ICMPs if they arrived too often. I did *not* say Cisco should answer all ICMP echo requests, just not break (or try and avoid it). "Even" Solaris has things to tweak ICMP response. With all respect (as you know far more about Cisco inards than I do) this sounds very like people at certain OS vendors who said "SYN flood attacks, ah yes, well such DoS attacks are inevitably extremely difficult to prevent". Indeed. But will it really take someone ICMP streaming with forged source addresses at Sprint's core routers for Cisco to fix it? Alex Bligh Xara Networks

On Fri, 20 Dec 1996, Jim Van Baalen wrote:

I have a question that fits this topic. Why does everybody seem to be so sold on Gigaswitch based Xchange points? I know that the busiest, highest membership NAPs are of this design, but with all the associated problems

I think the biggest reason the busiest and highest membership NAPs are using Gigaswitches because they work. When we were looking for a switch for the Atlanta-NAP we almost with with a Stratacom switch, but the Gigaswitch looked like a better start. We plan on adding a Stratacom

It is my opinion that the most promising architecture today is PBNap. The StrataCom switches far exceed Gigaswitch in addressing issues such as flow control, buffering, and aggregate throughput (which may not be comparable when, as with Gigaswitch, port congestion is such a bottleneck). In addition, with new line cards due out early next year, the BPXs will support ABR and, relatively speaking, huge buffers at high density OC3 and 2 port OC12.

Well when they get that out, and fix a few other problems it may be ready.

I know that ATM is a nasty word (or should I say acronym) to many on this list, but even some of the greatest ATM critics admit that it is a reasonable LAN technology. As far as the popular packet shredder (sp?) argument goes, I don't think that it relevant on any switch that supports EPD.

Just curious to see other perspectives.

Jim

Nathan Stratton CEO, NetRail, Inc. Tracking the future today! --------------------------------------------------------------------------- Phone (703)524-4800 NetRail, Inc. Fax (703)534-5033 2007 N. 15 St. Suite 5 Email sales@netrail.net Arlington, Va. 22201 WWW http://www.netrail.net/ --------------------------------------------------------------------------- "Therefore do not worry about tomorrow, for tomorrow will worry about itself. Each day has enough trouble of its own." Matthew 6:34

The GIGAswitch has theoretical difficulties. Head of line blocking and so on. But the exchanges that use GIGAswitches aren't having measurable problems which are traceable to the GIGAswitches. The "O" in NANOG is for "operations" and in the field of operations, anything that works gets used.

Is it the concensus of this group that the "GIGAswitches aren't having measurable problems"? I have recently received messages from the "Operators" at MAE-West that don't seem to support this position.

There is a conclusion to be drawn from this regarding ATM.

I run an IP/ATM network with which I am happy. Most NSPs either use ATM for transport or are rumored to be moving in that direction. Jim

I think that there's some lack of clarity on the problem here. Anyone can stream packets at ANY router and take it down. If it's not ICMP, you can simply forge routing protocol packets. It's a question of simply supersaturating the system. To truly deal with DoS attacks, there are basically three approaches:

Indeed. For instance SYN-flood the BGP port.

1) Throw money at the problem. Build a big box that has enough processor to deal with the incoming bandwidth for pessimal packets. Even then, the bad guys can simply supersaturate the incoming bandwidth.

2) Deal with it statistically. For example, most folks for the recent syn attacks will drop syns if they don't complete reasonably, thereby allowing some percentage of real traffic to get through.

3) Deal with it legally. This is what the telco's do. It implies that we would need real mechanisms for tracking down offenders.

Can I have 2(a) - deal with it statistically and intelligently. TCP/IP stacks which have got far greater public flak than Cisco's (Solaris 2.4 for instance) do not die when sent 128kb/s of ICMP. As I understand it 11.1 allows access lists based on icmp packet type, and this filtering is already done off CPU. So "all" the CPU has to do is block ICMPs from particular hosts, or (even) ICMP at all, if it is being flooded.

As to what cisco will do, you should probably ask cisco.

I did. They said "the problem doesn't exist". I am circulating the problem (before, like SYN flods, it becomes a serious operational problem) to those with larger annual Cisco spend than me. Background to bug: We discovered this when we had 2 telco lines running in parallel and wanted to check the performance of one from a host behind one router, and had no hosts of our own behind the other router. Naively we thought pinging the other (NAP) router would be a good test with our stochastic bandwidth modeling tool, which is based on ICMP. Rather an unpleasant thing happened to our transit. Just wait until someone decides you should measure your ISPs performance by running ping -s 1000 mae-east.sprintlink.net (8kb/s). Now get 16 people doing it at once, and ... Alex Bligh Xara Networks

...

Indeed. For instance SYN-flood the BGP port.

Won't work easily. On Criscos, the queue is per peer, not per port..

Spoof the source address to that of a peer. In fact I'd have thought anything out of sequence would generate an RST and clear the session. Hmmm.. fragile world we live in. Alex Bligh Xara Networks

Indeed. For instance SYN-flood the BGP port.

Correct me if I'm wrong but to the best of my recollection, in order for a packet to be accepted on the BGP port, it must be originating from a configured BGP peer. Since the SYN flood method relies on the attack originating from an unreachable (yet routable) address, it would seem that this approach will fail. If you're out for a true DoS attack, it need not even be a SYN attack. Simply flooding the BGP port would be quite enough to bring the system to its knees. Forge a known peer's source address, and even the CPU that it burns testing for authentication and discarding packets would be enough to be fatal. The important point is that you can't distinguish the good from the bad without a whole lot of work. Tony

Can I have 2(a) - deal with it statistically and intelligently. TCP/IP stacks which have got far greater public flak than Cisco's (Solaris 2.4 for instance) do not die when sent 128kb/s of ICMP. As I understand it 11.1 allows access lists based on icmp packet type, and this filtering is already done off CPU. So "all" the CPU has to do is block ICMPs from particular hosts, or (even) ICMP at all, if it is being flooded. You can have anything you like ... at Alice's Restaurant. ;-) Assuming we're still talking about a 7010, I suspect that you could do incoming ICMP filtering in the SSE and discard those. But then the bad guys simply attack your BGP port to circumvent your filters. And the filters are not intelligent enough to perform the authentication computation. I'm surprised it's as low as 128kb/s. It should be more around 2kpps. Not that this is a stretch. ;-) I did. They said "the problem doesn't exist". What? And you didn't believe them? ;-) I suspect that a better approach is to contact the people with clue directly.... it sounds like you went through TAC. Tony

why even do that? i'm not sure i want you triggering security mechanisms on my routers. Especially with the overhead implications, though that is the thread we're currently in [may it die soon]. building an acl that allows packets matching those you're interested in, and applying it to 'debug ip packet ACL detail' is fairly simple. just sit there doing 'clear ip cache A.B.C.D W.X.Y.Z'. Find the next hop it's coming from, trace it along, mail your friendly peer or transit provider, or mail your friendly hacker's admins. granted, this is limited to the domain of routers you control, but it's pretty effective for finding out where the syn attack is coming from. this assumes the people who are dumb enough to keep syn-ing continue to be stupid enough to use originating source addresses like 234.231.0.33. -alan ] > 3) Deal with it legally. This is what the telco's do. It implies that we ] > would need real mechanisms for tracking down offenders. ] ] Personally, I'd like to see a protocol that allows you to ask a ] router to which you were directly connected to stamp an interface ID on ] all incoming packets bound for a particular network. You could then trace ] back router by router, interface by interface, where the packets were ] entering a block of cooperating providers. ] ] Thus if I saw an incoming flood of SYN packets or ICMP echoes ] with forged origin addresses, I could ask my router to ask all its direct ] peers to begin stamping interface numbers (and/or interface IPs) on the ] packets they send to me. My router would eat those numbers/IPs so traffic ] would appear unaffected. ] ] Then my tracing tool would know which interface the packets were ] coming in on and could ask that router to do the same thing (on a ] hop-by-hop basis for security reasons). Thus I could track it back to a ] specific enough interface path that perhaps an automated method to ] install a filter would be sufficient. ] ] This stuff needs a lot of work, but might be a direction that ] would both facilitate emergency filtering and effective tracing for IP ] packets with forged origin addresses -- assuming the packets have enough ] in common to allow them to be detected (all pings, or heavy load, or all ] to same destination IP). ] ] David Schwartz ]

Except for one small problem, Unless you're _HUGE_ most NSP's (ie. MCI, sprint, uunet) don't give a flying fuck and won't spend the time and manhours it takes to track these things down. At one point one of our main machines was being synflooded on almost every port, mci refused to do a thing about it because it would 'take too long'. On Fri, 20 Dec 1996, Alan Hannan wrote:

why even do that? i'm not sure i want you triggering security mechanisms on my routers. Especially with the overhead implications, though that is the thread we're currently in [may it die soon].

building an acl that allows packets matching those you're interested in, and applying it to 'debug ip packet ACL detail' is fairly simple.

just sit there doing 'clear ip cache A.B.C.D W.X.Y.Z'. Find the next hop it's coming from, trace it along, mail your friendly peer or transit provider, or mail your friendly hacker's admins.

granted, this is limited to the domain of routers you control, but it's pretty effective for finding out where the syn attack is coming from.

this assumes the people who are dumb enough to keep syn-ing continue to be stupid enough to use originating source addresses like 234.231.0.33.

-alan

] > 3) Deal with it legally. This is what the telco's do. It implies that we ] > would need real mechanisms for tracking down offenders. ] ] Personally, I'd like to see a protocol that allows you to ask a ] router to which you were directly connected to stamp an interface ID on ] all incoming packets bound for a particular network. You could then trace ] back router by router, interface by interface, where the packets were ] entering a block of cooperating providers. ] ] Thus if I saw an incoming flood of SYN packets or ICMP echoes ] with forged origin addresses, I could ask my router to ask all its direct ] peers to begin stamping interface numbers (and/or interface IPs) on the ] packets they send to me. My router would eat those numbers/IPs so traffic ] would appear unaffected. ] ] Then my tracing tool would know which interface the packets were ] coming in on and could ask that router to do the same thing (on a ] hop-by-hop basis for security reasons). Thus I could track it back to a ] specific enough interface path that perhaps an automated method to ] install a filter would be sufficient. ] ] This stuff needs a lot of work, but might be a direction that ] would both facilitate emergency filtering and effective tracing for IP ] packets with forged origin addresses -- assuming the packets have enough ] in common to allow them to be detected (all pings, or heavy load, or all ] to same destination IP). ] ] David Schwartz ]

[-] Brett L. Hawn (blh@nol.net) [-] [-] Networks On-Line - Houston, Texas [-] [-] 713-467-7100 [-]

First let me appologize for sending my response to the whole list, working in several windows causes mental errors at times. My intent was only to reply to the original author rather than the rest of the list. That said, I'd love to hear from anyone thats actually had any sucess in dealing with their NSP and synfloods. I know quite a few folks who'd love to get some insight on how to force their upstream providers to help them track down the culprits, or at least stop the flow. I use IRC as an example (not because anyone actually cares about it, but because its a popular target for small children with even smaller minds.), servers are consistently synflooded by some twit who's mad because someone klined him for being naughty. Joe Admin sees this, calls his upstream, who politely tells him, (and you may consider this an almost direct quote from MCI during one of my experiences), "We're sorry, but we simply can't spend the time it would take to trace this and help you with that problem. It takes an act of God to get one of our supervisors to ok it, and even then it would take at least 10 or more man hours just to find where its entering our path." Now I understand that its time consuming and lord knows crisco's debug is hard on the eyes after a while but.. I'm willing to believe that if you make enough examples of busting the abusers word will get around. Granted there are always going to be people who think they can (and often do) beat the system, but the more effort you put into catching them, the better you become at it, and who knows.. maybe you'll develop a better method. On Fri, 20 Dec 1996, Alan Hannan wrote:

mileage will vary with provider and the person within the company with whom you're working . while ideally results would just appear, I do believe that with proper escalation and persistence, you can get assistance. your sales person can be of help here.

i know that uunet and mci do give sig. weight to syn attacks, and will work to determine the source. I'm sure others do too.

-alan

] ] Except for one small problem, Unless you're _HUGE_ most NSP's (ie. MCI, ] sprint, uunet) don't give a flying fuck and won't spend the time and ] manhours it takes to track these things down. At one point one of our main ] machines was being synflooded on almost every port, mci refused to do a ] thing about it because it would 'take too long'. ] ] On Fri, 20 Dec 1996, Alan Hannan wrote: ] ] > ] > ] > why even do that? i'm not sure i want you triggering security ] > mechanisms on my routers. Especially with the overhead ] > implications, though that is the thread we're currently in [may it ] > die soon]. ] > ] > building an acl that allows packets matching those you're ] > interested in, and applying it to 'debug ip packet ACL detail' ] > is fairly simple. ] > ] > just sit there doing 'clear ip cache A.B.C.D W.X.Y.Z'. Find ] > the next hop it's coming from, trace it along, mail your ] > friendly peer or transit provider, or mail your friendly hacker's ] > admins. ] > ] > granted, this is limited to the domain of routers you control, ] > but it's pretty effective for finding out where the syn attack is ] > coming from. ] > ] > this assumes the people who are dumb enough to keep syn-ing ] > continue to be stupid enough to use originating source addresses ] > like 234.231.0.33. ] > ] > -alan ] > ] > ] > ] > 3) Deal with it legally. This is what the telco's do. It implies that we ] > ] > would need real mechanisms for tracking down offenders. ] > ] ] > ] Personally, I'd like to see a protocol that allows you to ask a ] > ] router to which you were directly connected to stamp an interface ID on ] > ] all incoming packets bound for a particular network. You could then trace ] > ] back router by router, interface by interface, where the packets were ] > ] entering a block of cooperating providers. ] > ] ] > ] Thus if I saw an incoming flood of SYN packets or ICMP echoes ] > ] with forged origin addresses, I could ask my router to ask all its direct ] > ] peers to begin stamping interface numbers (and/or interface IPs) on the ] > ] packets they send to me. My router would eat those numbers/IPs so traffic ] > ] would appear unaffected. ] > ] ] > ] Then my tracing tool would know which interface the packets were ] > ] coming in on and could ask that router to do the same thing (on a ] > ] hop-by-hop basis for security reasons). Thus I could track it back to a ] > ] specific enough interface path that perhaps an automated method to ] > ] install a filter would be sufficient. ] > ] ] > ] This stuff needs a lot of work, but might be a direction that ] > ] would both facilitate emergency filtering and effective tracing for IP ] > ] packets with forged origin addresses -- assuming the packets have enough ] > ] in common to allow them to be detected (all pings, or heavy load, or all ] > ] to same destination IP). ] > ] ] > ] David Schwartz ] > ] ] > ] ] [-] Brett L. Hawn (blh@nol.net) [-] ] [-] Networks On-Line - Houston, Texas [-] ] [-] 713-467-7100 [-] ]

[-] Brett L. Hawn (blh@nol.net) [-] [-] Networks On-Line - Houston, Texas [-] [-] 713-467-7100 [-]

On Fri, 20 Dec 1996, Brett L. Hawn wrote:

Except for one small problem, Unless you're _HUGE_ most NSP's (ie. MCI, sprint, uunet) don't give a flying fuck and won't spend the time and manhours it takes to track these things down. At one point one of our main machines was being synflooded on almost every port, mci refused to do a thing about it because it would 'take too long'.

Then sue MCI for denial of service. You will win in court because it is trivial to show that the packets came into your network from MCI and that MCI's employees could not and would not provide any information to you that would indicate that these packets do not originate from MCI. Thus they are not providing you the service that they contracted to provide. Not everything has a technical solution. Mind you, you might do better to be a bit more diplomatic and take up the issue with MCI's management. Just keep in mind the basic principle that MCI is responsible for the DoS traffic unless they are willing to identify the source as somewhere other than MCI. Michael Dillon - Internet & ISP Consulting Memra Software Inc. - Fax: +1-604-546-3049 http://www.memra.com - E-mail: michael@memra.com

Not all DoS attacks come in the form of packets. A couple of months ago, some unhappy person (not an employee or former employee even) placed a single phone call to Bell Atlantic and another to our 800 provider, and convinced them both to disconnect all our NOC voice, data and fax phone lines as well as our 800 service. The disconnects took place within a few hours of the phone calls and also after the business offices were closed. The upshot was, it took over 20 hours to get the service restored. When we tried to involve law enforcement, they laughed at us. Just because you are paranoid, does not mean they aren't out to get you, and will not find new ways to do it. Best Regards and Seasons Greetings, Robert Laughlin ---------------------------------------------------------------------------- DataXchange sales: 800-863-1550 http://www.dx.net Network Operations Center: 703-903-7412 -or- 888-903-7412 ----------------------------------------------------------------------------

At 3:25 PM -0500 12/19/96, David Miller wrote:

Would someone please tell me just why icmp echos are "cpu intensive"?

Yes, I know they're in software. So what? A PC can respond to an ethernet loaded with them with a trivial percentage of it's CPU cycles.

a loaded ethernet is not the problem, a dozen OC3's worth of traffic is the challenge. Please let us not digress the NANOG list into "router design 101" again. thanks, dave