- NANOG - lists.nanog.org

Re: Ping flooding (fwd)
by Daniel W. McRobb 10 Jul '96

10 Jul '96

> On Jul 9, 20:31, "Daniel W. McRobb" <dwm(a)ans.net> wrote: > > When I asked Cisco about this (a while ago), they said flow-switching > > incurred a 20% overhead (which someone there called "minimal", which I > > didn't agree with). > > I guess this has changed; we have a clear performance improvement. > > > 'tis very unwise to run IP accounting on a very busy router. We > > wouldn't dare turn it on for a core router w/ 40,000 routes. Some of > > our customers who have done so on border routers immediately turn arond > > and complain about performance problems. :-( And we're not talking > > Yes, if the box doesn't have the CPU to do it, don't. But I said > 'edge routers'. Core routers are the wrong places to do it anyway, > depending on paradigm; but generally, traffic will be collected from > and broken down into fairly small streams. It's only when it passes > through core routers the proportions are too large; a 'border > router' for a large network/customer is in this context a core router. "fairly small" streams is only true if you don't have bandwidth-heavy customers. We have several customers w/ lots of bandwidth, who use it. > > 'tis also very important to know the route taken for a net when > > analyzing the data. Discrepancies here can be huge and completely > > invalidate any conclusions you might make about how much traffic is > > traversing a given path. > > For this we like combined interface stats; AS-based traffic matrices > are (from our point of view) mostly useful for end-to-end measurements. > > > traversing a given path. This is particularly true for busy end routers > > (like NAP peers) and core routers. > > One wouldn't do it here anyway, as noted. Then you potentially miss huge contributions to your network load (NAP routers <-> T3 customers, for example). > > I also prefer measurements that won't potentially get dropped in > > transit (throwing a big unknown into confidence level of data). Is > > there an efficient means of retrieving flow data from a Cisco and not > > potentially dropping a bunch of it on the floor? > > You may drop something from time to time, but this simply turns the > data collection into sampling. With a sufficient sample size, this > is as good as anything; you still want to correlate your information > with other sources, such as interface stats, and adjust accordingly. > But please, let us not turn this into "my box is better than your > box". I wasn't doing that. I asked an honest question. Dropping data at indeterminate points, unknown to the measurement tool, is not turning collection into sampling (well, at least not useful sampling). If you have no means of determining how many were dropped, and under what conditions, you have essentially useless data. Interface stats won't get you that information. If you drop traffic at times of heavy bursts you lose some of the most interesting data (that which may be causing your network real pain). Is there an efficient means of retrieving flow data from a Cisco and not potentially dropping a bunch of it on the floor? Will there be at some point in the future? We'd use it if it was available. We'd even pay for it, I imagine. :-) Daniel ~~~~~~

3 3

Re: Ping flooding (fwd)
by Sean Doran 10 Jul '96

10 Jul '96

| But please, let us not turn this into "my box is better than your | box". Why not? I'm sure there are people who would LOVE to see people buying up all the NSSes that ANS is throwing away. They're *fabulous*. Sean.

2 1

Re: Ping flooding
by rmg＠ranma.com 10 Jul '96

10 Jul '96

From: Michael Dillon <michael(a)memra.com> > On Tue, 9 Jul 1996, Daniel W. McRobb wrote: > > > There will likely never be a means for a single NSP to track down the > > real source of spoofed packets using IPv4. Service providers won't be > > letting other service providers track spoofed packets through their > > network. > > Why not? Don't telcos do this? Yes, telcos do this, but they (used) to have the same problem we all have in the ISP world, in that your average DMS-100 voice switch is optimized for call processing, not for call-detail searches. Your average call-detail search used to take 1-2 hours for a 5 minute window. (I say "used to" as now the SS7 STP processors now do the call-detail recording, and call lookups are a matter of keystrokes and seconds away.) Router mfgrs are still in the stages of switching packets as fast as they can, not detail management. And of course, nooone of us want to drop our routers down to process switching to track packets. > Or if your answer is that telcos only do it for the police and not for > each other, then my question would be why can't we form an Internet > equivalent, maybe affiliated with something like CERT, that can make these > requests and with whom NSP's would cooperate. Telco call-detail lookups for law enforcement constitute <-.01% of those lookups (I did work at MCI's Western Region Net Mgt canter). All the other lookups are for maint purposes (like finding marginal trunks, tracking call patterns, making sure routing databases are working right, etc). It's obviously going to be different in our case. rob.

2 1

Re: Ping flooding (fwd)
by Sean Doran 10 Jul '96

10 Jul '96

| I have always been under the impression that Cisco flow switching and | high performance were mutually exclusive if there were too many active | flows as is the case for the major US ISPs at least. Not quite. The flow switching path was tuned manually by Darren Kerr, who is an impressive bit-twiddler. He spent some time watching how his path behaved in a huge network (mine) and in a big network (Dorian's) and adjusted it accordingly. The current path appears -- to me, qualitatively -- to consume less CPU per packet per second in 7500s than the other released switching methods. Yes, I was seriously surprised too, however People That Know with three letter logins weren't. There are some memory concerns and some concerns about mixing optimum switching and flow switching in a busy box with recent code, however these are bugs that don't affect normal processing. | Another difference is with the flow switching, you need to catch them | in the act. With the sampling and collection, you can call hours | later (days or weeks actually, years if you count going to tape) and | still determine the candidate entry points for the traffic. I don't | think there is a practical way to get the same sort of historic | archive from the flow switching stats. conf t flow-export <ip-address> ^Z and have a client do whatever you want with the stream of UDP datagrams that will be sent to you. The dkerr stuff is neat. I'm looking forward to the completion of the other work in progress by the folks who've inherited it, since that has considerable neatness potential too. Sean.

2 1

Re: Ping flooding (fwd)
by Daniel W. McRobb 10 Jul '96

10 Jul '96

> | But please, let us not turn this into "my box is better than your > | box". > > Why not? I'm sure there are people who would LOVE to see > people buying up all the NSSes that ANS is throwing away. > > They're *fabulous*. > > Sean. Folks, that wasn't the point (at least not mine). The point was (from me anyway) that this kind of data collection can be very useful for network architecture, and I'd love to have it in all our routers (and I really couldn't care less who the vendor is; it just so happens we have a lot of Ciscos so that's just one where I'd like to have it). We've used the data quite a bit. Blurbs about the NSSes were just clarifications of what it is we'd like to have elsewhere (there are several things about the NSS collection of this data that are less-than-optimal in a world with lots of CIDR and subnetting, if that gives anyone a warm fuzzy :-)). I don't think ANS is alone here, but if so, I'll crawl back under my rock. ;-) I know there were folks at ISMA interested in this kind of data; maybe they've changed their minds. >From my perspective, it would be nice if we (as ISPs) actually had a reasonable consensus on what kinds of data we want from our routers that we can't currently get (whether due to no implementation, troublesome implementation, or just bad hair days). Just MHO, and possibly well outside nanog. I'm still curious about field experiences w/ running the flow-switching stuff on a busy router. Anyone out there doing it? Define busy however you'd like, as long as you clarify. Private email is fine. Daniel ~~~~~~

1 0

On another topic ... :-)
by John Scoggin 10 Jul '96

10 Jul '96

I ran across an interesting document on the FCC web site and was wondering if there was an Internet analog - or maybe there should be. Apparently, the FCC started a Network Reliability Council composed of IECs, LECs, CAPs, and vendors to compose a best practices manual. What makes it interesting is that they surveyed the various carriers five years later to determine which practices had been adopted, to what extent they had been implemented, and whether they were deemed cost effective. The results were rendered in a report from the Office of Engineering & Technology (FCC). I sent mail to the FCC folks yesterday trying to track down the original report to aid me in starting our carrier operations off on the right foot. Does something like this best practices document exist? If not, here is an opportunity for someone to make some money putting one together - I'd gladly pay a couple hundred bucks for one! gratituitous barb follows --> [maybe Metcalfe could write one :-) ] John K. Scoggin, Jr., Staff Technical Consultant, Advanced Technologies / Chief Technical Advisor, Telecom Infrastructure Services Delmarva Power (scoggin(a)delmarva.com) 302-451-5200

3 2

Re: Ping flooding (fwd)
by Daniel W. McRobb 10 Jul '96

10 Jul '96

> On Jul 9, 14:21, Curtis Villamizar <curtis(a)ans.net> wrote: > > The NSS routers allow us to do statistical sampling continuously and > > the occurance of a source address at an entry point where it does not > > usually enter can be detected and has in the past been used to > > followup these sort of attacks after the fact. Other routers are not > > capable of doing this but if the offense is repeated, successive > > monitoring can be set up until the source is isolated. > > > > We have requested the same sort of statistical sampling from Cisco and > > Bay (and BNR/NSC). It is a long ways back on the development schedule > > Maybe I'm missing something, but flow switching stats from Ciscos > should do exactly this: > > SrcIf SrcIPaddress DstIf DstIPaddress Pr DstP SrcP Pkts B/Pk Active > Se1/0 194.130.16.17 Se1/6 130.144.65.1 11 0035 0035 2 69 0.0 > Et0/2 193.122.198.1 Se1/1 128.218.14.87 06 0050 0FA3 2 40 0.0 > Se1/5 130.144.65.1 Se1/0 194.130.16.17 11 0035 0035 2 69 0.0 > Se1/1 153.36.40.52 Et0/1 193.74.242.1 06 0413 0050 4 44 9.6 > Se1/5 194.178.24.22 Se1/7 146.228.10.11 06 0407 0050 124 40 207.6 > Se1/7 146.228.10.11 Se1/6 194.178.24.22 06 0050 0405 648 550 673.4 > Se1/5 194.165.95.69 Se1/0 205.216.146.69 06 0430 0050 5 164 6.2 > > etc, etc. Dump, then grep. The sampling Curtis mentioned on the NSS routers is a bit different. For one, it doesn't really impact forwarding performance (and hopefully, if/when implemented on the Bay, will not impact forwarding performance). Secondly, there's a lot more information available if you need it (packet headers or even payload). But what we're specifically looking for in terms of continuous sampling (such as that we do on the NSS routers) is a net matrix... if you changed the Cisco flow switching stuff to use network numbers (and mask), you'd have something very much like what we're looking for in terms of continuous sampling. From there we build AS matrices, etc. The other thing about the flow-switching data that's different than the NSS (and probably what we'll get from Bay) is that there aren't really any nice ways of retrieving/storing the data automatically. As Curtis pointed out, the data is mostly used for capacity planning and other network architecture issues. I have not receieved a request to track down a cracker with this type of data in a very long time. Maybe I've just been lucky. :-) Daniel ~~~~~~

3 2

Re: Ping flooding (fwd)
by Daniel W. McRobb 10 Jul '96

10 Jul '96

> > 'tis also very important to know the route taken for a net when > > analyzing the data. Discrepancies here can be huge and completely > > invalidate any conclusions you might make about how much traffic is > > traversing a given path. > > For this we like combined interface stats; AS-based traffic matrices > are (from our point of view) mostly useful for end-to-end measurements. Forgot to answer this one point... Combined interface stats aren't very useful for a scenario we commonly entertain: what will happen if I move Big Customer A's ingress to ANSnet from ANS border router X to ANS border router Y? I need to know who they're talking to in order to answer that question. Typically large portions of the ANS mesh will be affected, and I need to know how. Daniel ~~~~~~

1 0

Re: Ping flooding (fwd)
by Daniel W. McRobb 10 Jul '96

10 Jul '96

> On Jul 9, 17:57, "Daniel W. McRobb" <dwm(a)ans.net> wrote: > > The sampling Curtis mentioned on the NSS routers is a bit different. > > For one, it doesn't really impact forwarding performance (and hopefully, > > if/when implemented on the Bay, will not impact forwarding performance). > > Not to pick nits, but what I quoted was a cache snapshot; caches > don't impact performance under normal circumstances, though their > construction may do so. When I asked Cisco about this (a while ago), they said flow-switching incurred a 20% overhead (which someone there called "minimal", which I didn't agree with). > > But what we're specifically looking for in terms of continuous sampling > > (such as that we do on the NSS routers) is a net matrix... if you > > changed the Cisco flow switching stuff to use network numbers (and > > mask), you'd have something very much like what we're looking for in > > terms of continuous sampling. From there we build AS matrices, etc. > > Yes ... but you shouldn't need anything special for that. We have > been doing the same for a long time, using regular IP accounting on > the edge routers, which is then summarised over a full routing table. > The only discrepancies that occur are if changes in routing occur > between the time of accounting and processing, but this tends not to > be a problem. 'tis very unwise to run IP accounting on a very busy router. We wouldn't dare turn it on for a core router w/ 40,000 routes. Some of our customers who have done so on border routers immediately turn arond and complain about performance problems. :-( And we're not talking about access products either. 'tis also very important to know the route taken for a net when analyzing the data. Discrepancies here can be huge and completely invalidate any conclusions you might make about how much traffic is traversing a given path. This is particularly true for busy end routers (like NAP peers) and core routers. > > The other thing about the flow-switching data that's different than the > > NSS (and probably what we'll get from Bay) is that there aren't really > > any nice ways of retrieving/storing the data automatically. > > This used to be true, but "probably" isn't true any longer. Maybe, maybe not. I can't imagine trying to get flow stat data from a router w/ 40,000 routes and (probably) a monstrous flow table via UDP. A lot of this boils down to granularity... for continuous collection on busy routers for network architecture purposes, host-level granularity is frivolous and overly consumptive of many resources. I also prefer measurements that won't potentially get dropped in transit (throwing a big unknown into confidence level of data). Is there an efficient means of retrieving flow data from a Cisco and not potentially dropping a bunch of it on the floor? Daniel ~~~~~~

2 1

Re: Ping flooding (fwd)
by Doug Stanfield 10 Jul '96

10 Jul '96

At 06:43 AM 7/9/96 +0000, you wrote: >Michael Dillon writes: >> >> On Tue, 9 Jul 1996, Daniel W. McRobb wrote: >> >> > There will likely never be a means for a single NSP to track down the >> > real source of spoofed packets using IPv4. Service providers won't be >> > letting other service providers track spoofed packets through their >> > network. >> >> Why not? Don't telcos do this? >> Or if your answer is that telcos only do it for the police and not for >> each other, then my question would be why can't we form an Internet >> equivalent, maybe affiliated with something like CERT, that can make these >> requests and with whom NSP's would cooperate. > >What sort of incentive or penalty do you think would enable this >cooperation? > > Nevin > These garbage packets are flooding through somebodies network to get to the target. I'd think it would benefit the intervening operators to eliminate traffic tthat is harmful, probably illegal, and wasting the resource they sell, bandwidth. Doug Stanfield "The significant problems we face cannot be solved Oceanic Cable at the same level of thinking we were at when we Project Engineer created them." - Albert Einstein dougs(a)oceanic.com (808) 625-8455 fax (808) 625-5888

2 1