- NANOG - lists.nanog.org

Re: Routing flaps, was Re: Ping flooding
by Justin W. Newton 12 Jul '96

12 Jul '96

At 10:37 AM 7/12/96 -0400, Paul Ferguson wrote: >At 07:40 AM 7/12/96 -0600, Forrest W. Christian wrote: > >> >>However, the world never needs to see this - it's an internal routing >>issue and when the internal route flaps it shouldn't affect the external >>routes. >> >>So in summary: >> >>1) Never flap your routes in public. What you do in your own home is >>your own business. >> > >I'm sorry, but I'm afraid that you're missing the point here. > >In most larger ISP backbones, the behavior of their IGP is indeed >visible to the public, since in most instances, most of the Internet >traffic relies on the stability of these interior (an esoteric term) >networks. Therefore, whether interior or exterior flap is really of >no relevance in this context. I think the real difference is this. If my routes are only seen at one place then it makes sense for me to announce them whether or not I can actually get to them. If, however, it is possible to get to parts of my network from one exchange, but not another then the routes should only be announced where they can actually be gotten to from (obviously). I.e. if I am a small to mid-sized provider who has multiple t-1's to the net to different providers, and then those t's come into the same physical network (either the same router or one's that are topologically adjacent to one another) chances are good that if one of those routers can't get to part of my network the other probably can't either. In this instance it makes sense for me to null0 the routes and do my part to not affect instability (only routes which originate in my AS though, if I have any multihomed downstreams then they would not want me announcing their routes when I can't get to them). If, however, I have a network which has multiple egresses to the internet at large it is very possible for one part of my network not to be able to reach certain sites while other parts can. I would then want to "announce my instability" so that I am only announcing networks where they can actually be reached from. Justin "Long babbling posts R us" Newton Justin Newton Internet Architect Erol's Internet Services

1 0

Re: Ping flooding
by Vern Paxson 12 Jul '96

12 Jul '96

> Maybe in a perfect world, but given that all ISPs are not created equal, it > is usually the case that the two paths don't have the same latency and packet > loss characteristics. That's the point I'm trying to make. In the real world these don't actually matter, in terms of meaning that asymmetric routing itself presents a TCP performance problem. The two directions of a symmetric path can also have different packet loss characteristics, due to different levels of congestion. So this is not a problem specific to asymmetric routing. (It's also usually not a problem for bulk-transfer TCP because the acks take so little bandwidth compared to the data, and loss of an ack is not as serious as loss of a data packet because acks are cumulative.) That the two directions of an asymmetric path have different latencies doesn't in most ways matter either. TCP's performance is determined by the RTT - whether it can fill the pipe, whether it times out and retransmits correctly - and not by the unidirectional prop time. The only major real-world application I came across that has trouble with asymmetric routing is NTP. It has robust algorithms to avoid being fooled by asymmetric paths, but if two NTP communities are only connected by an asymmetric path, then they will indeed keep incorrect time with respect to one another (assuming they're not separately synchronized, which of course is how you solve this problem in practice). Asymmetric routing is also a network measurement and troubleshooting pain - I don't mean to discount that. Vern

1 0

Re: Ping flooding
by Paul Ferguson 12 Jul '96

12 Jul '96

At 08:53 PM 7/11/96 -0500, Alan Hannan wrote: > Vadim, > >] 2) Don't Do Any Dynamic Routing Where Only One Path Exists. > > Certainly I would not agree with this rule. > > If I have a tail router that is down, I do not want to send > traffic to him, when he is not there to receive it. Rather, I > would want my intermediate router to reject it right off. > Furthermore, I do not want to extend nondynamic notification in my > network. Alan, 'Tis better to black-hole packets than to flap. - paul

3 2

Re: Ping flooding
by avg＠ncube.com 12 Jul '96

12 Jul '96

>> That's once again a matter of defaults -- routers should _by >> default_ discard all packets from interfaces which they won't >> use for forwarding those packets back. >This is a sweeping statement. Note that _by default_ implies that there's a way to override it. Now, in practically all LANs and corporate networks paths are symmetrical unless something is misconfigured. Forceful prevention of asymmetrical paths is the Good Thing, then -- that allows to identify problems before something crashes. All single-homed customers of ISPs also have tail-link part of their paths symmetrical; and forced rejection of wrong source addresses brings no harm and allows to prevent a lot of damage caused by malicious attacks or simply misconfigured machines. Backbones use asymmetrical routing a lot, they also have a skilled personnel (well, usually) who would know how to disable the backward-path checks when necessary. > 1. Prevention of IP spoofing. Yes. > 2. Prevention of asymmetric routing. Not. Backbones have to have asymmetry, in general case, if you want to compress routing information. > 3. Better TCP windowing and better performance when all > packets follow the same path. Not really. TCP mostly cares about round-trip, not timing of forward and backward paths. >However, are the benefits worth the overhead? That's not significant, as such filtering is done at tail-link routers, which are not overloaded (generally). >What are the >effects on network redundancy? Does limiting ability to do hard-to-trace denial of service attacks count as improving network redundancy? >What other issues am I overlooking? Inertia. Even obviously Good Things like doing persistent static routing by default aren't getting deployed, although the current cisco's default of "floating" static routes is the #1 contributor to route flap. Does any major ISP override the default by adding "backup" routes to Null0 in a systematic way? I think not. In the same vein, how many people don't forget to add "no ip proxy-arp" to their interface definitions? I know quite a few who were bitten by forgetting about that default. Or take interfaces being active (and listening!) to routing updates by default, or silly recommendations in manuals, or ... (you continue, the list is nearly endless). The Rules of Safe (and Reliable) Networking are: 1) Disable All Features You Don't Need. 2) Don't Do Any Dynamic Routing Where Only One Path Exists. 3) Do Not Allow Packets To Go To Where They Are Not Supposed To Go To And Come From Where They Aren't Supposed To Come From. (The short form of that maxim is "Filter!".) 4) Don't Accept Suspicious Routes And Watch What You're Announcing. (The short form of that maxim is also "Filter!"). 5) K.I.S.S. 6) Don't Trust Anyone. Unfortunately the feature-happy router vendors do everything to make following those rules very hard by enabling virtually everything by default and so encouraging users to shoot themselves in the feet. I.e. to _disable_ stuff you need to be an expert and know what it does! Most simply don't bother. --vadim

5 4

Re: Ping flooding
by Vern Paxson 12 Jul '96

12 Jul '96

> Really, consider a link where the path carrying most of the data has > a time of N and the path carrying the ack has a path 2N or N + 1/2N > You are computing an average which seems like it could be skewed based > on the path the ack takes. This doesn't matter. The key notions for TCP are (1) what's the time scale for feedback, and (2) what's the pipe size (= bandwidth-delay product). Both of these are determined by the RTT and not the one-way prop time. > Does anyone know of any papers on the effect of asymetric paths on TCP > performance? Well, I thought about this quite a bit for my end-to-end routing study (ftp://ftp.ee.lbl.gov/papers/routing.SIGCOMM.ps.Z) I really wanted to come up with some reason why asymmetric routing has serious implications for TCP performance, but wasn't able to. I guess this is a good thing, since 50% of the paths in my study were asymmetric in terms of visiting at least one different city in the two directions. 30% visited at least one different AS. Vern

2 1

Re: Routing flaps, was Re: Ping flooding
by avg＠ncube.com 12 Jul '96

12 Jul '96

>I doubt that a large provider such as Sprint or MCI or Agis or any other >major internet backbone is going to be able to hide much of their >internal route flapping, as their internal routes ARE their external routes. Well, not. BGP (exterior) table is cleanly separated from IGP table in modern backbones, so bouncing interior circuits do not generally produce routing flap visible outside (if it is not severe enough to cause partitioning). The links to exchange points and customer-access routers is another kettle of fish altogether... There are more than few ways to induce interesting instabilities in the iBGP-hack based backbones, some going as far as to produce self-feeding route oscillations. However, fascistic filtering on both IGP and BGP routing information (so they won't interact in unexpected ways) helps. >My original note was intended to reply to the specific case of using >dynamic routing to distribute routes which could just as easily be done >with static routes - I.E. where there is one and only one path to the >destination. In this case, it is sometimes desirable and necessary to >run some sort of interior routing protocol, but it is not desirable that >flaps caused by a break along the one and only one path be propogated to >the outside world, but instead the packets should be null0'd at the point >of path convergence. Yes, you can eliminate global impact of interior flaps this way; but believe me, to design a truely reliable and booletproof interior routing takes no less efforts than just place all the static routes. If you don't think so you probably never had people starting gated on their workstations just for the heck of it :) --vadim

1 0

Re: Ping flooding
by avg＠ncube.com 12 Jul '96

12 Jul '96

You need to have _very_ different path parameters to cause TCP noticeable problems (and then if one of the paths is that bad you're screwed up anyway). Most of asymmetrical paths are quite "symmetrical" parameter-wise. --vadim Vern, > (ftp://ftp.ee.lbl.gov/papers/routing.SIGCOMM.ps.Z) I really wanted to > come up with some reason why asymmetric routing has serious implications > for TCP performance, but wasn't able to. I guess this is a good thing, Maybe in a perfect world, but given that all ISPs are not created equal, it is usually the case that the two paths don't have the same latency and packet loss characteristics. Jeff

1 0

Re: On another topic ... :-)
by Sean Donelan 11 Jul '96

11 Jul '96

>>PS, Mr. Metcalfe. Reports for the old reliable telephone system show >>over the course of a year, a daily average of 30,000 US telephone customers >>are out of service for five hours. > >Source please? The ATIS network reliability council steering committee report "Keeping the Network Alive And Well: Solving the problem of cable dig-ups." <http://www.atis.org/atis.nrsc/whatsne2.htm>: >Facility outages continue to be the leading contributor to outages in the >Public Switched Network. Approximately 50% of FCC-reportable service outages >and their impact have been caused by facility outages. Roughly 33 million >customers over the past three years or 30,000 customers a day have lost >access to the Public Switched network for an average of 5 hours because of >these problems. The ATIS NRSC Facilities Solution Team was chartered to >determine the causes of these facility outages and to recommend ways to >reduce their number and impact. A copy of the Facilities Solution Team report entitled "Keeping the Network Alive and Well: Solving the Problem of Cable Dig-Ups" can be obtained by contacting ATIS on (202) 628-6380. -- Sean Donelan, Data Research Associates, Inc, St. Louis, MO Affiliation given for identification not representation

2 1

Re: Ping flooding
by avg＠ncube.com 11 Jul '96

11 Jul '96

Sorry, i was unclear. It should read "forwarding return packets back to the original source". In other words -- assume that forwards and backwards paths are both crossing the same interface. That assumption is practically always valid for tail circuits. --vadim > That's once again a matter of defaults -- routers should _by > default_ discard all packets from interfaces which they won't > use for forwarding those packets back. Why? -- Jerry Anderson jerry(a)gi.net Principal Engineer (402) 436-3030 Implementation & Consulting http://www.gi.net/jerry Global Internet Network Services (formerly known as MIDnet)

2 1

Re: Ping flooding
by avg＠ncube.com 11 Jul '96

11 Jul '96

If you want to run large network you'd better think about tools to configure routers and maintain configurations. In my practice more than 70% of non-link related problems in corporate networks (and i've seen _lots_ of them) is due to unnecessary dynamic routing. Just watch some luser turing on RIP on his itty bitty terminal server or someting and causing the entire corporation to go banana. In public Internet, dynamic routing over tail links is downright antisocial. I hope i don't need to explain _that_. As an addendum to my remarks -- if you think you have complete knowledge of how you dynamically routed network behaves in case of various failures you're deceiving yourself. The distributed algorithms too often behave counterintuitively, and implementations often have subtle bugs. --vadim

1 0