- Test - lists.nanog.org

Re: problem at mae-west tonight?
by Rob Liebschutz 15 Jul '96

15 Jul '96

Actually I've seen the arp problem as well. What arp entry do you have for them? We have: mae-west.netcom.net (198.32.136.15) at 0:0:c:61:b:0 If yours is different, you could try deleting it and if it still comes back wrong, you could try temporarily adding a static entry. Are either you or netcom connected through a netedge? > ps. At this point, we still can't see Netcom... problem has lasted for 1 day, > 8 hours. From my interface (198.32.136.18) I cannot ping Netcom at > 198.32.136.15, but I can ping Alternet at 198.32.136.42. That points to > a layer 2 discontinuity. > >

1 0

Re: problem at mae-west tonight?
by Rob Liebschutz 15 Jul '96

15 Jul '96

Yes, this type of topology failure is possible too. I can recall times when there were problems on the links between Ames and MFS. Yes, the link is 2 OC3s. > > Robert, > > It's my understanding that MAE-W is composed of two facilities, > one at MFS, and the second at NASA-AMES. > > It's my understanding that while a virtual medium is created, they > actually reside on two separate gigaswitches. I seem to recall 1 > or 2 OC3s connecting the two..... > > Given this, it is, then, possible that the route server is located > on segment A, while customers x,y, and z are located on segment B. > > If you are on segment A, and try to follow the RSes advice to talk > to x, y, and z, you are depending on the links connecting the two > segments. > > However, I may be incorrect. > > -alan

1 0

Re: problem at mae-west tonight?
by matthew＠scruz.net 15 Jul '96

15 Jul '96

Original message <199607150340.UAA00784(a)elite.exodus.net> From: Robert Bowman <rob(a)elite.exodus.net> Date: Jul 14, 20:40 Subject: Re: problem at mae-west tonight? > > We experienced the same thing with Netcom. Currently we are peered with over > 40 netwroks through the RS, but I have only had this problem with Netcom. > > Is it really a next-hop problem or a Netcom internal problem? Last time > this happened, about 2 weeks ago, they cleared their RA session and did > some other things and everything came up fine. I did not get details from > the routing folks over there. > > I don't quite see how and where the layer 2 topology comes into play here. > Netcom should simply be seeing routes (through the RS) that state your MW > IP address and the routes advertised from it. Is there some reason that your MW > IP would be unreachable by Netcom? I am confused as to why this would ever > happen in the MW scenario. Now the PB-NAP is a different story with the > non-fully meshed scenario. > > Please explain what you mean Matt. Sure. Netcom changed out an FDDI card. There is a bad ARP entry or bridge table entry somewhere, which causes packets transmitted by Netcom towards me to be black-holed. MFS is, as far as I know, _still_ investigating this. However we can see the route server as can Netcom. The route server states our MW IP address, and so they transmit packets towards us using that, and yet the underlying layer 2 fabric is discarding those before they get to us. Yes, IF EVERYTHING WAS WORKING PROPERLY it is fully meshed... but things don't always work properly. What I know is that when I stopped believing the route server's routes to Netcom, there was still a discontinuity, but when I stopped telling the route server my routes (and it then stopped telling Netcom), Netcom switched to sending my packets to our transit provider, and my Netcom connectivity came back. That required both manual intervention AND tearing down the whole session with the route server (I can't easily get the route server to stop telling Netcom about me while still telling everyone else about me without editing my policy and waiting for it to appear on the RS) -matthew kaufman matthew(a)scruz.net ps. At this point, we still can't see Netcom... problem has lasted for 1 day, 8 hours. From my interface (198.32.136.18) I cannot ping Netcom at 198.32.136.15, but I can ping Alternet at 198.32.136.42. That points to a layer 2 discontinuity.

1 0

Re: problem at mae-west tonight?
by Rob Liebschutz 15 Jul '96

15 Jul '96

On a regular basis, MAE West has failures that cause only some of the other routers to become unreachable. I believe all of the instances that I have heard of are related to problems with the Netedges which show up mostly under high load. I've heard various people call it the "Sleeping Interface" problem. It usually goes away when you reset the netedges, but under heavy load it can come back quickly. We have seen it extensively, when our 10mb connection became heavily loaded. Just yesterday, we upgraded to a DS3, so were hoping not to see it for a while. I know AGIS has also had this problem and I believe Best has seen it as well. Maybe you haven't seen it because you have a colocated router at MAE-WEST. Rob > We experienced the same thing with Netcom. Currently we are peered with over > 40 netwroks through the RS, but I have only had this problem with Netcom. > > Is it really a next-hop problem or a Netcom internal problem? Last time > this happened, about 2 weeks ago, they cleared their RA session and did > some other things and everything came up fine. I did not get details from > the routing folks over there. > > I don't quite see how and where the layer 2 topology comes into play here. > Netcom should simply be seeing routes (through the RS) that state your MW > IP address and the routes advertised from it. Is there some reason that your MW > IP would be unreachable by Netcom? I am confused as to why this would ever > happen in the MW scenario. Now the PB-NAP is a different story with the > non-fully meshed scenario. > > Please explain what you mean Matt. > > Rob > Exodus Communications Inc. > > > > > The problem I have with the route server this evening is that I announce > > > my routes to the route server, and my policy configuration in the route server > > > reflects that I peer with Netcom, and so the route server tells Netcom how > > > to reach me. Unfortunately, packets leaving Netcom headed to me at layer 2 > > > are going into a black hole. To fix this, I've had to dump my peering with > > > the route server entirely, so that Netcom is only seeing my routes from AGIS > > > (our transit provider) and not from the route server. Ugh. My fears about > > > the route server not knowing the status of the layer 2 topology have come true, > > > and there's no way to fix this that doesn't involve manual intervention. > > > > > > -matthew kaufman > > > matthew(a)scruz.net > > > > > > > > > > Well, I run gated on a BSDI box for the Hooked MAE West router. I'm > > thinking about implementing a "pingnouse INTERVAL" option on the > > peer/group commands in gated, so it will periodically ping next hops > > received from the route servers and set the nouse bit if the nexthop > > is unreachable. Any better ideas? > > > > It would be nice to come up with a good mechanism for doing 3rd party > > keepalives that cisco and other router vendors would be willing to > > implement. > > > > Rob > > > >

1 0

Re: Re: problem at mae-west tonight?
by eautheme＠BayNetworks.com 14 Jul '96

14 Jul '96

Rob, In the OSPF community,we faced this same issue a few years back -- the issue being that all members of a single subnet were not reachable either by design or by error and fooling the listeners of the Designated Router (like the route server) into thinking that the nexthops were really reachable. We felt the pains of this especially over virtual circuit networks (Frame Relay, SMDS, and ATM). Temporary workarounds were created. The two better solutions were: 1) Using recommendations in RFC1586,"Guidelines for Running OSPF Over Frame Relay Networks". 2) Using Point-MultiPoint (P-MP) OSPF which is in the latest Internet-Draft of the revision to the RFC1583 (V2 OSPF Spec.). I prefer the P-MP solution because it is an automatic mechanism requiring very little configuration on a network administrators part and keeps the number of routing hops to a minimum. I vote for P-MP as a "Standard" solution for BGP-4 over discontiguous networks. I believe this concept was proposed on this mailing list a few months back. Now that ATM VCs are being used at NAPs with route servers, you'll see more and more of these problems arising...especially at NAPs where there isn't much coordination and cooperation in building and maintaining these high speed, virtual connections. Eural Authement Bay Networks Professional Services ******** > The problem I have with the route server this evening is that I announce > my routes to the route server, and my policy configuration in the route server > reflects that I peer with Netcom, and so the route server tells Netcom how > to reach me. Unfortunately, packets leaving Netcom headed to me at layer 2 > are going into a black hole. To fix this, I've had to dump my peering with > the route server entirely, so that Netcom is only seeing my routes from AGIS > (our transit provider) and not from the route server. Ugh. My fears about > the route server not knowing the status of the layer 2 topology have come true, > and there's no way to fix this that doesn't involve manual intervention. > > -matthew kaufman > matthew(a)scruz.net > > Well, I run gated on a BSDI box for the Hooked MAE West router. I'm thinking about implementing a "pingnouse INTERVAL" option on the peer/group commands in gated, so it will periodically ping next hops received from the route servers and set the nouse bit if the nexthop is unreachable. Any better ideas? It would be nice to come up with a good mechanism for doing 3rd party keepalives that cisco and other router vendors would be willing to implement. Rob

1 0

Re: Routing flaps, was Re: Ping flooding
by Paul Ferguson 12 Jul '96

12 Jul '96

At 07:40 AM 7/12/96 -0600, Forrest W. Christian wrote: > >However, the world never needs to see this - it's an internal routing >issue and when the internal route flaps it shouldn't affect the external >routes. > >So in summary: > >1) Never flap your routes in public. What you do in your own home is >your own business. > I'm sorry, but I'm afraid that you're missing the point here. In most larger ISP backbones, the behavior of their IGP is indeed visible to the public, since in most instances, most of the Internet traffic relies on the stability of these interior (an esoteric term) networks. Therefore, whether interior or exterior flap is really of no relevance in this context. In smaller ISPs, where not a great deal of public traffic is transiting their (interior) backbone, granted, it is of lesser importance and visibility. - paul

3 3

Re: Routing flaps, was Re: Ping flooding
by Justin W. Newton 12 Jul '96

12 Jul '96

At 10:37 AM 7/12/96 -0400, Paul Ferguson wrote: >At 07:40 AM 7/12/96 -0600, Forrest W. Christian wrote: > >> >>However, the world never needs to see this - it's an internal routing >>issue and when the internal route flaps it shouldn't affect the external >>routes. >> >>So in summary: >> >>1) Never flap your routes in public. What you do in your own home is >>your own business. >> > >I'm sorry, but I'm afraid that you're missing the point here. > >In most larger ISP backbones, the behavior of their IGP is indeed >visible to the public, since in most instances, most of the Internet >traffic relies on the stability of these interior (an esoteric term) >networks. Therefore, whether interior or exterior flap is really of >no relevance in this context. I think the real difference is this. If my routes are only seen at one place then it makes sense for me to announce them whether or not I can actually get to them. If, however, it is possible to get to parts of my network from one exchange, but not another then the routes should only be announced where they can actually be gotten to from (obviously). I.e. if I am a small to mid-sized provider who has multiple t-1's to the net to different providers, and then those t's come into the same physical network (either the same router or one's that are topologically adjacent to one another) chances are good that if one of those routers can't get to part of my network the other probably can't either. In this instance it makes sense for me to null0 the routes and do my part to not affect instability (only routes which originate in my AS though, if I have any multihomed downstreams then they would not want me announcing their routes when I can't get to them). If, however, I have a network which has multiple egresses to the internet at large it is very possible for one part of my network not to be able to reach certain sites while other parts can. I would then want to "announce my instability" so that I am only announcing networks where they can actually be reached from. Justin "Long babbling posts R us" Newton Justin Newton Internet Architect Erol's Internet Services

1 0

Re: Ping flooding
by Vern Paxson 12 Jul '96

12 Jul '96

> Maybe in a perfect world, but given that all ISPs are not created equal, it > is usually the case that the two paths don't have the same latency and packet > loss characteristics. That's the point I'm trying to make. In the real world these don't actually matter, in terms of meaning that asymmetric routing itself presents a TCP performance problem. The two directions of a symmetric path can also have different packet loss characteristics, due to different levels of congestion. So this is not a problem specific to asymmetric routing. (It's also usually not a problem for bulk-transfer TCP because the acks take so little bandwidth compared to the data, and loss of an ack is not as serious as loss of a data packet because acks are cumulative.) That the two directions of an asymmetric path have different latencies doesn't in most ways matter either. TCP's performance is determined by the RTT - whether it can fill the pipe, whether it times out and retransmits correctly - and not by the unidirectional prop time. The only major real-world application I came across that has trouble with asymmetric routing is NTP. It has robust algorithms to avoid being fooled by asymmetric paths, but if two NTP communities are only connected by an asymmetric path, then they will indeed keep incorrect time with respect to one another (assuming they're not separately synchronized, which of course is how you solve this problem in practice). Asymmetric routing is also a network measurement and troubleshooting pain - I don't mean to discount that. Vern

1 0

Re: Ping flooding
by Paul Ferguson 12 Jul '96

12 Jul '96

At 08:53 PM 7/11/96 -0500, Alan Hannan wrote: > Vadim, > >] 2) Don't Do Any Dynamic Routing Where Only One Path Exists. > > Certainly I would not agree with this rule. > > If I have a tail router that is down, I do not want to send > traffic to him, when he is not there to receive it. Rather, I > would want my intermediate router to reject it right off. > Furthermore, I do not want to extend nondynamic notification in my > network. Alan, 'Tis better to black-hole packets than to flap. - paul

3 2

Re: Ping flooding
by avg＠ncube.com 12 Jul '96

12 Jul '96

>> That's once again a matter of defaults -- routers should _by >> default_ discard all packets from interfaces which they won't >> use for forwarding those packets back. >This is a sweeping statement. Note that _by default_ implies that there's a way to override it. Now, in practically all LANs and corporate networks paths are symmetrical unless something is misconfigured. Forceful prevention of asymmetrical paths is the Good Thing, then -- that allows to identify problems before something crashes. All single-homed customers of ISPs also have tail-link part of their paths symmetrical; and forced rejection of wrong source addresses brings no harm and allows to prevent a lot of damage caused by malicious attacks or simply misconfigured machines. Backbones use asymmetrical routing a lot, they also have a skilled personnel (well, usually) who would know how to disable the backward-path checks when necessary. > 1. Prevention of IP spoofing. Yes. > 2. Prevention of asymmetric routing. Not. Backbones have to have asymmetry, in general case, if you want to compress routing information. > 3. Better TCP windowing and better performance when all > packets follow the same path. Not really. TCP mostly cares about round-trip, not timing of forward and backward paths. >However, are the benefits worth the overhead? That's not significant, as such filtering is done at tail-link routers, which are not overloaded (generally). >What are the >effects on network redundancy? Does limiting ability to do hard-to-trace denial of service attacks count as improving network redundancy? >What other issues am I overlooking? Inertia. Even obviously Good Things like doing persistent static routing by default aren't getting deployed, although the current cisco's default of "floating" static routes is the #1 contributor to route flap. Does any major ISP override the default by adding "backup" routes to Null0 in a systematic way? I think not. In the same vein, how many people don't forget to add "no ip proxy-arp" to their interface definitions? I know quite a few who were bitten by forgetting about that default. Or take interfaces being active (and listening!) to routing updates by default, or silly recommendations in manuals, or ... (you continue, the list is nearly endless). The Rules of Safe (and Reliable) Networking are: 1) Disable All Features You Don't Need. 2) Don't Do Any Dynamic Routing Where Only One Path Exists. 3) Do Not Allow Packets To Go To Where They Are Not Supposed To Go To And Come From Where They Aren't Supposed To Come From. (The short form of that maxim is "Filter!".) 4) Don't Accept Suspicious Routes And Watch What You're Announcing. (The short form of that maxim is also "Filter!"). 5) K.I.S.S. 6) Don't Trust Anyone. Unfortunately the feature-happy router vendors do everything to make following those rules very hard by enabling virtually everything by default and so encouraging users to shoot themselves in the feet. I.e. to _disable_ stuff you need to be an expert and know what it does! Most simply don't bother. --vadim

5 4