- Test - lists.nanog.org

RE: InterNAP?
by Spolidoro, Guilherme 01 Nov '00

01 Nov '00

Since they don't own the backbone, their SLAs apply only to their PNAPs, i.e. they cannot guarantee performance, packet loss, delay, outages, etc over somebody else's backbone (e.g. Sprint, UUNet, AT&T, etc). Many large enterprises are leveraging on the Internet to deploy VPN connectivity between company sites as a replacement for Frame Relay or ATM PVCs. In such cases, it's very important to make sure that the service comes with the right SLAs. -----Original Message----- From: Tom Schmidt [mailto:tsch52@hotmail.com] Sent: Monday, October 30, 2000 5:42 PM To: nanog(a)merit.edu Subject: InterNAP? I need your opinion on InterNAP. I am currently have a DS3 to my current provider and want to add an additional DS3 for redundancy to the same location. We plan to run BGP4 on both connections. InterNap has some technology to avoid congested peering points. Does this technology actually work? Isn't it impossible to avoid these peering points? What are your experiences with InterNAP? _________________________________________________________________________ Get Your Private, Free E-mail from MSN Hotmail at http://www.hotmail.com. Share information about yourself, create your own public profile at http://profiles.msn.com.

3 2

AADS
by Mark Borchers 31 Oct '00

31 Oct '00

Anybody know anything about possible switch problems at AADS? I have multiple, but not all, pvc's down, but haven't been able to reach any Ameritech folks yet for information. Thanks, -- Mark Borchers Splitrock Services IP Engineering 9012 New Trails Dr. mborchers(a)splitrock.net The Woodlands, TX 77381 (281) 465-1931 http://www.splitrock.net/

1 0

Re: InterNAP
by Neil J. McRae 31 Oct '00

31 Oct '00

Mike Johnson writes: > I'm not a customer, but I spoke with a salesdriod there. All they do > is go in, set up their own geographicly disperse private peering points > and link them together with their private backbone. They set these > things up by buying service from the other large backbone folk. The > idea being that, in general, if they can get their traffic onto the same > backbone that the end user is connected to, then it's avoided the > public peering points. It's a good theory. Dunno if it works. In my opinion its not a peering point. Its an aggregation point of several carriers into one AS. There is no magic to this if you have a direct link to 5 of the biggest providers [1, 701, 1239 etc] then you have a good control over how your traffic flows on the outbound, a few lines of perl, rping and a BGP table dump and you can probably, to a limited extent have some effect on the path your traffic takes. I suspect that trying to implement the same architecture outside of the United States will be substantially harder though. Also at some point in time, one of their access nodes is bound to have a subsAtantial ammounts of traffic and then they would have to link some of the other access nodes to this one to avoid other congestion points. One thing that is good about the idea, is that they have a customer contract with their transit suppliers with some fairly useful SLA's contained within, which should mean they have some control over the quality of those providers. Regards, Neil. -- Neil J. McRae - Alive and Kicking D O M I N O . O R G neil(a)DOMINO.ORG Regards, Neil.

1 0

RE: InterNAP?
by Rishi Singh 30 Oct '00

30 Oct '00

We have two DS3s to them and we haven't been too happy. Outage are common from their router-->up and local loops as well. We have all monitoring service turned on with them and have never gotten a call, page or email during one of our outage and we have five people on the list. We're leaving them as soon as we get our other circuits. -----Original Message----- From: Tom Schmidt [mailto:tsch52@hotmail.com] Sent: Monday, October 30, 2000 5:42 PM To: nanog(a)merit.edu Subject: InterNAP? I need your opinion on InterNAP. I am currently have a DS3 to my current provider and want to add an additional DS3 for redundancy to the same location. We plan to run BGP4 on both connections. InterNap has some technology to avoid congested peering points. Does this technology actually work? Isn't it impossible to avoid these peering points? What are your experiences with InterNAP? _________________________________________________________________________ Get Your Private, Free E-mail from MSN Hotmail at http://www.hotmail.com. Share information about yourself, create your own public profile at http://profiles.msn.com.

1 0

RE: IS-IS protocol implementation problem
by rdobbins＠netmore.net 30 Oct '00

30 Oct '00

You're right - I'm looking for fault here in my own equipment, whereas a screwed-up IGP on their end could well have caused all the problems I was experiencing. Occam's Razor, of course! Your point about the iBGP confederations is well-taken, by the way. ----------------------------------------------------------- Roland Dobbins <rdobbins(a)netmore.net> // 818.535.5024 voice -----Original Message----- From: smd(a)clock.org [mailto:smd@clock.org] Sent: Monday, October 30, 2000 11:11 AM To: rdobbins(a)netmore.net Subject: RE: IS-IS protocol implementation problem [not to list] | No, I'm a single-AS hosting provider, no confederation. UUNET has confederations; I was doing a public thinking exercise (trying to coopt smart people who still read the NANOG list, too) about the extent to which the insulation of iBGP in a confederated AS from the disappearance of iBGP peers for an otherwise full-mesh iBGP layout interacts with the exposure of multi-AS-one-IGP confederations to failures which cause the iBGP next-hops to disappear. If it seems a bit esoteric, don't worry, it is... | The more I think | about it, the more I'm convinced that CEF simply stopped working; all my | interfaces were active, and there were no apparent problems with my IGP, | which is OSPF. Right, UUNET was having the problems; you were just a victim of their internal routing being so broken that they couldn't make packets move to you reliably, even though their routers knew how to get to your network; likewise their routers kept telling you they knew how to get to all sorts of networks which in fact they couldn't reach. This problem appears consistent with an IGP problem inside UUNET, which is known to use/have-used confederated ASes. | I think that major BGP wigginess caused the CEF problem; thanks very much | for you insight, I definitely need to think about it some more. What makes you think it was a CEF problem? Sean.

1 0

RE: IS-IS protocol implementation problem
by rdobbins＠netmore.net 30 Oct '00

30 Oct '00

No, I'm a single-AS hosting provider, no confederation. The more I think about it, the more I'm convinced that CEF simply stopped working; all my interfaces were active, and there were no apparent problems with my IGP, which is OSPF. I think that major BGP wigginess caused the CEF problem; thanks very much for you insight, I definitely need to think about it some more. -----Original Message----- From: smd(a)clock.org [mailto:smd@clock.org] Sent: Monday, October 30, 2000 7:28 AM To: nanog(a)merit.edu; rdobbins(a)netmore.net Cc: neil(a)colt.net; sean(a)donelan.com Subject: RE: IS-IS protocol implementation problem | I had a bizarre event occur on Thursday night/Friday morning, and this is | likely the culprit. Some of your symptoms are consistent with a badly-broken sloshing IGP, notably the drop in traffic load and large numbers of dying TCPs passing through the afflicted network. This is two sides of the same coin: a destination in your network, learned through (e)BGP is mapped to a next-hop address (typically the interface across which you are talking (e)BGP)and propagated through their network via iBGP. The IGP is used so that each iBGP-talking router knows how to get to each next-hop address. A sloshing IGP will break connectivity between a given router and all the addresses associated with a broken next-hop. A hypothesis: for each afflicted router, the failure of one next-hop-address to be reachable will cause your ENTIRE network to be unreachable by sources relying upon traffic passing through that router. This may mean a sizeable proportion of their customer base simply could not reach you reliably enough to maintain a TCP connection in equilibrium, or at all. Frequent transition to slow-start due to loss/out-of-order-packets *and* a reduction in the overall number of TCP "mice", would severly reduce traffic. An interesting question, however, is why would their iBGP TCP connections appear to remain functional (you aren't losing eBGP routes) in this sort of mess? Did loopback addresses not come and go, but interface addresses did? (That would be interesting to consider in the face of possible aggregation of interface addresses into the IGP). Is there significant partitioning because of, for example, AS confederating, mitigatiing the problem by removing iBGP's need to know about distant loopback addresses, but not distant next-hop-addresses? We are lucky to have what could be a very interesting case study in routing scalability trade-offs. What a pity nothing like outage(a)sprint.net exists any more, where we might find useful information from the victim provider. :-( Sean.

1 0

RE: IS-IS protocol implementation problem
by smd＠clock.org 30 Oct '00

30 Oct '00

| I had a bizarre event occur on Thursday night/Friday morning, and this is | likely the culprit. Some of your symptoms are consistent with a badly-broken sloshing IGP, notably the drop in traffic load and large numbers of dying TCPs passing through the afflicted network. This is two sides of the same coin: a destination in your network, learned through (e)BGP is mapped to a next-hop address (typically the interface across which you are talking (e)BGP)and propagated through their network via iBGP. The IGP is used so that each iBGP-talking router knows how to get to each next-hop address. A sloshing IGP will break connectivity between a given router and all the addresses associated with a broken next-hop. A hypothesis: for each afflicted router, the failure of one next-hop-address to be reachable will cause your ENTIRE network to be unreachable by sources relying upon traffic passing through that router. This may mean a sizeable proportion of their customer base simply could not reach you reliably enough to maintain a TCP connection in equilibrium, or at all. Frequent transition to slow-start due to loss/out-of-order-packets *and* a reduction in the overall number of TCP "mice", would severly reduce traffic. An interesting question, however, is why would their iBGP TCP connections appear to remain functional (you aren't losing eBGP routes) in this sort of mess? Did loopback addresses not come and go, but interface addresses did? (That would be interesting to consider in the face of possible aggregation of interface addresses into the IGP). Is there significant partitioning because of, for example, AS confederating, mitigatiing the problem by removing iBGP's need to know about distant loopback addresses, but not distant next-hop-addresses? We are lucky to have what could be a very interesting case study in routing scalability trade-offs. What a pity nothing like outage(a)sprint.net exists any more, where we might find useful information from the victim provider. :-( Sean.

2 1

RE: IS-IS protocol implementation problem
by rdobbins＠netmore.net 30 Oct '00

30 Oct '00

I had a bizarre event occur on Thursday night/Friday morning, and this is likely the culprit. I peer with AS701. At approximately 11:15PM PDT or thereabouts (2:15AM EDT), the 7507 which provides my connectivity to uu.net went belly-up in a very strange way. The BGP session with 701 showed active, with full tables; existing TCP connections stayed up. However, it appeared that all new connections inbound from 701 were being dropped on the floor, and my outbound traffic with them dropped from 40mb/sec down to about 5kb/sec. The same router was also handling a secondary connection to pbi.net; because the BGP stayed active and in a supposedly functional state, traffic didn't get routed in that direction as it should've been. I had to reload the router to get it to function properly. Very odd. Nothing in the logs, etc. The router just essentially went on strike, and I've no idea why. I don't run IS-IS, needless to say, especially with a foreign AS. This particular 7507 was running an 11.3.x CC-train IOS, and hadn't had any of the ISO/CLNS family of protocols enabled, ever. This is a bit earlier than the timeframe Sean cited, but I don't think it was a coincidence, either. ----------------------------------------------------------- Roland Dobbins <rdobbins(a)netmore.net> // 818.535.5024 voice -----Original Message----- From: Neil J. McRae [mailto:neil@COLT.NET] Sent: Monday, October 30, 2000 1:28 AM To: sean(a)donelan.com Cc: smd(a)clock.org; nanog(a)merit.edu Subject: Re: IS-IS protocol implementation problem > At approximately 7:37am EDT on Friday, about 258 Cisco 12000's on UUNET's > primary backbone reloaded. This appeared to be isolated to routers > in ASN 701. It disrupted reachability to about 15% of the world-wide Internet > based on data from Matrix measurements. A contributing cause was a bad > IS-IS packet which confused certain IOS versions in the 12.0 IOS software > train. I haven't heard what the root cause was or what originated the > bad IS-IS packet. The Cisco bug id is CSCdr05779. Any provider running the > affected IOS version may be vulnerable depending on what the root cause > turns out to be. > > Although the bad IS-IS packet didn't propagate to other providers, several > other providers did report BGP resets and route flaps about the same time. If a large AS such as AS701 starts flapping I wouldn't be surprised if other ASes start seeing BGP resets and route-flaps. Could be that crud routing information was exchange when that chaos started [jeez 258 routers I'd hate to have been the on duty NOC guy on that morning :-)] Interestingly though we still see alot routes with bad AS-PATH information people should be setting more stringent configurations on the routes the learn and subsequentally pass on to avoid this. Regards, Neil.

1 0

Re: IS-IS protocol implementation problem
by Sean Donelan 30 Oct '00

30 Oct '00

On Sun, 29 October 2000, smd(a)clock.org wrote: > P.S.: any chance you can be a bit more concrete about what's happening? When I'm concrete, providers complain I'm picking on them, and getting them bad press. But since you asked.... At approximately 7:37am EDT on Friday, about 258 Cisco 12000's on UUNET's primary backbone reloaded. This appeared to be isolated to routers in ASN 701. It disrupted reachability to about 15% of the world-wide Internet based on data from Matrix measurements. A contributing cause was a bad IS-IS packet which confused certain IOS versions in the 12.0 IOS software train. I haven't heard what the root cause was or what originated the bad IS-IS packet. The Cisco bug id is CSCdr05779. Any provider running the affected IOS version may be vulnerable depending on what the root cause turns out to be. Although the bad IS-IS packet didn't propagate to other providers, several other providers did report BGP resets and route flaps about the same time.

2 1

Re: IS-IS protocol implementation problem
by smd＠clock.org 30 Oct '00

30 Oct '00

| Because IS-IS is an IGP protocol, it does not propagate between | providers. This is not the reason why it will not propagate between separate ASes. The "saving factor" here is that nobody really routes CLNS natively, and therefore, the maximum hop-count of a CLNS datagram is 1. It would be possible to cascade an IS-IS problem across multiple separate ASes in the unfortunate event that more than one AS treated a single LAN (e.g. an IX) or point-to-point link as an internal one across which IS-IS is run, with the same key. This kind of mutual poisoning between separate ASes happens with some regularity, amusingly often with RIP as the IGP. An IGP based on a natively routed protocol (including routed CLNS) widens the scope for inter-AS poisoning. This is why it is important to have good authentication in one's IGP. Unfortunately, *no* IGPs currently in wide use have any such thing. :-( For clarity, a separate AS is really short hand for, "a collection of routers participating in a common IGP instantiation"; there are cases where different ASes (in the BGP sense) share a common IGP. Also, "propagating between providers" seems to ignore the fact that there are single providers who have multiple IGP instantiations. Sean. P.S.: any chance you can be a bit more concrete about what's happening?

1 0