October 2000 - Test - lists.nanog.org

InterNAP?
by Tom Schmidt 02 Nov '00

02 Nov '00

I need your opinion on InterNAP. I am currently have a DS3 to my current provider and want to add an additional DS3 for redundancy to the same location. We plan to run BGP4 on both connections. InterNap has some technology to avoid congested peering points. Does this technology actually work? Isn't it impossible to avoid these peering points? What are your experiences with InterNAP? _________________________________________________________________________ Get Your Private, Free E-mail from MSN Hotmail at http://www.hotmail.com. Share information about yourself, create your own public profile at http://profiles.msn.com.

6 5

The FBI tripping over itself again
by Kai Schlichting 31 Oct '00

31 Oct '00

In case any participants of NANOG 20 had any confidence left in the FBI after their Carniwhore presentation and getting shit from an attendee for an apparent illegal wiretap without a warrant in a network owned by said attendee's company, check this out: FBI confiscates computers of "curious" RPI student in aftermath of the yankees.com hack: Sounds like Operation Sundevil all over again (that one having gone down under SS direction): http://www.kuro5hin.org/?op=displaystory;sid=2000/10/31/0481/1037 http://slashdot.org/articles/00/10/31/025228.shtml If this gets a message across to network providers and individuals alike: The FBI is wasting your time, their own time and should be ignored and avoided at all cost. Imminent fall-out: Networks under my control (and more so some I've been called in to assist) are currently under attack by SMTP dictionary-attackers, which seems to be reincarnations of the ghosts of GeolistPro. [ scumbags that are trying to deliver spam, and/or are trying to learn every possible email address in a given domain, to the tune of up to 550,000 per attack (there's obviously a lot of usernames@ scraped from existing spam-lists). Did you ever have 500,000 mails in your Lotus Notes mail spool, because the product (by default) is too dumb to discard undeliverables and keeps them nicely for the postmaster to review ? This even brings Quad-PIII/600Mhz machines with 2GB of RAM to its knees. Now think of the attackers reselling these 500,000 'confirmed' email addresses to other spammers...the future damages of this bogus "data collection" will quickly reach 10,000's of dollars) Does the LE community honestly believe that I will encourage the owners of these networks to contact the authorities instead of suing the shit out the attackers in civil court instead? I didn't think so... And as far as this concerns this fucked up incident at RPI: Dear Almighty, please shine some clue rays down on Albany and Troy (apart from RPI, that's one VERY fucked up and run-down town), two towns that are badly in need of such grace, along with the local FBI office, which seems staffed by people who need to do a few more runs thru the Reston FLETC for 'computer training' (that moved to Quantico?), because the first time obviously didn't improve their clue level a whole lot and just caused confusion. (Hey Kim C., after 7 years, you still have a lot of work left to do with 'dem bloody recruits)

3 2

RE: InterNAP?
by Spolidoro, Guilherme 31 Oct '00

31 Oct '00

Since they don't own the backbone, their SLAs apply only to their PNAPs, i.e. they cannot guarantee performance, packet loss, delay, outages, etc over somebody else's backbone (e.g. Sprint, UUNet, AT&T, etc). Many large enterprises are leveraging on the Internet to deploy VPN connectivity between company sites as a replacement for Frame Relay or ATM PVCs. In such cases, it's very important to make sure that the service comes with the right SLAs. -----Original Message----- From: Tom Schmidt [mailto:tsch52@hotmail.com] Sent: Monday, October 30, 2000 5:42 PM To: nanog(a)merit.edu Subject: InterNAP? I need your opinion on InterNAP. I am currently have a DS3 to my current provider and want to add an additional DS3 for redundancy to the same location. We plan to run BGP4 on both connections. InterNap has some technology to avoid congested peering points. Does this technology actually work? Isn't it impossible to avoid these peering points? What are your experiences with InterNAP? _________________________________________________________________________ Get Your Private, Free E-mail from MSN Hotmail at http://www.hotmail.com. Share information about yourself, create your own public profile at http://profiles.msn.com.

3 2

AADS
by Mark Borchers 31 Oct '00

31 Oct '00

Anybody know anything about possible switch problems at AADS? I have multiple, but not all, pvc's down, but haven't been able to reach any Ameritech folks yet for information. Thanks, -- Mark Borchers Splitrock Services IP Engineering 9012 New Trails Dr. mborchers(a)splitrock.net The Woodlands, TX 77381 (281) 465-1931 http://www.splitrock.net/

1 0

Re: InterNAP
by Neil J. McRae 31 Oct '00

31 Oct '00

Mike Johnson writes: > I'm not a customer, but I spoke with a salesdriod there. All they do > is go in, set up their own geographicly disperse private peering points > and link them together with their private backbone. They set these > things up by buying service from the other large backbone folk. The > idea being that, in general, if they can get their traffic onto the same > backbone that the end user is connected to, then it's avoided the > public peering points. It's a good theory. Dunno if it works. In my opinion its not a peering point. Its an aggregation point of several carriers into one AS. There is no magic to this if you have a direct link to 5 of the biggest providers [1, 701, 1239 etc] then you have a good control over how your traffic flows on the outbound, a few lines of perl, rping and a BGP table dump and you can probably, to a limited extent have some effect on the path your traffic takes. I suspect that trying to implement the same architecture outside of the United States will be substantially harder though. Also at some point in time, one of their access nodes is bound to have a subsAtantial ammounts of traffic and then they would have to link some of the other access nodes to this one to avoid other congestion points. One thing that is good about the idea, is that they have a customer contract with their transit suppliers with some fairly useful SLA's contained within, which should mean they have some control over the quality of those providers. Regards, Neil. -- Neil J. McRae - Alive and Kicking D O M I N O . O R G neil(a)DOMINO.ORG Regards, Neil.

1 0

RE: InterNAP?
by Rishi Singh 30 Oct '00

30 Oct '00

We have two DS3s to them and we haven't been too happy. Outage are common from their router-->up and local loops as well. We have all monitoring service turned on with them and have never gotten a call, page or email during one of our outage and we have five people on the list. We're leaving them as soon as we get our other circuits. -----Original Message----- From: Tom Schmidt [mailto:tsch52@hotmail.com] Sent: Monday, October 30, 2000 5:42 PM To: nanog(a)merit.edu Subject: InterNAP? I need your opinion on InterNAP. I am currently have a DS3 to my current provider and want to add an additional DS3 for redundancy to the same location. We plan to run BGP4 on both connections. InterNap has some technology to avoid congested peering points. Does this technology actually work? Isn't it impossible to avoid these peering points? What are your experiences with InterNAP? _________________________________________________________________________ Get Your Private, Free E-mail from MSN Hotmail at http://www.hotmail.com. Share information about yourself, create your own public profile at http://profiles.msn.com.

1 0

RE: IS-IS protocol implementation problem
by rdobbins＠netmore.net 30 Oct '00

30 Oct '00

You're right - I'm looking for fault here in my own equipment, whereas a screwed-up IGP on their end could well have caused all the problems I was experiencing. Occam's Razor, of course! Your point about the iBGP confederations is well-taken, by the way. ----------------------------------------------------------- Roland Dobbins <rdobbins(a)netmore.net> // 818.535.5024 voice -----Original Message----- From: smd(a)clock.org [mailto:smd@clock.org] Sent: Monday, October 30, 2000 11:11 AM To: rdobbins(a)netmore.net Subject: RE: IS-IS protocol implementation problem [not to list] | No, I'm a single-AS hosting provider, no confederation. UUNET has confederations; I was doing a public thinking exercise (trying to coopt smart people who still read the NANOG list, too) about the extent to which the insulation of iBGP in a confederated AS from the disappearance of iBGP peers for an otherwise full-mesh iBGP layout interacts with the exposure of multi-AS-one-IGP confederations to failures which cause the iBGP next-hops to disappear. If it seems a bit esoteric, don't worry, it is... | The more I think | about it, the more I'm convinced that CEF simply stopped working; all my | interfaces were active, and there were no apparent problems with my IGP, | which is OSPF. Right, UUNET was having the problems; you were just a victim of their internal routing being so broken that they couldn't make packets move to you reliably, even though their routers knew how to get to your network; likewise their routers kept telling you they knew how to get to all sorts of networks which in fact they couldn't reach. This problem appears consistent with an IGP problem inside UUNET, which is known to use/have-used confederated ASes. | I think that major BGP wigginess caused the CEF problem; thanks very much | for you insight, I definitely need to think about it some more. What makes you think it was a CEF problem? Sean.

1 0

RE: IS-IS protocol implementation problem
by rdobbins＠netmore.net 30 Oct '00

30 Oct '00

No, I'm a single-AS hosting provider, no confederation. The more I think about it, the more I'm convinced that CEF simply stopped working; all my interfaces were active, and there were no apparent problems with my IGP, which is OSPF. I think that major BGP wigginess caused the CEF problem; thanks very much for you insight, I definitely need to think about it some more. -----Original Message----- From: smd(a)clock.org [mailto:smd@clock.org] Sent: Monday, October 30, 2000 7:28 AM To: nanog(a)merit.edu; rdobbins(a)netmore.net Cc: neil(a)colt.net; sean(a)donelan.com Subject: RE: IS-IS protocol implementation problem | I had a bizarre event occur on Thursday night/Friday morning, and this is | likely the culprit. Some of your symptoms are consistent with a badly-broken sloshing IGP, notably the drop in traffic load and large numbers of dying TCPs passing through the afflicted network. This is two sides of the same coin: a destination in your network, learned through (e)BGP is mapped to a next-hop address (typically the interface across which you are talking (e)BGP)and propagated through their network via iBGP. The IGP is used so that each iBGP-talking router knows how to get to each next-hop address. A sloshing IGP will break connectivity between a given router and all the addresses associated with a broken next-hop. A hypothesis: for each afflicted router, the failure of one next-hop-address to be reachable will cause your ENTIRE network to be unreachable by sources relying upon traffic passing through that router. This may mean a sizeable proportion of their customer base simply could not reach you reliably enough to maintain a TCP connection in equilibrium, or at all. Frequent transition to slow-start due to loss/out-of-order-packets *and* a reduction in the overall number of TCP "mice", would severly reduce traffic. An interesting question, however, is why would their iBGP TCP connections appear to remain functional (you aren't losing eBGP routes) in this sort of mess? Did loopback addresses not come and go, but interface addresses did? (That would be interesting to consider in the face of possible aggregation of interface addresses into the IGP). Is there significant partitioning because of, for example, AS confederating, mitigatiing the problem by removing iBGP's need to know about distant loopback addresses, but not distant next-hop-addresses? We are lucky to have what could be a very interesting case study in routing scalability trade-offs. What a pity nothing like outage(a)sprint.net exists any more, where we might find useful information from the victim provider. :-( Sean.

1 0

RE: IS-IS protocol implementation problem
by smd＠clock.org 30 Oct '00

30 Oct '00

| I had a bizarre event occur on Thursday night/Friday morning, and this is | likely the culprit. Some of your symptoms are consistent with a badly-broken sloshing IGP, notably the drop in traffic load and large numbers of dying TCPs passing through the afflicted network. This is two sides of the same coin: a destination in your network, learned through (e)BGP is mapped to a next-hop address (typically the interface across which you are talking (e)BGP)and propagated through their network via iBGP. The IGP is used so that each iBGP-talking router knows how to get to each next-hop address. A sloshing IGP will break connectivity between a given router and all the addresses associated with a broken next-hop. A hypothesis: for each afflicted router, the failure of one next-hop-address to be reachable will cause your ENTIRE network to be unreachable by sources relying upon traffic passing through that router. This may mean a sizeable proportion of their customer base simply could not reach you reliably enough to maintain a TCP connection in equilibrium, or at all. Frequent transition to slow-start due to loss/out-of-order-packets *and* a reduction in the overall number of TCP "mice", would severly reduce traffic. An interesting question, however, is why would their iBGP TCP connections appear to remain functional (you aren't losing eBGP routes) in this sort of mess? Did loopback addresses not come and go, but interface addresses did? (That would be interesting to consider in the face of possible aggregation of interface addresses into the IGP). Is there significant partitioning because of, for example, AS confederating, mitigatiing the problem by removing iBGP's need to know about distant loopback addresses, but not distant next-hop-addresses? We are lucky to have what could be a very interesting case study in routing scalability trade-offs. What a pity nothing like outage(a)sprint.net exists any more, where we might find useful information from the victim provider. :-( Sean.

2 1

RE: IS-IS protocol implementation problem
by rdobbins＠netmore.net 30 Oct '00

30 Oct '00

I had a bizarre event occur on Thursday night/Friday morning, and this is likely the culprit. I peer with AS701. At approximately 11:15PM PDT or thereabouts (2:15AM EDT), the 7507 which provides my connectivity to uu.net went belly-up in a very strange way. The BGP session with 701 showed active, with full tables; existing TCP connections stayed up. However, it appeared that all new connections inbound from 701 were being dropped on the floor, and my outbound traffic with them dropped from 40mb/sec down to about 5kb/sec. The same router was also handling a secondary connection to pbi.net; because the BGP stayed active and in a supposedly functional state, traffic didn't get routed in that direction as it should've been. I had to reload the router to get it to function properly. Very odd. Nothing in the logs, etc. The router just essentially went on strike, and I've no idea why. I don't run IS-IS, needless to say, especially with a foreign AS. This particular 7507 was running an 11.3.x CC-train IOS, and hadn't had any of the ISO/CLNS family of protocols enabled, ever. This is a bit earlier than the timeframe Sean cited, but I don't think it was a coincidence, either. ----------------------------------------------------------- Roland Dobbins <rdobbins(a)netmore.net> // 818.535.5024 voice -----Original Message----- From: Neil J. McRae [mailto:neil@COLT.NET] Sent: Monday, October 30, 2000 1:28 AM To: sean(a)donelan.com Cc: smd(a)clock.org; nanog(a)merit.edu Subject: Re: IS-IS protocol implementation problem > At approximately 7:37am EDT on Friday, about 258 Cisco 12000's on UUNET's > primary backbone reloaded. This appeared to be isolated to routers > in ASN 701. It disrupted reachability to about 15% of the world-wide Internet > based on data from Matrix measurements. A contributing cause was a bad > IS-IS packet which confused certain IOS versions in the 12.0 IOS software > train. I haven't heard what the root cause was or what originated the > bad IS-IS packet. The Cisco bug id is CSCdr05779. Any provider running the > affected IOS version may be vulnerable depending on what the root cause > turns out to be. > > Although the bad IS-IS packet didn't propagate to other providers, several > other providers did report BGP resets and route flaps about the same time. If a large AS such as AS701 starts flapping I wouldn't be surprised if other ASes start seeing BGP resets and route-flaps. Could be that crud routing information was exchange when that chaos started [jeez 258 routers I'd hate to have been the on duty NOC guy on that morning :-)] Interestingly though we still see alot routes with bad AS-PATH information people should be setting more stringent configurations on the routes the learn and subsequentally pass on to avoid this. Regards, Neil.

1 0