- Test - lists.nanog.org

ANNOUNCEMENT: New listserver - ISP-Telco
by Richard Cramer 06 Jul '96

06 Jul '96

Hopefully we will be filling a void. People have asked for a list that would have as its subject: Telco equipment as related to the ISP environment. We hope to deal with a broad subject matter; CSU/DSU configuration, equipment feature comparisons, T-Carrier analyzer tips, lease line procurement, etc, etc, etc. The list is now open for subscriptions. We will take subscriptions for about 3-5 days. This way no one will miss anything and we can also shake out any last minute problems. Postings will be accepted starting July 10th unless notified otherwise. SUBSCRIPTION INFO: Subscribe to: majordomo(a)list.sytex.net, with no subject, and "subscribe isp-telco" in the body of the message. Regards, Dick -- Richard Cramer rcramer(a)sytex.net Phone: 703-425-2515 President Fax: 703-425-4585 Sytex Access Ltd. POB 2385, Fairfax, VA 22031-0385

1 0

re: AGIS Route Flaps Interrupting its Peering?
by Peter Kline, Sr. Network Engineer 05 Jul '96

05 Jul '96

Here's some background: AGIS's router is not colocated at the MAE parking garage, but is in fact colocated at WorldCom in downtown Washington DC. Our bits get from there to the MAE via a DS3, and that DS3 is terminated at each end with a device called a NetEdge, which does the FDDI to DS3 ATM conversion. These NetEdges seem to have three different possible operating states: completely working (which doesn't happen often enough); broken (often, right out of the box); and kind of working (which happens all too often). This third operating state results in some very interesting, possibly misleading, and sometimes damaging behavior. It looks quite similar to the kind of behavior you get when you change the MAC layer device but keep the same ip address at either of the MAE's: ARP caches get inconsistent, and BGP sessions with other routers flop around, leading to routes getting flap dampened by those running the appropriate code. Here's what happened: AGIS's connection to MAE-East experienced one of these kind-of-working problems which resulted in the erratic behavior above. Digex customers wishing to reach AGIS customers called the Digex NOC, and the posting which started this all was made to the Digex internal news group. Similarly, AGIS customers had problems, and we worked with MFS to get the problem resolved (they must have a warehouse full of swapped-out NetEdges at this point). In the interval, a short-on-facts bozo spit into the wind and got us and Digex wet. I'm in private correspondence with Ed Kern to postmortem the situation. Peter At 10:25 AM 7/5/96 -0400, Ed Kern wrote: >> >> One key point is that we have not received any complaints or reports >> of any sort concerning any perceived issues at mae-east from any >> mae-east peers. Digex made no attempt to contact us. We were already >> working with Advantis on the unreachable issue above, but the first we >> heard of the "AGIS attacks mae-east" report was when a Digex customer >> sent us a report similar to that forwarded to all of you by Cook. > >Went into this in the last message...Digex will try and be more >proactive with pointing out Agis flapping prefixes in the future. > > >> An appropriate audience would have been the AGIS noc and the Digex >> noc. I think the Cook approach was inappropriate because the issue >> was purely between Digex and AGIS until Cook distributed it to the >> three widespread mailing lists. > >I agree.. > > >> > How is the report flawed? >> >> I see that Ed Kern has already replied indicating that the report was >> indeed flawed. I don't think that there is anything to be gained by >> going into further detail. > >What I was referring to was the internal circulation here...which I was >under the impression got to external customers....now im not so >sure... > >The internal report was flawed because it relied to much on source >routes and came to some bad conclusions on the internal state of agis. > > >> My key point is that nothing of interest happened. This was a >> non-issue until the misinformation was blasted around the Internet >> technical universe. >> > >I would argue that the external message that got sent around was >misinformation...It was correct information from what the people >could see at the time it was released...(lots of dampened prefixes and >a down peer).. > > >Ed > > > _____________________________________________________________________ Peter Kline Senior Network Engineer| 313-730-5151 AGIS - Internet Backbone Services | _Lucem Diffundo_ Post-Traumatic Success Disorder+ | ///////////////////////////////////////////////////////////////////// You can pretend to care, but you can't pretend to be there.

3 2

Re: Internet access and telco usage patterns
by Hong Chen 05 Jul '96

05 Jul '96

Eric, As a matter of fact, it is quite doable. Aimnet developed a roaming server (check www.aimnet.com) that allows international ISPs to use each other's network to provide dialup services. A group of ISPs have joined a consortium GRIC (Global Reach Internet Consortium) lead by Aimnet. The roaming server is based on Radius protocol. A telco company can install modems and route the authentication to the specific ISP for authentication. I just came back from Montreal INet 96 last week and a new roaming IETF group will be started. We are working on the IETF draft for the roaming and stay tuned. Hong Aimnet -------- Begin Included Message ------- > At 05:36 PM 7/4/96 -0700, Michael Dillon wrote: > >> > Has anybody looked at the practicality of installing modem pools in each > >> > telco switch location to grab the dialup user's traffic at the IP level > >> > and then route it to their ISP of choice for authentication and further > >> > routing. In other words, an ISP wouldn't buy phone lines, modems and > >> > terminal servers, they would buy IP access ports. > > > >I now find that BC-Tel is offering this service in parts of British > >Columbia. BC-Tel is half owned by GTE. > > Michael, > > Also of possible interest here is that Westel, another telco in British > Columbia started offering this service early 3rd quarter of last year (with > Livingston Portmasters, ISP managed RADIUS authentication, and its own > Internet connection) which pressured to BCTel to offer a similar service. > Eric Woodward. > ejw(a)wedirect.ca. > > -------- End Included Message --------

1 0

re: AGIS Route Flaps Interrupting its Peering?
by Peter Kline, Sr. Network Engineer 05 Jul '96

05 Jul '96

At 06:41 AM 7/5/96 +0100, Sean Doran wrote: >| Since ANS seems to be passing our interface address as the >| next-hop directly to some nets (e.g., Digex and Advantis), the failure >| as I described above did lead to a loss of connectivity between AGIS >| and at least Digex and Advantis. Pending the solution of the MFS >| problem, it would have been possible to work around the issue if the >| affected nets had routed _through_ their transit provider. > >Let me reiterate the point that propagating third-party >next-hops in the absence of guaranteed fate-sharing is EVIL, >or at least very very risky. I agree. >My opinion is probably at one pole of the spectrum of ideas >about NAPs and MAEs, however it's essentially this: do not >propagate other people's next-hops at all to your NAP/MAE peers, >either using next-hop-self (or the equivalent) or announcing only >those prefixes for which you have yourself as a next-hop. Moreover, >one should be very cagey about accepting third-party next-hops >from one's peers, and either refuse routes with such next-hops, >or (with permission only), rewrite the next-hops in question, >unless there is a very good reason to do otherwise. AGIS always configures with next-hop-self, so that the only way we can propagate a nonworking next-hop is if our own router is down. I'm considering seriously Sean's recommendation about vetting next hops from others. [...chop...] >an issue. However, this should not be the default behaviour >at any exchange-point, because the AGIS/DIGEX disconnectivity >is a well-known and formerly oft-seen problem. If "well-known" means "the net is all a-twitter about it", then I agree. If "well-known" means "seen over a long period of time" then I disagree. In any case, as Sean now characterizes the problem as "formerly", he obviously considers it fixed. Further, as AGIS sees AS2548 directly at MAE-East, as well as behind AS690 and AS1239_1800 (!), we'd have to be broken two or three ways at each of 5 exchange points to completely lose connectivity to Digex. [...whack...] >On another front, *weird* MAE and NAP setups have caused >so much trouble that I sometimes wonder when the next time >I get to say "I told you so" about multi-fabric-bridging-from-hell >will be, and how bad it will hurt. I'm afraid it will be all too soon. LAN-based exchange points presuppose that every router at the exchange point will want to talk to every other router at the exchange point. That deprives the router operator of some meaningful control. Peter _____________________________________________________________________ Peter Kline Senior Network Engineer| 313-730-5151 AGIS - Internet Backbone Services | _Lucem Diffundo_ Post-Traumatic Success Disorder+ | ///////////////////////////////////////////////////////////////////// You can pretend to care, but you can't pretend to be there.

1 0

re: AGIS Route Flaps Interrupting its Peering?
by Mark Kent 05 Jul '96

05 Jul '96

> So, Mark, you're saying that your routers did not have a problem? > If your routers did have a problem, did it affect other providers? Right. Our routers did not create any problems that we are aware of. There was a minor problem at mae-east that appears to have been caused by the MFS NetEdge. Several IP addresses in 192.41.177.0/24 became unreachable from our router at 192.41.177.145, but not our router at 192.41.177.170. We contacted MFS about the problem and they rebooted the NetEdge in the middle of last night. Things are back to normal. Since ANS seems to be passing our interface address as the next-hop directly to some nets (e.g., Digex and Advantis), the failure as I described above did lead to a loss of connectivity between AGIS and at least Digex and Advantis. Pending the solution of the MFS problem, it would have been possible to work around the issue if the affected nets had routed _through_ their transit provider. > If it did affect other providers, were you available to resolve > the problems in a timely manner? I do not believe that we affected other providers in the manner implied by the [internal] Digex report. We had three bgp sessions down. We could not ping those neighbours. MAE-East transit customers of other networks could have been affected as I described above, if they were not protected by "next-hop-self" from their transit provider. One key point is that we have not received any complaints or reports of any sort concerning any perceived issues at mae-east from any mae-east peers. Digex made no attempt to contact us. We were already working with Advantis on the unreachable issue above, but the first we heard of the "AGIS attacks mae-east" report was when a Digex customer sent us a report similar to that forwarded to all of you by Cook. > ] Gordon Cook asks of an inappropriate audience: > > I'm not certain it's innapropriate. If someone's trying to play > with us, and causing us problems, then I would like to know about > it. An appropriate audience would have been the AGIS noc and the Digex noc. I think the Cook approach was inappropriate because the issue was purely between Digex and AGIS until Cook distributed it to the three widespread mailing lists. > ] The Digex report is seriously flawed. Ed Kern of Digex has told us > ] that he is looking into how the conclusions in that report were reached. > > How is the report flawed? I see that Ed Kern has already replied indicating that the report was indeed flawed. I don't think that there is anything to be gained by going into further detail. > ] However, I'm sure that Kern didn't think that he would have to share > ] his findings with nanog, inet-access, and agislist... > > Could you share your findings? If there was a problem, I think > we'd all prefer an explanation rather than a finger shaking for > having interest. My key point is that nothing of interest happened. This was a non-issue until the misinformation was blasted around the Internet technical universe. -mark

2 1

re: AGIS Route Flaps Interrupting its Peering?
by Ed Kern 05 Jul '96

05 Jul '96

> > One key point is that we have not received any complaints or reports > of any sort concerning any perceived issues at mae-east from any > mae-east peers. Digex made no attempt to contact us. We were already > working with Advantis on the unreachable issue above, but the first we > heard of the "AGIS attacks mae-east" report was when a Digex customer > sent us a report similar to that forwarded to all of you by Cook. Went into this in the last message...Digex will try and be more proactive with pointing out Agis flapping prefixes in the future. > An appropriate audience would have been the AGIS noc and the Digex > noc. I think the Cook approach was inappropriate because the issue > was purely between Digex and AGIS until Cook distributed it to the > three widespread mailing lists. I agree.. > > How is the report flawed? > > I see that Ed Kern has already replied indicating that the report was > indeed flawed. I don't think that there is anything to be gained by > going into further detail. What I was referring to was the internal circulation here...which I was under the impression got to external customers....now im not so sure... The internal report was flawed because it relied to much on source routes and came to some bad conclusions on the internal state of agis. > My key point is that nothing of interest happened. This was a > non-issue until the misinformation was blasted around the Internet > technical universe. > I would argue that the external message that got sent around was misinformation...It was correct information from what the people could see at the time it was released...(lots of dampened prefixes and a down peer).. Ed

1 0

re: AGIS Route Flaps Interrupting its Peering?
by Sean Doran 05 Jul '96

05 Jul '96

| Since ANS seems to be passing our interface address as the | next-hop directly to some nets (e.g., Digex and Advantis), the failure | as I described above did lead to a loss of connectivity between AGIS | and at least Digex and Advantis. Pending the solution of the MFS | problem, it would have been possible to work around the issue if the | affected nets had routed _through_ their transit provider. Let me reiterate the point that propagating third-party next-hops in the absence of guaranteed fate-sharing is EVIL, or at least very very risky. My opinion is probably at one pole of the spectrum of ideas about NAPs and MAEs, however it's essentially this: do not propagate other people's next-hops at all to your NAP/MAE peers, either using next-hop-self (or the equivalent) or announcing only those prefixes for which you have yourself as a next-hop. Moreover, one should be very cagey about accepting third-party next-hops from one's peers, and either refuse routes with such next-hops, or (with permission only), rewrite the next-hops in question, unless there is a very good reason to do otherwise. On that point, and with respect to "neighbor foo next-hop-self" at an exchange-point, one has to remember that one will be doing multiple hops across the same medium, which is an inefficient way of making use of both one's (or maybe everyone's, depending on the particular fabric) exchange-point bandwidth, and also one's router capacity. One may, given careful thinking, decide to accept and/or propagate third-party next-hops, if this is an issue. However, this should not be the default behaviour at any exchange-point, because the AGIS/DIGEX disconnectivity is a well-known and formerly oft-seen problem. Finally, there is wording in some bilaterals floating out there with respect to third-party transit at exchange points, which may push towards one pole or the other of opinions on third-party next-hops on non-fate-shared (and often *weird*) LISes. On another front, *weird* MAE and NAP setups have caused so much trouble that I sometimes wonder when the next time I get to say "I told you so" about multi-fabric-bridging-from-hell will be, and how bad it will hurt. Sean.

2 1

re: AGIS Route Flaps Interrupting its Peering?
by Ed Kern 05 Jul '96

05 Jul '96

Depends on which report your talking about...there were two... one that was digex internal and the one gordon sent around... The digex internal one did have a number of flaws and many conclusions that were plain false. The one gordon sent around about having things unreachable inside agis...well thats true...at the time we were having lots of problems with unstable prefixes from agis....as it turns out this may have been a netedge problem with the agis side...(no fault of agis at all) While agis keeps saying that we never contacted them about this problem thats true...We dont get proactive with other providers that have flapping problems..(and that at the time was all that our noc thought it was) Technical problem now fixed...things are back to relative abnormal. Any problems with that Kent? Ed PS. Is it just me or does gordon remind you of fire marshal bill...."Hey letme show ya something" POOF - ---------- Forwarded message ---------- Date: Thu, 4 Jul 1996 14:30:32 -0400 (EDT) From: Mark Kent <mark(a)agisgate.agis.net> To: agislist(a)interstice.com, inet-access(a)earth.com, nanog(a)merit.edu Subject: re: AGIS Route Flaps Interrupting its Peering? Regarding an internal Digex report of AGIS instability, Gordon Cook asks of an inappropriate audience: >> does anyone know what happened? The Digex report is seriously flawed. Ed Kern of Digex has told us that he is looking into how the conclusions in that report were reached. However, I'm sure that Kern didn't think that he would have to share his findings with nanog, inet-access, and agislist... - -mark ------- end -------

1 0

Re: Internet access and telco usage patterns
by Eric Woodward 05 Jul '96

05 Jul '96

At 05:36 PM 7/4/96 -0700, Michael Dillon wrote: >> > Has anybody looked at the practicality of installing modem pools in each >> > telco switch location to grab the dialup user's traffic at the IP level >> > and then route it to their ISP of choice for authentication and further >> > routing. In other words, an ISP wouldn't buy phone lines, modems and >> > terminal servers, they would buy IP access ports. > >I now find that BC-Tel is offering this service in parts of British >Columbia. BC-Tel is half owned by GTE. Michael, Also of possible interest here is that Westel, another telco in British Columbia started offering this service early 3rd quarter of last year (with Livingston Portmasters, ISP managed RADIUS authentication, and its own Internet connection) which pressured to BCTel to offer a similar service. Eric Woodward. ejw(a)wedirect.ca.

1 0

Internet access and telco usage patterns
by Michael Dillon 05 Jul '96

05 Jul '96

There is an interesting column in PC Mag http://www.pcmag.com/issues/1513/pcmg0017.htm regarding "Will the Internet Crash" but Machrone understands the technology better than to do a chicken little imitation. Anyway, he has an anecdote about what PC Mag's BBS did to telco usage patterns with about 100 dialup lines in use nationally. I know that Bell Canada in the toronto area of Canada and BC Tel in Vancouver have also run into problems caused by Internet dialup usage patterns that lead BC-Tel to drop selling Centrex to ISP's and Bell Canada to attempt putting ISP's into a higher tariff category. I suppose other telcos are running into similar capacity problems. Anyway, since some of the core providers in North America are telcos (Sprint, MCI) and since all the other ones work rather closely with the telcos for obvious reasons, I have a question. Has anybody looked at the practicality of installing modem pools in each telco switch location to grab the dialup user's traffic at the IP level and then route it to their ISP of choice for authentication and further routing. In other words, an ISP wouldn't buy phone lines, modems and terminal servers, they would buy IP access ports. I have heard of something like this (I think) going on in Britain where Demon Internet services has Energis providing their dialin lines via something called the Virtual POP in which Demon has no equipment located outside their NOC and has the ability to switch capacity from location to location as load patterns change. Of course, this is probably not working at the IP level but simply dealing with routing of digitized switched audio calls. Michael Dillon ISP & Internet Consulting Memra Software Inc. Fax: +1-604-546-3049 http://www.memra.com E-mail: michael(a)memra.com

4 4