- Test - lists.nanog.org

NETCOM downtime a programming error. (fwd)
by Michael Dillon 28 Jun '96

28 Jun '96

---------- Forwarded message ---------- Date: Fri, 28 Jun 1996 17:42:30 -0400 (EDT) From: Brian Tao <taob(a)io.org> Reply-To: inet-access(a)earth.com To: inet-access(a)earth.com Subject: NETCOM downtime a programming error. (fwd) Resent-Date: Fri, 28 Jun 1996 16:45:28 -0500 (CDT) Resent-From: inet-access(a)earth.com Not sure where this first showed up, but it sounds like some sort of trade publication... ---------- Forwarded message ---------- IA: Take us through what happened... GARRISON: Think about the network in three layers. The first layer is Network Access Points, where we have peer agreements with other careers. It's the entry point to the Internet. (They're about a half dozen across the U.S.) The next level down are hubs, which are our internal virtual private network routing hubs that look at traffic and direct it along the speediest line available. The third level down is where the customer actually logs on, at an access POP (Point Of Presence). At each of those levels there are routers made by Cisco and others that have instruction tables on them -- "IF THEN" statements that tell the traffic where to go, what route to go to get to its destination. At the network access level, you have some pretty complex code that says, 'If the traffic comes from this party, then do the following thing with it.' And because of the number of new access providers or changes in the access providers, there are daily changes made at the network access layer. And these are changes that are made in software to the routers. It's done in a language called BGP, or Border Gateway Protocol. So, there was one line of code that said, literally, "No redist bgp access list 25 in," just a line of code that revised an instruction. Because the two sentences were put together as opposed to being done on separate lines, the network read it as an "AND" statement instead of an "OR" or an "IF statement. So, what happens is the network automatically replicates the instruction set from the network access point from where this was entered, which was Washington, DC, and it replicated itself to the other network access points. Because of the way the code was written, it then said, 'ah hah, it's a network instruction, not a peering instruction -- I'd better send it out to the hubs.' The hubs saw it, and said, 'ah hah, I'd better send it out to the POPs.' Well, the POPs memory -- the routers at the lower levels of the network -- do not have the memory or capacity for the peering instructions because they don't interface with anybody else, so they don't need that capacity. So, when they got it, it basically froze the routers down at the third level of the network. Meantime, we're sitting reprogramming the routers, but as fast as we can reprogram the replication feature of the intelligent network, it overwhelms our ability to reprogram. Basically our decision was to shut down the network to reboot the routers, to put in a fresh instruction set. That's a long winded explanation, but because your readers are more technical, it's worthwhile! ============================== ISP Mailing List ============================== Email ``unsubscribe'' to inet-access-request(a)earth.com to be removed. inet-access archives are at ftp://ftp.earth.com/pub/archive/inet-access/

1 0

Network Maps/Directory
by William B. Norton 28 Jun '96

28 Jun '96

Hi all - At last NANOG we talked a little bit about network maps, contact info, etc. Brian Noto (BoardWatch) mailed me their latest BoardWatch directory, which lists a rather astounding number of ISPs across the U.S. along with some ISP history, contact #, services and prices, topology maps for some, etc. It looks pretty cool, and after talking with Brian, he agreed to make a copy of the directory available for free to the NANOG community (there is a form to fill out and e-mail below). Bill >From: Jack Rickard[SMTP:jack.rickard@boardwatch.com] >Sent: Tuesday, June 25, 1996 4:09 PM >To: 'Brian' >Cc: 'bob_metcalfe(a)infoworld.com' >Subject: NANOG Participation > >Brian: > >Here's what I think we can do for the NANOG members vis a vis the directory and ONE ISPCON that they might find attractive. > > >BACKGROUND > >We had contacted NANOG earlier in the year regarding participation at the Online Networking Exposition and Internet Service Provider Convention scheduled for August 7-10 in San Francisco. Bill Norton had demurred on holding a NANOG event at ONE ISPCON, but had agreed to come present a briefing on NANOG activities. > >In the June 17th issue of InfoWorld, columnist Bob Metcalfe noted rather publicly the desireability of getting some dialog going between NANOG members and the rank and file small ISP's that make up the bulk of ONE ISPCON. He had also spoken directly with Bill Norton on this same subject at the NANOG meeting. > >By way of explanation and disclaimer, Dr. Metcalfe is not connected with Boardwatch Magazine or ONE, Inc., or ONE ISPCON in any financial sense, and is not responsible for our annual bevy of mistakes and miscues. But he does stand tall in our world quite beyond his role as InfoWorld columnist and Ethernet inventor. He has served a purely mentor/advisor role on our Executive Advisory Board and has spoken at our last two annual shows. He's also the smartest man I've ever met, in my unqualified but nonetheless strongly held opinion, and I routinely find myself tongue tied in his presence. Bottom line, if Bob wants it, we go into scramble mode to make it happen. > >He thinks NANOG and ultimately the Internet could benefit from some level of dialog between the "top of the net" and the little guys at the bottom hooking real people up. Here's my official "me too" then. > >THE DIRECTORY > >In March 1987, Boardwatch Magazine published a Directory of Internet Service Providers - their first effort listing some 1447 Internet Service Providers who had actually been contacted by the publication and interviewed. > >As a result of response to the first directory, the second issue, June 1996, was revised to include more detailed information about the architecture of the Internet. Nine "profiles" of large national backbone providers were added with color maps of their backbones, POP lists, product lists and prices, Internconnects, etc. along with a description of the NAP architecture. The directory listings had grown to some 2266 Internet Service Providers. > >The directory has been sufficiently successful to warrant further development and we are already working on the Fall issue. Bill Norton commented on the value of the architecture descriptions. In discussion with Norton, he thought we might find input from NANOG members valuable. Since anything we can do to make the depiction of the Internet more accurate and useful directly benefits our publishing efforts, we concur. So we are attempting to devise a kind of informal peer review process. The objective here is a bit questionable of course. We are trying to diagram and present a rational picture of the Internet two dimensionally on paper. We are bound to fail for obvious reasons, but hope to do so artfully. From Mr. Norton's response to our June efforts, I think we've achieved that. But we would like to improve it. > >To do this, we've agreed to provide a free copy of the Directory of Internet Service Providers to any NANOG member who cares to identify themselves. The book sells on the newsstand at $9.95 and we normally sell them directly at $9.95 plus $4 s&h. We will provide a copy via first class mail at our expense FREE to any NANOG member who requests one. > >To obtain your review copy, send the following information to BRIAN.NOTO(a)BOARDWATCH.COM > >Name: >Job Title: >Company/Organization: >Street Address: >State/Province: >Zip or Postal Code: >Country: >Voice Telephone: >Fax: >Electronic Mail: > >Please note in the e-mail that you are a NANOG member. We will send a copy of the June 1996 Spring issue, along with a brochure for ONE ISPCON via first class mail. We would appreciate any comments/critique to BRIAN.NOTO(a)BOARDWATCH.COM prior to August 1, 1996. > >ONE ISPCON > >ONE ISPCON is a very focused trade show specifically for Internet Service Providers. It is NOT a general Internet show to get people together and talk about how cool the Internet is. We do have vendor exhibitors at Moscone including Cisco, DEC, Microsoft, Livingston, Bay Networks, U.S. Robotics, etc. - about 60 vendors as of June 15. We also have about 100 educational sessions slated including Mark Corbitt of the FCC, John C. Dvorak, David Graves and Kim Hubbard of NSI, and so on. Sessions will cover marketing, financing, and a lot of technical sessions on building local/regional NAPS, multihoming issues, ADSL, HDSL, wireless, etc. Again, the focus is on how to actually do this stuff as a small ISP - typically 12 employees and 1500 customers. We expect about 1000-1200 true Internet Service Providers, 400-500 ISP wannabees in the main convention/session schedule at San Francisco Hilton Towers, with 3000-4000 peripherally interested vendors, software developers, and assorted hangers on at the vendor exhibits in Moscone. The core "meeting" then at the Hilton will be relatively small and some three blocks distant from the vendor exhibits. > >Registration is $425 until July 15th. > >To encourage NANOG members to come see how the smaller half lives and fulfill the Metcalfe initiative, we will discount this a full $100 to $325 for identified NANOG members registering by July 15. > >MORE INFORMATION > >More information is available at http://www.ispcon.com. The ONE, Inc. office is available at (303)693-5253. Direct questions, particularly regarding session schedules and speakers to BRIAN.NOTO(a)BOARDWATCH.COM. > >Jack Rickard >Editor and Publisher, Boardwatch Magazine >President, ONE, Inc. > >===================================================== >Jack Rickard Boardwatch Magazine >Editor/Publisher 8500 West Bowles Ave., Ste 210 >jack.rickard(a)boardwatch.com Littleton, CO 80123 >303.973.6038 voice 303.973.3731 fax http://www.boardwatch.com >======================================================== > > > >

1 0

RE: Telco discussions
by John R. Sosebee 27 Jun '96

27 Jun '96

from Vadim: > The lack of any meaningful clock source negotiation and > idiotic defaults in CSU/DSUs is a major source of rapid > link flap, too. It seems that most (if not all) equipment > vendors simply don't have any clue of what effect their > design choices have in the real world. I'm seeing that also. I've been using ADC/Kentrox T-1 CSUs lately and they come out of the box set for D4/AMI ... go figure ... BTW: Is there a mailing list for ISP related telco hardware? Or telco hardware as related to ISP's? :) Not that I'm aware of. That would really be very vendor specific. News groups are good for this but like you said it's not data only, voice/video equipment gets mixed in. We talk incessantly about BGP configs, route flaps, saturation stats and Powerpoint graphs, and I don't hear boo about CSU/DSU caveats, typical misconfigs, tips on how to spot DXC/OCU mis-options (Can You Say B8ZS through a C/O that's optioned for AMI???), NIU settings, T-Carrier analyzer tips, etc, etc, etc... It's all black magic for most people. I did hardware/telco at BARRNet for so long, I still have withdrawl symptoms when I see a DSX patch panel or walk into a Wiltel co-lo. me too at SURAnet ... and when I pass a MCI c/o Yes, I know, there are endless mailing lists about telco hardware in general, but those tend to be geared for voice apps, and besides, how long would you read it when the discussions turn to how good the latest NorTel M-1/3 is compared to Brand-X (especially when probably <-1% of ISP's would configure an M-1/3 anyway). Yeah, so you're large enough to hire a contractor to do all your telco. What about us small-fry who can't afford contractors. See if @home is hiring ;) Case-in-point: How many of you tried to figure out why the arrow keys only worked 50% of the time when you used a telnet session to configure a Digital Link Encore DSU / Ensemble shelf? And do you know what Digital Link did as a work around (ans: use the "U" and "D" keys for up/down). DL was real good in getting us a CSU firmware fix for the CSUs. There was a problem where accessing the console port of the DL chassis through a terminal server, the three characters of the arrow key would be received with too much spacing/time between them ... packetman helped picking this up ... Anyway, I made my point. Just curious. nay, too vendor specific ... newsgroups ... John rob.

2 1

RE: Telco discussions
by John Oliveira 27 Jun '96

27 Jun '96

At 04:50 PM 6/25/96 -0400, John R. Sosebee wrote: > > >I'm seeing that also. I've been using ADC/Kentrox T-1 CSUs lately and they come out of the box set for D4/AMI ... go figure ... > This is because most vendors still believes that T1 circuits are provision as D4/AMI..:-) Digital Link Encore csu/dsu by default has it setup as ESF/B8ZS. > >Case-in-point: How many of you tried to figure out why the arrow keys >only worked 50% of the time when you used a telnet session to configure >a Digital Link Encore DSU / Ensemble shelf? And do you know what Digital >Link did as a work around (ans: use the "U" and "D" keys for up/down). > Believe it or not I was the person who made Digital Link come up with the "U" and "D" keys ( don't forget the "L" and "R" for left and right keys). I had meetings with their Engineers to come up with this solution since we had problems telnetting to the csu/dsu through the Terminal Server due to very poor delay response using the arrow keys. It would take at least 5/6 strokes on one arrow key just to toggle it up/down or left/right. This still applies on the 3800 Digital Link Imuxes. John

4 3

Re: Netcom Outage (Was: My InfoWorld Column About NANOG)
by Sean Doran 26 Jun '96

26 Jun '96

| As for routing loops we all know whom to thank for the wise | decision to give packet forwarding higher priority than to | handling routing updates Fixed. | and for using textual regexp matching in policy evaluation, too. Also fixed. C'mon, beat up on cisco for things that are STILL broken, not for things that have been fixed after being discovered as problems. Sean.

1 0

Re: Netcom Outage (Was: My InfoWorld Column About NANOG)
by avg＠postman.ncube.com 26 Jun '96

26 Jun '96

Stephen Balbach <stephen(a)clark.net> wrote: >Having a fully meshed/redundant network should be the goal of any serious >ISP. The only one that claims it with any substance IMO is UUNET. The "full mesh" in this case is a figment of imagination. Level-2 mapping of cirtuits over the same (non-meshed) physical wires does not make network a tiny weeny bit more reliable, and the added complexity brought by such mapping actually makes system less robust. Other ISPs who happen to be in position to control physical routing of circuits use IP-level rerouting to attack the problem. The IGP rerouting with modern link-state IGPs is sub-second so the redundancy of interior paths is easy to achieve (perticularly if you use tricks like BGP confederations which eliminate need to recompute iBGP routing in case of IGP changes). The hard part is exterior routing where topology changes require massive crunching of BGP tables. Multiplying paths actually makes the problem worse. >We are trying to build one and its not easy. Haveing redundant links >in place does not guarantee instant fall over of traffic. Static routes, IGRP, >iBGP, bridgeing, rip1 vs rip2, etc. are some of the issues we are running >into. The Golden Rule of engineering (often forgotten in US) -- the simplier the system is the better it works. The root of many Internet woes is in overly complicated router software, that complexity appears to be running out of control. An ISP engineer's nightmare is a Bysantine-mode failure -- when redundancy does not help because a problem in one place triggers failures in a lot of other places. Any RISKs reader knows that software problems in distributed systems often have that nature. Internet configurations are particularly prone to that, especially considering that they're in state of constant flux. That's why draconian revision controls and highly skilled backbone engineering staff (which is actually able to understand global consequences of all their actions) are vital to operations of any serious ISP. Netcom seems to be in stage of learning that hard way. >As well as when an interface is down, but actually looks up to the >router, etc..it can be done, but there are so many possible points of >failure and unforseen scenarios, it is very difficult to construct and >certainly takes time to develop. That example is a perfect illustration on why many backbone engineers are sceptical about "advanced" level-2 technologies. Even good ol' Ethernet may have numerous quite interesting ways to fail, if not built properly (remember ol' MAE-E? :) --vadim

2 1

Re: Netcom Outage (Was: My InfoWorld Column About NANOG)
by avg＠ncube.com 26 Jun '96

26 Jun '96

| As for routing loops we all know whom to thank for the wise | decision to give packet forwarding higher priority than to | handling routing updates >Fixed. Ugh! Took, like, four years. Do they do silicon switching into black hole now, too? And allow to squash trivial loops by providing means to say "don't send packets back on the same interface"? | and for using textual regexp matching in policy evaluation, too. >Also fixed. You mean cacheing? That's not exactly "fix", that's workaround. I don't think it works when you change filter and restart the session. The real fix would be to implement something saner than doing regexps (hint -- you never really need backtracking when you match AS paths to policies). >C'mon, beat up on cisco for things that are STILL broken, not >for things that have been fixed after being discovered as problems. I always thought it's more productive not to create silly problems in the first place. I guess i'm lagging on what constitutes the current set of silly problems, though i admit i sleep much better since i stopped worrying about them. --vadim

1 0

Re: Netcom Outage (Was: My InfoWorld Column About NANOG)
by avg＠ncube.com 26 Jun '96

26 Jun '96

>> The IGP rerouting with modern link-state IGPs is sub-second >> so the redundancy of interior paths is easy to achieve (perticularly >> if you use tricks like BGP confederations which eliminate >> need to recompute iBGP routing in case of IGP changes). > Can you be more specific about this? How do you avoid >the recompute and also avoid persistent routing loops after a >topology change? When BGP confederations are used, iBGP routes point to the exit interface address in a satellite AS, instead of the border router's address. That address is invariant (unlike the border router's, which may change with changes in interior routing). This allows to preserve BGP table unchanged in case of IGP changes, and simply re-run one step of recursion (assembly of iBGP and IGP tables) instead of sending costly BGP updates with new border router's addresses. Of course, this works only if backbone is not fractured (i.e. threre's still connectivity to all exits). As for routing loops we all know whom to thank for the wise decision to give packet forwarding higher priority than to handling routing updates, and for using textual regexp matching in policy evaluation, too. --vadim

1 0

My InfoWorld Column About NANOG
by bob_metcalfe＠infoworld.com 25 Jun '96

25 Jun '96

Dear NANOG List, Thanks for your critiques of my NANOG meeting critique column in InfoWorld. Below is a copy of a draft (before editing) of the offending column, just in case some of you have been reading only one another's critiques instead of the column itself. Of course I stand by it. Some of you guys/gals are very good at ad hominem attacks. Flaming is alive and well on the Internet. Tisk tisk. But then I asked for it. Anyway, the attention is flattering. Thank you. A few of you missed one point at least. I am NOT suggesting that any of YOU start wearing suits, especially if you find them uncomfortable, or that they make a statement you are not willing to make -- none of that, no -- good engineers are too valuable to overdress. I am suggesting that more of the kind of people who ALREADY wear suits should start paying attention to the important work NANOG is attempting and start attending your meetings so they can pitch in on the non-engineering aspects of operating the Internet. Is that clearer now? By the way, there are reports from two days ago that 400,000 people lost their Internet access for 13 hours. Sounds like an outage approaching "collapse." Was that just a Netcom thing that NANOG has no interest in? Netcom is not talking very much about what happened. Any clues/facts out there? Were any NAPs involved? /Bob Metcalfe, InfoWorld ---------------------- InfoWorld / From the Ether / Bob Metcalfe NANOG Meeting Column DRAFT TWO The North American Network Operations Group (NANOG) remains our best bet for managing through the Internet's coming collapses. Problem is, like the Internet, NANOG itself is struggling to scale up. I've just been among the 350 mostly engineers attending NANOG's May meeting at George Washington University. It's clear now, even if they hate the idea, that if NANOG is to lead us toward an industrial-strength Internet, then it must now urgently attract the active participation of many more men and women who routinely wear suits. Here, on April Fool's Day, I nominated NANOG as that organization best positioned to lift the Internet out of its current, disfunctional operations anarchy. I then incorrectly identified NANOG as part of the Internet Society's Internet Engineering and Planning Group (IEPG), a seemingly defunct sister of the Internet Engineering Task Force (IETF). Turns out I was wrong about what I'd read on the Web at http://info.isoc.org:80/adopsec. For the next two weeks postings on NANOG's message archive flamed me for not knowing that NANOG is moderated by the Merit Network at the University of Michigan (http://www.merit.edu) NANOG, I was told, has nothing to do with the Internet Society. And further, the Internet Society has nothing to do anymore with the IETF. Checking with a pal at the Society, I was told that IETF has been arguing about disassociating from the Internet Society, and, oh by the way, Merit is "irrelevant." Yes, I found pettiness and bureaucratic infighting among the groups I had hoped would be pulling the Internet together. I stand corrected, but not reassured. Back at NANOG, I was surrounded by people whose life is about "running code." I twiddled as these mostly engineers, unaccustomed as they are to public speaking, stood up one by one in front of 350 people without having ever tried their slides on GWU's projection system. We all waited while Windows booted. If you have running code, it seems, you don't have to respect your audience by checking your slides at least once in advance. Or by wearing a suit. NANOG's opening presentations on "The State of the Internet" were given by the four Network Access Points (NAPs). Pacific Bell (http://www.PacBell.COM/Products/NAP) Sprint (http://www.sprintlink.net/SPLK/HB21.html) Ameritech (http://www.ameritech.com/products/data/nap) and MFS Datanet (http://www.mfsdatanet.com/) each showed how very connected they are to various of the big Internet Service Providers (ISPs). They are installing new equipment to meet ramping demand, are operating well below capacity, and are not losing even a single Internet packet ever, they said. Then came the three large Network Service Providers (NSPs). Sprint, ANS (http://www.ans.net) and MCI (http://www.mci.com/resources) each showed, after some Macintosh booting, that they are installing new equipment to meet ramping demand, are operating well below capacity, and are not losing even a single Internet packet ever, they said. Then the fit hit the shan. Various earnest young speakers from Merit stood up one by one to report "alarming" statistics from the Internet -- rapidly increasing packet loss rates and routing instabilities (http://nic.merit.edu/routing.arbiter/RA/statistics) They asked the NAPs and NSPs, "Where are so many packets being lost?" "Somewhere else," came the denial. Then followed an afternoon and another morning of pleadings. For standards on traffic measurements. For regular outage reporting. For cooperation on gathering topological information to use in Internet operations management. For streamlining multilateral "peering agreements" among ISPs. For systematic use of an Internet Routing Registry. And, from an actual Internet user, pleadings for cooperation on end-to-end service measurements. Sadly, there was nobody at NANOG with the organizational sophistication to grab hold of these pleadings and accelerate them toward action. So, hey, I've got an idea, let's ask the business executives to whom current attendees of NANOG report to buy some T-shirts and take over. The Internet needs more than running code. Now, what would happen if some of NANOG's big university, NAP, and NSP regulars showed up among the many small commercial ISPs expected August 8-10 at ONE ISPCON in San Francisco? I'll be summarizing there. See www.boardwatch.com or call 800-933-6038. END ______________________________________________ ______________________________________________ Dr. Robert M. ("Bob") Metcalfe Executive Correspondent, InfoWorld and VP Technology, International Data Group Internet Messages: bob_metcalfe(a)infoworld.com Voice Messages: 617-534-1215 Conference Chairman for ACM97: The Next 50 Years of Computing San Jose Convention Center March 1-5, 1997 ______________________________________________ ______________________________________________

11 10

Re: Netcom Outage (Was: My InfoWorld Column About NANOG)
by Dick St.Peters 22 Jun '96

22 Jun '96

> Was this a foreshock of the coming Metcalfean Big One I'm having trouble perceiving what the big deal is in all this. It's not like this sort of thing hasn't happened before. An event those who were present and sentient in late 1965 may remember was when a smallish failure on the power grid in southeastern Canada caused a cascading collapse that took out power throughout the northeastern US. Those who, like me, can remember their telephone number being extended from 3 digits to 4 will find nothing new in dire predictions of the large scale collapse of a communications network. It wasn't just predicted, it happened - repeatedly. The telephone network did not sprout from scratch as a multiply-redundant mesh with backup everything. It grew into one, with plenty of 'events' along the way to propel that process forward. These events happen, and they will happen on the Internet. They are not apocalypses; life goes on, perhaps with some minor readjustments in incomes. The Internet can be put back together when it breaks. Certainly on NANOG it is worth lively discussion of how to minimize and recover from such events, but I see little of that in this thread. Of course, now I'm guilty too, so I'll shut up and go back to lurking. -- Dick St.Peters, Gatekeeper, Pearly Gateway, Ballston Spa, NY stpeters(a)NetHeaven.com Owner, NetHeaven 518-885-1295/800-910-6671 Internet for Albany/Saratoga, Glens Falls, North Creek, & Lake Placid First Internet service based in the 518 area code

1 0