- Test - lists.nanog.org

Re: Netcom Outage (Was: My InfoWorld Column About NANOG)
by avg＠postman.ncube.com 26 Jun '96

26 Jun '96

Stephen Balbach <stephen(a)clark.net> wrote: >Having a fully meshed/redundant network should be the goal of any serious >ISP. The only one that claims it with any substance IMO is UUNET. The "full mesh" in this case is a figment of imagination. Level-2 mapping of cirtuits over the same (non-meshed) physical wires does not make network a tiny weeny bit more reliable, and the added complexity brought by such mapping actually makes system less robust. Other ISPs who happen to be in position to control physical routing of circuits use IP-level rerouting to attack the problem. The IGP rerouting with modern link-state IGPs is sub-second so the redundancy of interior paths is easy to achieve (perticularly if you use tricks like BGP confederations which eliminate need to recompute iBGP routing in case of IGP changes). The hard part is exterior routing where topology changes require massive crunching of BGP tables. Multiplying paths actually makes the problem worse. >We are trying to build one and its not easy. Haveing redundant links >in place does not guarantee instant fall over of traffic. Static routes, IGRP, >iBGP, bridgeing, rip1 vs rip2, etc. are some of the issues we are running >into. The Golden Rule of engineering (often forgotten in US) -- the simplier the system is the better it works. The root of many Internet woes is in overly complicated router software, that complexity appears to be running out of control. An ISP engineer's nightmare is a Bysantine-mode failure -- when redundancy does not help because a problem in one place triggers failures in a lot of other places. Any RISKs reader knows that software problems in distributed systems often have that nature. Internet configurations are particularly prone to that, especially considering that they're in state of constant flux. That's why draconian revision controls and highly skilled backbone engineering staff (which is actually able to understand global consequences of all their actions) are vital to operations of any serious ISP. Netcom seems to be in stage of learning that hard way. >As well as when an interface is down, but actually looks up to the >router, etc..it can be done, but there are so many possible points of >failure and unforseen scenarios, it is very difficult to construct and >certainly takes time to develop. That example is a perfect illustration on why many backbone engineers are sceptical about "advanced" level-2 technologies. Even good ol' Ethernet may have numerous quite interesting ways to fail, if not built properly (remember ol' MAE-E? :) --vadim

2 1

Re: Netcom Outage (Was: My InfoWorld Column About NANOG)
by avg＠ncube.com 26 Jun '96

26 Jun '96

| As for routing loops we all know whom to thank for the wise | decision to give packet forwarding higher priority than to | handling routing updates >Fixed. Ugh! Took, like, four years. Do they do silicon switching into black hole now, too? And allow to squash trivial loops by providing means to say "don't send packets back on the same interface"? | and for using textual regexp matching in policy evaluation, too. >Also fixed. You mean cacheing? That's not exactly "fix", that's workaround. I don't think it works when you change filter and restart the session. The real fix would be to implement something saner than doing regexps (hint -- you never really need backtracking when you match AS paths to policies). >C'mon, beat up on cisco for things that are STILL broken, not >for things that have been fixed after being discovered as problems. I always thought it's more productive not to create silly problems in the first place. I guess i'm lagging on what constitutes the current set of silly problems, though i admit i sleep much better since i stopped worrying about them. --vadim

1 0

Re: Netcom Outage (Was: My InfoWorld Column About NANOG)
by avg＠ncube.com 26 Jun '96

26 Jun '96

>> The IGP rerouting with modern link-state IGPs is sub-second >> so the redundancy of interior paths is easy to achieve (perticularly >> if you use tricks like BGP confederations which eliminate >> need to recompute iBGP routing in case of IGP changes). > Can you be more specific about this? How do you avoid >the recompute and also avoid persistent routing loops after a >topology change? When BGP confederations are used, iBGP routes point to the exit interface address in a satellite AS, instead of the border router's address. That address is invariant (unlike the border router's, which may change with changes in interior routing). This allows to preserve BGP table unchanged in case of IGP changes, and simply re-run one step of recursion (assembly of iBGP and IGP tables) instead of sending costly BGP updates with new border router's addresses. Of course, this works only if backbone is not fractured (i.e. threre's still connectivity to all exits). As for routing loops we all know whom to thank for the wise decision to give packet forwarding higher priority than to handling routing updates, and for using textual regexp matching in policy evaluation, too. --vadim

1 0

My InfoWorld Column About NANOG
by bob_metcalfe＠infoworld.com 25 Jun '96

25 Jun '96

Dear NANOG List, Thanks for your critiques of my NANOG meeting critique column in InfoWorld. Below is a copy of a draft (before editing) of the offending column, just in case some of you have been reading only one another's critiques instead of the column itself. Of course I stand by it. Some of you guys/gals are very good at ad hominem attacks. Flaming is alive and well on the Internet. Tisk tisk. But then I asked for it. Anyway, the attention is flattering. Thank you. A few of you missed one point at least. I am NOT suggesting that any of YOU start wearing suits, especially if you find them uncomfortable, or that they make a statement you are not willing to make -- none of that, no -- good engineers are too valuable to overdress. I am suggesting that more of the kind of people who ALREADY wear suits should start paying attention to the important work NANOG is attempting and start attending your meetings so they can pitch in on the non-engineering aspects of operating the Internet. Is that clearer now? By the way, there are reports from two days ago that 400,000 people lost their Internet access for 13 hours. Sounds like an outage approaching "collapse." Was that just a Netcom thing that NANOG has no interest in? Netcom is not talking very much about what happened. Any clues/facts out there? Were any NAPs involved? /Bob Metcalfe, InfoWorld ---------------------- InfoWorld / From the Ether / Bob Metcalfe NANOG Meeting Column DRAFT TWO The North American Network Operations Group (NANOG) remains our best bet for managing through the Internet's coming collapses. Problem is, like the Internet, NANOG itself is struggling to scale up. I've just been among the 350 mostly engineers attending NANOG's May meeting at George Washington University. It's clear now, even if they hate the idea, that if NANOG is to lead us toward an industrial-strength Internet, then it must now urgently attract the active participation of many more men and women who routinely wear suits. Here, on April Fool's Day, I nominated NANOG as that organization best positioned to lift the Internet out of its current, disfunctional operations anarchy. I then incorrectly identified NANOG as part of the Internet Society's Internet Engineering and Planning Group (IEPG), a seemingly defunct sister of the Internet Engineering Task Force (IETF). Turns out I was wrong about what I'd read on the Web at http://info.isoc.org:80/adopsec. For the next two weeks postings on NANOG's message archive flamed me for not knowing that NANOG is moderated by the Merit Network at the University of Michigan (http://www.merit.edu) NANOG, I was told, has nothing to do with the Internet Society. And further, the Internet Society has nothing to do anymore with the IETF. Checking with a pal at the Society, I was told that IETF has been arguing about disassociating from the Internet Society, and, oh by the way, Merit is "irrelevant." Yes, I found pettiness and bureaucratic infighting among the groups I had hoped would be pulling the Internet together. I stand corrected, but not reassured. Back at NANOG, I was surrounded by people whose life is about "running code." I twiddled as these mostly engineers, unaccustomed as they are to public speaking, stood up one by one in front of 350 people without having ever tried their slides on GWU's projection system. We all waited while Windows booted. If you have running code, it seems, you don't have to respect your audience by checking your slides at least once in advance. Or by wearing a suit. NANOG's opening presentations on "The State of the Internet" were given by the four Network Access Points (NAPs). Pacific Bell (http://www.PacBell.COM/Products/NAP) Sprint (http://www.sprintlink.net/SPLK/HB21.html) Ameritech (http://www.ameritech.com/products/data/nap) and MFS Datanet (http://www.mfsdatanet.com/) each showed how very connected they are to various of the big Internet Service Providers (ISPs). They are installing new equipment to meet ramping demand, are operating well below capacity, and are not losing even a single Internet packet ever, they said. Then came the three large Network Service Providers (NSPs). Sprint, ANS (http://www.ans.net) and MCI (http://www.mci.com/resources) each showed, after some Macintosh booting, that they are installing new equipment to meet ramping demand, are operating well below capacity, and are not losing even a single Internet packet ever, they said. Then the fit hit the shan. Various earnest young speakers from Merit stood up one by one to report "alarming" statistics from the Internet -- rapidly increasing packet loss rates and routing instabilities (http://nic.merit.edu/routing.arbiter/RA/statistics) They asked the NAPs and NSPs, "Where are so many packets being lost?" "Somewhere else," came the denial. Then followed an afternoon and another morning of pleadings. For standards on traffic measurements. For regular outage reporting. For cooperation on gathering topological information to use in Internet operations management. For streamlining multilateral "peering agreements" among ISPs. For systematic use of an Internet Routing Registry. And, from an actual Internet user, pleadings for cooperation on end-to-end service measurements. Sadly, there was nobody at NANOG with the organizational sophistication to grab hold of these pleadings and accelerate them toward action. So, hey, I've got an idea, let's ask the business executives to whom current attendees of NANOG report to buy some T-shirts and take over. The Internet needs more than running code. Now, what would happen if some of NANOG's big university, NAP, and NSP regulars showed up among the many small commercial ISPs expected August 8-10 at ONE ISPCON in San Francisco? I'll be summarizing there. See www.boardwatch.com or call 800-933-6038. END ______________________________________________ ______________________________________________ Dr. Robert M. ("Bob") Metcalfe Executive Correspondent, InfoWorld and VP Technology, International Data Group Internet Messages: bob_metcalfe(a)infoworld.com Voice Messages: 617-534-1215 Conference Chairman for ACM97: The Next 50 Years of Computing San Jose Convention Center March 1-5, 1997 ______________________________________________ ______________________________________________

11 10

Re: Netcom Outage (Was: My InfoWorld Column About NANOG)
by Dick St.Peters 22 Jun '96

22 Jun '96

> Was this a foreshock of the coming Metcalfean Big One I'm having trouble perceiving what the big deal is in all this. It's not like this sort of thing hasn't happened before. An event those who were present and sentient in late 1965 may remember was when a smallish failure on the power grid in southeastern Canada caused a cascading collapse that took out power throughout the northeastern US. Those who, like me, can remember their telephone number being extended from 3 digits to 4 will find nothing new in dire predictions of the large scale collapse of a communications network. It wasn't just predicted, it happened - repeatedly. The telephone network did not sprout from scratch as a multiply-redundant mesh with backup everything. It grew into one, with plenty of 'events' along the way to propel that process forward. These events happen, and they will happen on the Internet. They are not apocalypses; life goes on, perhaps with some minor readjustments in incomes. The Internet can be put back together when it breaks. Certainly on NANOG it is worth lively discussion of how to minimize and recover from such events, but I see little of that in this thread. Of course, now I'm guilty too, so I'll shut up and go back to lurking. -- Dick St.Peters, Gatekeeper, Pearly Gateway, Ballston Spa, NY stpeters(a)NetHeaven.com Owner, NetHeaven 518-885-1295/800-910-6671 Internet for Albany/Saratoga, Glens Falls, North Creek, & Lake Placid First Internet service based in the 518 area code

1 0

Re: Netcom Outage (Was: My InfoWorld Column About NANOG)
by Paul Ferguson 22 Jun '96

22 Jun '96

At 04:48 PM 6/21/96 +0800, Vadim Antonov wrote: > >The Golden Rule of engineering (often forgotten in US) -- the >simplier the system is the better it works. The root of many >Internet woes is in overly complicated router software, that >complexity appears to be running out of control. > Amen, brother. - paul

1 0

Re: Netcom Outage (Was: My InfoWorld Column About NANOG)
by Paul Ferguson 22 Jun '96

22 Jun '96

At 06:35 PM 6/21/96 -0400, Stephen Balbach wrote: > >Having a fully meshed/redundant network should be the goal of any serious >ISP. The only one that claims it with any substance IMO is UUNET. We are >trying to build one and its not easy. Haveing redundant links in place >does not guarantee instant fall over of traffic. Static routes, IGRP, >iBGP, bridgeing, rip1 vs rip2, etc. are some of the issues we are running >into. As well as when an interface is down, but actually looks up to the >router, etc..it can be done, but there are so many possible points of >failure and unforseen scenarios, it is very difficult to construct and >certainly takes time to develop. > This is where having an excellent engineering staff pays off. :-) - paul

1 0

Re: critique of NANOG meeting
by Matthew Cable/USA.NET Inc. 22 Jun '96

22 Jun '96

On Thu, 20 Jun 1996, Arun Welch wrote: > > > > HINT: Get somebody to tie the tie and then have it sewed together or > > something so it doesn't accidentally come undone. > > > > Or get a clip-on tie. Even works with t-shirts. > Or even better....a dilbert tie #!/usr/bin/perl -- Matthew Cable -- USA.NET -- Senior System Administrator $fof='8a*)v2*^Gf#*5S="!jh!;F)]#T):)#&f5kR^(%!E<F#Pf)@2farf&*#ahgu)%C:V5R'; print;$arf=eval{$foo="t1!A53%%1!RBF13!\@$%r/R!$7A39\@aB-z^*#\\)BAS/13/4d"; eval{print;};$foo=~tr/A-Z0-9%$!@!//d;$foo;};$tmp="\$fof=~$arf;";eval $tmp; eval;eval;eval;$\=unpack(u35,$fof)."\n";print #;)>#;0>#:|#8)#;P#80#:o#;)#;

1 0

Re: Withdrawls and announcements attempt 2
by avg＠postman.ncube.com 21 Jun '96

21 Jun '96

The 30 second figure sounds pretty much like the good ol' line flap with default cisco keepalive. I was bothering cisco folks about making static routes persistent by default for quite a while. Although there's a workaround very few people actually bother to use it thus causing bouncing tail links to produce route flap. My suspiction is that this kind of flap accounts for no less than 90% of the total. Another probable contributor is RIP (shouldn't we declare it RIP already?) Most network administrators never heard of a simple maxim -- "Never use dynamic routing when you have a single path". As a result most LANs are plagued by RIP with no reason whatsoever (and many of them inject their IGP routes into BGP!) The lack of any meaningful clock source negotiation and idiotic defaults in CSU/DSUs is a major source of rapid link flap, too. It seems that most (if not all) equipment vendors simply don't have any clue of what effect their design choices have in the real world. --vadim

2 1

Re: My InfoWorld Column About NANOG
by John Curran 21 Jun '96

21 Jun '96

At 11:31 6/21/96, Bob Metcalfe wrote: >... >A few of you missed one point at least. I am NOT suggesting that any of >YOU start wearing suits, especially if you find them uncomfortable, or that >they make a statement you are not willing to make -- none of that, no -- >good engineers are too valuable to overdress. I am suggesting that more of >the kind of people who ALREADY wear suits should start paying attention to >the important work NANOG is attempting and start attending your meetings so >they can pitch in on the non-engineering aspects of operating the Internet. >Is that clearer now? Bob, I believe that you'll find quite a few folks "concerned with the non-engineering aspects of operating the Internet" amongst the attendee list of each NANOG and IEPG meeting. There is a great deal of activity in trying to scale the current Internet, but it doesn't always happen via the group presentations. One of the disadvantages of the decentralized nature of the Internet is that different organizations have different views on the best way to move forward; I'm not certain that a more centralized model can respond to change as quickly as we've accomplished to date. Are there particular issues that you think should be addressed at NANOG and/or IEPG? Both organizations have a fairly open policy with respect to agenda items and I imagine that your suggestions would be most welcome. >By the way, there are reports from two days ago that 400,000 people lost >their Internet access for 13 hours. Sounds like an outage approaching >"collapse." An interesting characterization... you'd definitely need to talk to NETCOM to determine if it was a "collapse" as opposed to a "serious event" or just a "typical failure mode". >Was that just a Netcom thing that NANOG has no interest in? >Netcom is not talking very much about what happened. Any >clues/facts out there? Were any NAPs involved? Again, NETCOM would be the authoritative source for such info... Reuter's June 19th coverage noted that the problem was related to a problem with a routine change to the "routing table". I will note that operating a large Internet backbone requires availability of staff with very specialized skills in both Internet routing and high-speed networking, and all Internet service providers face quite a challenge in maintaining such staff. This is a risk factor to Internet growth and stablility that receives little attention but nevertheless is quite real. /John p.s. Bob - It's possible that not all NANOG folks have a strong interest in this thread; have you considering creating a mailing list to host a discussion on the Internet's "imminent collapse"?

2 1