InfoWorld Column on Netcom-Cisco Ampersand Collapse
Dear NANOG, You saw that Craig Huegen has caught me in ANOTHER error. He's of course right (way below) that: The Network Works. No Excuses. is Cisco's slogan, not Netcom's, as I incorrectly wrote in my current InfoWorld column. Netcom doesn't seem to have a slogan. I stand corrected AGAIN. A copy of my incorrect InfoWorld column is below FYI. Nothing slips by NANOG (;->). Not to get back at Mr. Huegen, but he should note that Cisco is not "cisco" anymore. Gotcha! By the way, Mr. Huegen, the well-known fact that the Internet offers no service guarantees has not, as you've written, escaped me. This well-known fact is one of those we are working to FIX. Also, tell us, what was the "original purpose" of the Internet? Not that it matters much. Ever your fan and loyal opposition, /Bob Metcalfe, InfoWorld --------------------------------- InfoWorld, July 8, 1996 (www.infoworld.com) Netcom-Cisco outage could foreshadow much bigger collapses ahead When a tropical storm grows large enough -- winds exceeding 75 mph -- we call it a hurricane and give it a name. It's time we do something similar with Internet outages. Borrowing a threshold used by our Federal Communications Commission in the reporting of telephone outages, when more than 50,000 people are denied their Internet access for more than an hour, let's call it an Internet collapse and give it a name. Let's call the threshold a 50Kx1 collapse, or a 50 kilolapse. Then we can say that, two weeks ago, the Internet suffered a 400Kx13 collapse, or a 5.2 megalapse. Beginning in the afternoon of June 18, the 400,000 customers of Netcom On-Line Communication Services Inc. (http://www.netcom.com) experienced not just the usual worsening afternoon Internet brownout but lost their Internet mail and Web access for 13 hours. (See "Netcom service forced into 12-hour shutdown," June 24, page 3.) Netcom says that this 5.2 megalapse was triggered by an engineer incorrectly typing an ampersand into a router made by Cisco Systems Inc. (http://www.cisco.com). This typo was followed by "a flood of non-Netcom BGP [Boundary Gateway Protocol] routes being introduced into our OSPF [open shortest path first] network backbone. This led to a chain reaction of routing protocol fluctuations, which in turn overloaded a majority of the gateway routers on the Netcom WAN. Our network support staff diagnosed the problem early and worked through the night rebuilding the routing tables of our hub and POP routers." So let's name this the Netcom-Cisco Ampersand Collapse. Netcom CEO Dave Garrison apologized to his customers on KGO talk radio in San Francisco. He explained that the collapse was caused by human error. He admitted that the Ampersand Collapse had overwhelmed Netcom's telephone support. He promised to meet with Cisco, maker of most of Netcom's 100 routers, about preventing future outages. Interviewed by The Boston Globe, Garrison explained the Internet is growing rapidly and there is plenty of room for competition. Then he said, "Internet companies face ruthless competition and don't have billions to spend on reliability upgrades." Uh-oh, this despite Netcom's trademarked slogan: The Network Works. No Excuses. Now, to err is human, and Internet fogies ask us to accept this latest megalapse as nothing new, no big deal. But Garrison's upcoming meeting with Cisco is important. Cisco should continuously improve the software with which its routers are programmed so that catastrophic human errors are less likely. Ed Kozel, Cisco's chief technology officer, writes that "network routing is quite susceptible to human error... complete flexibility is driving routing architecture development... in recent years a lot of work has gone into creating interdomain routing firewalls and untrusted routing gateway functions, the result being that, in general, routing misbehavior is usually confined to a specific domain." So we should be encouraged that the Netcom-Cisco Ampersand Collapse did not escape Netcom and go Internetwide, this time. While Netcom and Cisco are at it, they should find a way to make Internet error messages more informative. Throughout the Ampersand Collapse, Netcom customers were told that their user names and passwords were incorrect, their calls were failing, their network connections were lost, or nothing at all as their starting session screens hung. Now why has Netcom not offered each of its 400,000 customers a refund for the access lost during the megalapse? Let's see, that would be, say, half a day out of 30, or typically 33 cents each. Seems only fair. The Netcom-Cisco Ampersand Collapse and other major outages should be prominent agenda items at upcoming meetings of Internet service providers. Unfortunately, my favorite of such meetings, those of the North American Network Operators Group (NANOG), are not likely to take systematic outage analysis seriously. As one NANOG wag put it, "This is the 'net, people, deal with it." What's needed is for NANOG to deal with it. Another NANOG participant minimized the Netcom-Cisco 5.2 megalapse with this arithmetic: Since the Internet has 60 million users, the Netcom outage inconvenienced far fewer than 1 percent -- some collapse. He has a point. There is ample room for much bigger Internet collapses ahead, maybe eventually some gigalapses. (See what else the NANOG wags are writing about at http://www.merit.edu/mail.archives/html/nanog.) Bob Metcalfe invented Ethernet in 1973 and founded 3Com Corp. in 1979. He receives E-mail at bob_metcalfe@infoworld.com via the Internet. Copyright © 1996 by InfoWorld Publishing Company At 4:15 PM 7/8/96, Craig A. Huegen wrote:
Received: by ccmail from lserver.infoworld.com
From c-huegen@quad.quadrunner.com X-Envelope-From: c-huegen@quad.quadrunner.com Received: from quad.quadrunner.com by lserver.infoworld.com with smtp (Smail3.1.29.1 #12) id m0udR8U-000x1xC; Mon, 8 Jul 96 17:58 PDT Received: from localhost (c-huegen@localhost) by quad.quadrunner.com (8.7.5/8.7-quad) with SMTP id QAA12003; Mon, 8 Jul 1996 16:14:36 -0700 Date: Mon, 8 Jul 1996 16:14:36 -0700 (PDT) From: "Craig A. Huegen" <c-huegen@quad.quadrunner.com> To: Michael Dillon <michael@memra.com> cc: nanog@merit.edu, bob_metcalfe@infoworld.com Subject: Re: Hurricanes redefined! In-Reply-To: <Pine.BSI.3.93.960708104725.22916I-100000@sidhe.memra.com> Message-ID: <Pine.QUAD.3.94.960708155947.11988A-100000@quad.quadrunner.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII
On Mon, 8 Jul 1996, Michael Dillon wrote:
==>The official definition of a hurricane is winds in excess of 5 km/hr ==>for a duration of at least 1 hour. Read all about it at ==>http://www.infoworld.com/cgi-bin/displayNew.pl?/metcalfe/metcalfe.htm
Interesting as well is Bob's incorrectness once again:
"Uh-oh, this despite Netcom's trademarked slogan: The Network Works. No Excuses. "
See http://www.cisco.com/public/copyright.html, in which you'll find:
"All rights reserved. No portion of this service may be reproduced in any form, or by any means, without prior written permission from Cisco Systems, Inc. [...], The Network Works. No Excuses. are service marks; [...] of Cisco Systems, Inc.[...]"
Bob also states that:
"Cisco should continuously improve the software with which its routers are programmed so that catastrophic human errors are less likely. "
Which gives the connotation that cisco Systems doesn't constantly improve IOS; everyone who's worked with cisco's software knows it's constantly being improved.
Bob also states the following:
"Unfortunately, my favorite of such meetings, those of the North American Network Operators Group (NANOG), are not likely to take systematic outage analysis seriously. As one NANOG wag put it, "This is the 'net, people, deal with it.""
What Bob fails to mention is that no one has a service-level agreement with the Internet. The Internet is designed this way--my network connects to your network. It is _NOT_ under control of one body. It's very hard to GUARANTEE outages to _anyone_ without monetary value involved. And generally, "my networks connects to your network" does not have enough monetary value to warrant SLA contracts of service. Bob, I challenge you to find an Internet Service Provider that gives an END-TO-END service level agreement for the Internet. That is, if my web site isn't fast enough, you have escalation/remedy procedures. If Joe Blow's sendmail has crapped out, you have escalation/remedy procedures. I'll buy you dinner if you find one.
Bob once again forgets the original purpose behind the Internet, and he apparently has permanently doffed his engineer hat for his non-technical businessperson hat long ago.
/cah
______________________________________________ ______________________________________________ Dr. Robert M. ("Bob") Metcalfe Executive Correspondent, InfoWorld and VP Technology, International Data Group Internet Messages: bob_metcalfe@infoworld.com Voice Messages: 617-534-1215 Conference Chairman for ACM97: The Next 50 Years of Computing San Jose Convention Center March 1-5, 1997 ______________________________________________ ______________________________________________
On Tue, 9 Jul 1996, Bob Metcalfe wrote: ==>Not to get back at Mr. Huegen, but he should note that Cisco is not "cisco" ==>anymore. Gotcha! Take a look at the logo sometime. In fact, at the time, it created quite a stir within the company, and a lot of long-time employees still use cisco... ==>By the way, Mr. Huegen, the well-known fact that the Internet offers no ==>service guarantees has not, as you've written, escaped me. This well-known ==>fact is one of those we are working to FIX. My point, as I've reflected in private e-mail to you, is that you will never get this without losing value in this network. The Internet is based on sharing information. The minute you begin to charge for each and every little piece of data someone requests from you, the value is lost. When you have a large 'cloud' of providers, you can never guarantee end-to-end connectivity unless one of two things happens: 1. One company buys it all (I don't see this happening), and then the government gets hold of it and 'regulates' it. or 2. You massively change the price structures and force providers to demand money from other providers on a peer level. This is bad, because a LOT of the current structure relies on bi-lateral route peering agreements that are free. Each provider giving a bit to help other providers out. You insert money into there, and a lot of providers won't have incentives to establish agreements for better paths. ==>Also, tell us, what was the "original purpose" of the Internet? Not that ==>it matters much. Now, now, Bob, it really does matter much here. You see, the 'original purpose' of the internet was for research, and fostered a spirit of cooperation between the entities involved. This spirit of cooperation is what has kept this network alive. Insert more money into the equation, and as we've seen since the transition from NSF support, you begin to lose that cooperation in favor of competition, or just plain isolation. /cah
On Tue, 9 Jul 1996, Craig A. Huegen wrote:
Insert more money into the equation, and as we've seen since the transition from NSF support, you begin to lose that cooperation in favor of competition, or just plain isolation.
You mean like X.25 ? :-) Michael Dillon ISP & Internet Consulting Memra Software Inc. Fax: +1-604-546-3049 http://www.memra.com E-mail: michael@memra.com
Bob Metcalfe writes:
By the way, Mr. Huegen, the well-known fact that the Internet offers no service guarantees has not, as you've written, escaped me. This well-known fact is one of those we are working to FIX.
No, you aren't working to fix anything. Your work usually known as "yellow journalism", which is a distinctly different field from any of the professions where people produce things. The NANOG types are the ones working hard to fix things. You make fun of people like Huegen for pointing out that you constantly have factual errors in your articles, like misattributing slogans, quotes, and affiliations. You drip sarcasm when people point out these errors, as though you shouldn't be humilliated by them. Any honest journalist who had gone through a decent education would feel ashamed of even small factual errors. The fact that you have no fact checkers to clear even the simplest details from your missives and that you make fun of the suggestion that you are prone to error demonstrates your overall contempt for your adopted profession. An article in the Journal or the Times would never contain such errors, and editors would feel contrite, not belligerent, if errors were discovered. You aren't, of course, a journalist. You are a man on a vendetta. The NANOG participants insulted you, and now you are going to "show them", eh. Lets keep the facts in mind. The Internet is very young. In spite of this, it works almost all the time, and reliability statistics keep getting better and better. The net is far more reliable than the phone or electrical systems were at the same point in their development -- it is even more reliable than the phone and electrical systems are in most of the countries on earth. I don't think there has been any internet failure as big as the AT&T phone system failure of seven years back. There also hasn't been an internet failure as big as the blackout that hit half the west a couple of weeks ago. However, lets say ocassionally problems as bad as have happened on in the phone network or electric power grid occur. How unexpected! How shocking! Engineers working on a new technology making mistakes, and learning from them! The world must be coming to an end. Of course, we should ignore that, and put Bob Metcalfe in charge of the internet. He'll make sure that people in neckties make all the technical decisions -- he has opinied that the problem is the lack of neckties -- and that way, as any reader of Dilbert can tell you, no problems will ever occur again. Perry
On Jul 9, 11:49, Bob Metcalfe <bob_metcalfe@infoworld.com> wrote:
By the way, Mr. Huegen, the well-known fact that the Internet offers no service guarantees has not, as you've written, escaped me. This well-known fact is one of those we are working to FIX.
"We"? See below.
Borrowing a threshold used by our Federal Communications Commission in the reporting of telephone outages, when more = than 50,000 people are denied their Internet access for more than an hour, let's call it an Internet collapse and give it a name.
You are moving the goalpost. Your new term, "Internet collapse", is rather much out of line with accepted definitions of "collapse". But never mind. You say you want to "fix" the Internet. You borrow a threshold for telephone outages, and apply this to the Internet. You want "guarantees", as in the telephone industry. In other words, your definition of a "fix" is FCC/telco style regulated service provisioning -- instead of letting the market decide, you want a committee to think on behalf of people. You are too late for this. The idea is past its sell-by date. But never mind the philosophy. Have you ever wondered what Internet access costs per minute, whether local, long distance, or international? Your average dial-up Internet user pays 10-40 dollars per month these days. He would pay that in a few hours, using the telephone model. So go ahead and call for your "fix" to the Internet. If you succeed, the customers -- your readers -- won't like what's coming. But you're doing it all for them, right? -- ------ ___ --- Per G. Bilse, Mgr Network Operations Ctr ----- / / / __ ___ _/_ ---- EUnet Communications Services B.V. ---- /--- / / / / /__/ / ----- Singel 540, 1017 AZ Amsterdam, NL --- /___ /__/ / / /__ / ------ tel: +31 20 6233803, fax: +31 20 6224657 --- ------- 24hr emergency number: +31 20 421 0865 --- Connecting Europe since 1982 --- http://www.EU.net e-mail: bilse@EU.net
From: bob_metcalfe@infoworld.com (Bob Metcalfe)
By the way, Mr. Huegen, the well-known fact that the Internet offers no service guarantees has not, as you've written, escaped me. This well-known fact is one of those we are working to FIX.
Who is "we"? I ask since you sign this "...your fan and loyal opposition" Has "we" explored the potential cost impact to the consumer of some arbitrary level of service obligation? Or is the assumption that this should all be accomplished, to any degree which one might fantasize about while sitting at their keyboard, while maintaining the same price? Just as a reality-check, in the US being able to communicate over the phone system to someone in Europe for around $30/hour off-peak is considered a very good price, to the Pacific rim about twice that, call it $50/hour. Personal internet service to anywhere in the world is considered somewhat expensive at $1/hour.
Also, tell us, what was the "original purpose" of the Internet? Not that it matters much.
To remain robust as a defense communications medium in the face of nuclear attack. But that presumed of course that the defense establishment was willing to spend billions of dollars of taxpayers' money and build an infrastructure according to those specifications. It's not some magical property of IP header packet formats. The original purpose of Velcro was to hold space suits closed, but purchasing a yard of Velcro at your local Woolworth's hardly gets you a seat on the next space shuttle launch. I say this with great regret, Mr Metcalfe, but the more I read from you both on the net (such as here) and in your column the less I think of you. I hope you don't take that entirely wrong, but you're being exceedingly silly and for some reason have chosen to cast your formerly sterling reputation, as someone who understands things, to the wind. Sad. It's certainly not a matter of "loyal opposition", that would be an easy rationalization. We argue on the net constantly, nothing unusual. It's that what you represent as criticisms and observations strike me as, I dunno, the cheapest sort of shallow demagoguery I guess. Why not just reduce your columns to "IT SHOULD ALL BE HALF-PRICE AND TWICE AS GOOD!" and save yourself and your readers a lot of time? -- -Barry Shein Software Tool & Die | bzs@world.std.com | http://www.std.com Purveyors to the Trade | Voice: 617-739-0202 | Login: 617-739-WRLD
Dear NANOG,
Actually, Father-O'-The-Internet, (or can I just call you 'Dad'?) you are old enough and wise enough and experienced enough that you don't really need to CC NANOG on this. Your sentences start out with "Mr. Huegen..." and that means you should have used what we call "email," not "annoy the mailing list."
You saw that Craig Huegen has caught me in ANOTHER error.
It's nothing new. You're a biased columnist with an axe to grind, so accuracy is expected to suffer at the expense of your paycheck... If you really gave a shit about the quality of anything you'd write you'd pass it by someone who understands the technology BEFORE You published it. (No, not Gordon...) ...
Not to get back at Mr. Huegen, but he should note that Cisco is not "cisco" anymore. Gotcha!
That's -10 maturity points. Gotcha.
By the way, Mr. Huegen, the well-known fact that the Internet offers no service guarantees has not, as you've written, escaped me. This well-known fact is one of those we are working to FIX. ^^
/laugh. Which royal we do you think you are part of? You're not part of the management group or the operations group.
Also, tell us, what was the "original purpose" of the Internet? Not that it matters much. ^^^^^^^^^^^^ ^^^^^^^^^^^^
Dad? Do you mean the original ARPANET, which had one purpose? Or do you mean the NSFNET, which had another purpose? Or do you mean the ANSnet/Co+Re backbone, which had another purpose? I know it's tough, while you're eating your oatmeal and your prune juice to consider that "the Internet" is an evolving entity, and how it began (military connection of imps via DDS lines) is in no way, shape, or form RELEVANT to what it is today.
Ever your fan and loyal opposition,
Bob, I use ethernet. My hardware's royalties to you is tribute enough. You want anything else (including my readership), you EARN IT. You EARN it by having QUALITATIVELY USEFUL INFORMATION. There are plenty of columnists who do it. Ehud p.s. Your signature is 11 lines long. In "the Internet", Dad, us young kids have established this concept of brevity. Try 4 lines.
Dr. Robert M. ("Bob") Metcalfe Executive Correspondent, InfoWorld and VP Technology, International Data Group
Internet Messages: bob_metcalfe@infoworld.com Voice Messages: 617-534-1215
Conference Chairman for ACM97: The Next 50 Years of Computing San Jose Convention Center March 1-5, 1997 ______________________________________________ ______________________________________________
participants (7)
-
Barry Shein -
bob_metcalfe＠infoworld.com -
Craig A. Huegen -
Ehud Gavron -
Michael Dillon -
Per Gregers Bilse -
Perry E. Metzger