- Test - lists.nanog.org

Bob Metcalfe: Re: InfoWorld Column on Netcom-Cisco Ampersand Collapse
by Perry E. Metzger 09 Jul '96

09 Jul '96

I thought folks might want to see this. ------- Forwarded Message Date: Tue, 9 Jul 1996 15:33:32 -0400 X-Sender: bob(a)kelmscott.com Message-Id: <v02130502ae082a519beb(a)[204.117.49.7]> To: perry(a)piermont.com From: bob_metcalfe(a)infoworld.com (Bob Metcalfe) Subject: Re: InfoWorld Column on Netcom-Cisco Ampersand Collapse Dear Mr. Piermont, Stay out of this. Since I'm such a creep, please ignore me in the future. /Bob Metcalfe [...] ------- End of Forwarded Message

3 2

InfoWorld Column on Netcom-Cisco Ampersand Collapse
by bob_metcalfe＠infoworld.com 09 Jul '96

09 Jul '96

Dear NANOG, You saw that Craig Huegen has caught me in ANOTHER error. He's of course right (way below) that: The Network Works. No Excuses. is Cisco's slogan, not Netcom's, as I incorrectly wrote in my current InfoWorld column. Netcom doesn't seem to have a slogan. I stand corrected AGAIN. A copy of my incorrect InfoWorld column is below FYI. Nothing slips by NANOG (;->). Not to get back at Mr. Huegen, but he should note that Cisco is not "cisco" anymore. Gotcha! By the way, Mr. Huegen, the well-known fact that the Internet offers no service guarantees has not, as you've written, escaped me. This well-known fact is one of those we are working to FIX. Also, tell us, what was the "original purpose" of the Internet? Not that it matters much. Ever your fan and loyal opposition, /Bob Metcalfe, InfoWorld --------------------------------- InfoWorld, July 8, 1996 (www.infoworld.com) Netcom-Cisco outage could foreshadow much bigger collapses ahead When a tropical storm grows large enough -- winds exceeding 75 mph -- we call it a hurricane and give it a name. It's time we do something similar with Internet outages. Borrowing a threshold used by our Federal Communications Commission in the reporting of telephone outages, when more than 50,000 people are denied their Internet access for more than an hour, let's call it an Internet collapse and give it a name. Let's call the threshold a 50Kx1 collapse, or a 50 kilolapse. Then we can say that, two weeks ago, the Internet suffered a 400Kx13 collapse, or a 5.2 megalapse. Beginning in the afternoon of June 18, the 400,000 customers of Netcom On-Line Communication Services Inc. (http://www.netcom.com) experienced not just the usual worsening afternoon Internet brownout but lost their Internet mail and Web access for 13 hours. (See "Netcom service forced into 12-hour shutdown," June 24, page 3.) Netcom says that this 5.2 megalapse was triggered by an engineer incorrectly typing an ampersand into a router made by Cisco Systems Inc. (http://www.cisco.com) This typo was followed by "a flood of non-Netcom BGP [Boundary Gateway Protocol] routes being introduced into our OSPF [open shortest path first] network backbone. This led to a chain reaction of routing protocol fluctuations, which in turn overloaded a majority of the gateway routers on the Netcom WAN. Our network support staff diagnosed the problem early and worked through the night rebuilding the routing tables of our hub and POP routers." So let's name this the Netcom-Cisco Ampersand Collapse. Netcom CEO Dave Garrison apologized to his customers on KGO talk radio in San Francisco. He explained that the collapse was caused by human error. He admitted that the Ampersand Collapse had overwhelmed Netcom's telephone support. He promised to meet with Cisco, maker of most of Netcom's 100 routers, about preventing future outages. Interviewed by The Boston Globe, Garrison explained the Internet is growing rapidly and there is plenty of room for competition. Then he said, "Internet companies face ruthless competition and don't have billions to spend on reliability upgrades." Uh-oh, this despite Netcom's trademarked slogan: The Network Works. No Excuses. Now, to err is human, and Internet fogies ask us to accept this latest megalapse as nothing new, no big deal. But Garrison's upcoming meeting with Cisco is important. Cisco should continuously improve the software with which its routers are programmed so that catastrophic human errors are less likely. Ed Kozel, Cisco's chief technology officer, writes that "network routing is quite susceptible to human error... complete flexibility is driving routing architecture development... in recent years a lot of work has gone into creating interdomain routing firewalls and untrusted routing gateway functions, the result being that, in general, routing misbehavior is usually confined to a specific domain." So we should be encouraged that the Netcom-Cisco Ampersand Collapse did not escape Netcom and go Internetwide, this time. While Netcom and Cisco are at it, they should find a way to make Internet error messages more informative. Throughout the Ampersand Collapse, Netcom customers were told that their user names and passwords were incorrect, their calls were failing, their network connections were lost, or nothing at all as their starting session screens hung. Now why has Netcom not offered each of its 400,000 customers a refund for the access lost during the megalapse? Let's see, that would be, say, half a day out of 30, or typically 33 cents each. Seems only fair. The Netcom-Cisco Ampersand Collapse and other major outages should be prominent agenda items at upcoming meetings of Internet service providers. Unfortunately, my favorite of such meetings, those of the North American Network Operators Group (NANOG), are not likely to take systematic outage analysis seriously. As one NANOG wag put it, "This is the 'net, people, deal with it." What's needed is for NANOG to deal with it. Another NANOG participant minimized the Netcom-Cisco 5.2 megalapse with this arithmetic: Since the Internet has 60 million users, the Netcom outage inconvenienced far fewer than 1 percent -- some collapse. He has a point. There is ample room for much bigger Internet collapses ahead, maybe eventually some gigalapses. (See what else the NANOG wags are writing about at http://www.merit.edu/mail.archives/html/nanog.) Bob Metcalfe invented Ethernet in 1973 and founded 3Com Corp. in 1979. He receives E-mail at bob_metcalfe(a)infoworld.com via the Internet. Copyright © 1996 by InfoWorld Publishing Company At 4:15 PM 7/8/96, Craig A. Huegen wrote: >Received: by ccmail from lserver.infoworld.com >>From c-huegen(a)quad.quadrunner.com >X-Envelope-From: c-huegen(a)quad.quadrunner.com >Received: from quad.quadrunner.com by lserver.infoworld.com with smtp > (Smail3.1.29.1 #12) id m0udR8U-000x1xC; Mon, 8 Jul 96 17:58 PDT >Received: from localhost (c-huegen@localhost) by quad.quadrunner.com >(8.7.5/8.7-quad) with SMTP id QAA12003; Mon, 8 Jul 1996 16:14:36 -0700 >Date: Mon, 8 Jul 1996 16:14:36 -0700 (PDT) >From: "Craig A. Huegen" <c-huegen(a)quad.quadrunner.com> >To: Michael Dillon <michael(a)memra.com> >cc: nanog(a)merit.edu, bob_metcalfe(a)infoworld.com >Subject: Re: Hurricanes redefined! >In-Reply-To: <Pine.BSI.3.93.960708104725.22916I-100000(a)sidhe.memra.com> >Message-ID: <Pine.QUAD.3.94.960708155947.11988A-100000(a)quad.quadrunner.com> >MIME-Version: 1.0 >Content-Type: TEXT/PLAIN; charset=US-ASCII > >On Mon, 8 Jul 1996, Michael Dillon wrote: > >==>The official definition of a hurricane is winds in excess of 5 km/hr >==>for a duration of at least 1 hour. Read all about it at >==>http://www.infoworld.com/cgi-bin/displayNew.pl?/metcalfe/metcalfe.htm > >Interesting as well is Bob's incorrectness once again: > > "Uh-oh, this despite Netcom's > trademarked slogan: The Network > Works. No Excuses. " > >See http://www.cisco.com/public/copyright.html, in which you'll find: > >"All rights reserved. No portion of this service may be reproduced in any >form, or by any means, without prior written permission from Cisco >Systems, Inc. [...], The Network Works. No Excuses. are service marks; >[...] of Cisco Systems, Inc.[...]" > >Bob also states that: > > "Cisco should > continuously improve the software with > which its routers are programmed so > that catastrophic human errors are less > likely. " > >Which gives the connotation that cisco Systems doesn't constantly improve >IOS; everyone who's worked with cisco's software knows it's constantly >being improved. > >Bob also states the following: > > "Unfortunately, my favorite of such > meetings, those of the North American > Network Operators Group (NANOG), > are not likely to take systematic outage > analysis seriously. As one NANOG > wag put it, "This is the 'net, people, deal > with it."" > >What Bob fails to mention is that no one has a service-level agreement >with the Internet. The Internet is designed this way--my network connects >to your network. It is _NOT_ under control of one body. It's very hard >to GUARANTEE outages to _anyone_ without monetary value involved. And >generally, "my networks connects to your network" does not have enough >monetary value to warrant SLA contracts of service. Bob, I challenge you >to find an Internet Service Provider that gives an END-TO-END service >level agreement for the Internet. That is, if my web site isn't fast >enough, you have escalation/remedy procedures. If Joe Blow's sendmail has >crapped out, you have escalation/remedy procedures. I'll buy you dinner >if you find one. > >Bob once again forgets the original purpose behind the Internet, and he >apparently has permanently doffed his engineer hat for his non-technical >businessperson hat long ago. > >/cah ______________________________________________ ______________________________________________ Dr. Robert M. ("Bob") Metcalfe Executive Correspondent, InfoWorld and VP Technology, International Data Group Internet Messages: bob_metcalfe(a)infoworld.com Voice Messages: 617-534-1215 Conference Chairman for ACM97: The Next 50 Years of Computing San Jose Convention Center March 1-5, 1997 ______________________________________________ ______________________________________________

7 6

Re: Ping flooding (fwd)
by Justin W. Newton 09 Jul '96

09 Jul '96

At 11:57 AM 7/9/96 +0900, you wrote: >Hi, >About the only way you can stop this attack would be for ISPs to >filter out bogus source addresses from their customers. Of course, >then the mobile IP people would whine. However, given a future of >more attacks of this nature, I think the mobile IP people are going to >lose. As long as the mobile IP address stays inside of one provider then it isn't a problem. Justin Newton Internet Architect Erol's Internet Services

1 0

Re: InfoWorld Column on Netcom-Cisco Ampersand Collapse
by Sean Doran 09 Jul '96

09 Jul '96

| Ever your fan and loyal opposition, The only thing you seem to fan is flames. Why don't you write something useful, like why is it that outage reporting is so spotty, and why is it that people don't seem willing to share technical details of outages such that other technical folks can learn from mistakes and disasters? I can tell you why outage(a)sprint.net isn't what it used to be: journalists and PR folks would turn detailed technical reports of the scope, cause, and cure for our outages into columns dedicated to bashing cisco, Sprint, or both. There is a disease on the Internet, and that disease is "us and them"ism. You are well-placed to help cure that disease, yet right now you are one of its biggest transmission vectors, Bob, and it's a huge pity. Sean.

1 0

Re: Ping flooding (fwd)
by Daniel W. McRobb 09 Jul '96

09 Jul '96

> On Mon, 8 Jul 1996, Daniel W. McRobb wrote: > > > The problem is not really a technical one. It's administrative. It's > > much more of a headache to backtrack through 30 routers that aren't in > > your own network than to backtrack to the ingress to your own network > > domain and filter it out there (which is the typical response to this > > kind of thing). Getting everyone in the path to cooperate with > > backtracking is difficult in many instances, impossible in others. > > I recall that people have cooperated in the past on some sort of > performance analysis tool that transported packets through a tunnel to > some remote point and initiated an analysis of some sort from that point > I believe this was done by NLANR and had something to do with vBNS. > > I don't think this is all that different. If some means existed for an NSP > to initiate a trace on a specific source address to backtrack it to the > real source then an easy to use tool could be built. Of course, first of > all router vendors need to make a quick and relatively painless way to > track down the interface that a packet comes in from, maybe There will likely never be a means for a single NSP to track down the real source of spoofed packets using IPv4. Service providers won't be letting other service providers track spoofed packets through their network. > set icmp-source-trace 148.32.45.67 on > > and later.... > > show icmp-source-trace > > IP address Interface > ---------- --------- > 148.32.45.67 NO TRACE > > Note that the source trace was active for a period of time and then > expired automatically with no new ICMP packets bearing the specified > source address in that period of time. If this facility is available an > easy to use tool could be built. In the case of a spoofed-source, denial of service attack, the source address is often of less use than the destination address/port/protocol in tracking down the real source. The attacker just switches the source address and walks right through your trace (or filters). Don't get me wrong; I think packet sniffing capabilities (even in their simplest forms) can be very useful and I wish there were more facilities in typical routers for tracking traffic via IP header information. > > that doesn't even take into account the cases where an attacker has > > multiple paths into your network and is using multiple forged source > > addresses, much less the fact that the attacker can turn off the attack > > when he/she chooses, thwarting your effort to track them. > > No doubt about it. Being a detective is hard boring plodding work and > sometimes you just never find the crook. But it's still worth trying. Define worth. I live in a capitalist society where catching a criminal is of little worth (particularly an ICMP bomber who's arguably not much worse than a USENET spammer) in it's own right and often only worthwhile if there's monetary compensation involved (either from a legal settlement, reward or just recovery of service and time spent fixing things that are broken by an attacker). :-) Daniel ~~~~~~

4 4

Re: since it's national pick-on-bob-metcalfe week anyway...
by avg＠ncube.com 09 Jul '96

09 Jul '96

>bob's contribution to what we're doing here is larger than mine, and >perhaps larger than almost anybody's. where would we be without Ethernet? Using some saner technology like today's full-duplex switched "ether"nets. It's like presuming that we'd be in deep trouble w/o benefits of MS-DOS. Not. --vadim

1 0

Finally -- the May 1996 pages are done
by sob＠academ.com 09 Jul '96

09 Jul '96

The NANOG pages are done. I apologize for the running late, but I had an exceedingly busy June. Comments on the pages are welcome. http://www.academ.com/nanog/may1996 -- Stan | Academ Consulting Services |internet: sob(a)academ.com Olan | For more info on academ, see this |uucp: {mcsun|amdahl}!academ!sob Barber | URL- http://www.academ.com/academ |Opinions expressed are only mine.

1 0

Re: AGIS Route Flaps Interrupting its Peering?
by Brett D. Watson 09 Jul '96

09 Jul '96

> On Fri, 5 Jul 1996, Mark Kent wrote: > > > An appropriate audience would have been the AGIS noc and the Digex > > noc. > > Wrong! It's an operations issue and it SHOULD be on NANOG or a global > outages list if such should some day exist. i have to agree, strongly. > > > Could you share your findings? If there was a problem, I think > > > we'd all prefer an explanation rather than a finger shaking for > > > having interest. > > > > My key point is that nothing of interest happened. > > There are hundreds of providers learning how to deal with multihoming and > peering. Anything like this is of interest because people need to know how > to recognize and solve problems to keep the global network running. > > > This was a > > non-issue until the misinformation was blasted around the Internet > > technical universe. > > Yep, no doubt about it. If you create the environment for misinformation > to spread then it WILL spread. However, you can fix this. If timely an > accurate outage information is available from everybody then > misinformation won't spread and where it does manage to propogate it will > merely be static. this is the major problem i had with the large provider i worked for. the corporate motto was "if we have something broken, we don't want *anyone* else to know about it..." "it's none of *their* business if we have routers melting down..." and my favorate response to customers asking for an "outage notification list" was to create the list, not use it for 6 months, and then come up with some generic statements like "we are having technical difficulty, we;re working on it". complete crap in my opinion. it did nothing but insult customers' intelligence. i agree with what michael is saying (i wont' be struck by lightening will i?) 100%. *timely* disclosure of problems like this would avoid misinformation and bring back a small bit of the spirit of the net that was lost some time back.... *cooperation*. -brett

1 0

Re: Ping flooding (fwd)
by Daniel W. McRobb 09 Jul '96

09 Jul '96

> On Mon, 8 Jul 1996, George Eddy wrote: > > > yes, forging a ping attack is pretty easy and can be done from > > anywhere with any source address (of course, who knows where the > > responses will end up), the routing proximity is irrelavant, since the > > source is not looked at (unless filters have been put in place, such > > as what the upstream provider has apparently done). > > > > the only _I can think of_ in tracking it down, would be to backtrack > > the possible paths into the router. either by sniffing the possible > > lines coming into router, or by temporarily disabling icmp echo reqs. > > from all but one incoming line, until you've found the offending line, > > continuing back. > > > > of course this may be impossible in many cases since you probably > > don't have access to the equipment (or cooperation) outside of your > > domain. > > OK. So what if somebody is currently planning a ping battle on the global > Internet, kind of like corewars in the netwrk. Then what? Do the NSP's all > roll over and play dead? > > If I were to crosspost this reply to alt.2600 it wouldn't take long to > happen you know. BTW, I won't be crossposting it there, but you get the > idea, security by obscurity, etc... I'm quite certain that tons of people know about these kinds of attacks and how to implement them (as well as defend against them). alt.2600 can't possibly be so far behind the times as to not know about forging headers and the possibilities, regardless of whether or not it's ICMP traffic we're talking about. Denial of service attacks are very old in theory and in practice. This is not a security by obscurity issue at all. > Is anyone working on tools to help NSP's quickly backtrack this kind of > thing? The problem is not really a technical one. It's administrative. It's much more of a headache to backtrack through 30 routers that aren't in your own network than to backtrack to the ingress to your own network domain and filter it out there (which is the typical response to this kind of thing). Getting everyone in the path to cooperate with backtracking is difficult in many instances, impossible in others. And that doesn't even take into account the cases where an attacker has multiple paths into your network and is using multiple forged source addresses, much less the fact that the attacker can turn off the attack when he/she chooses, thwarting your effort to track them. Typically a denial of service attack is used to leverage an attack on something more interesting than just pure denial of service. So the denial of service often stops once the attacker has managed to get access to what he/she was really looking for (which is not usually something as uninteresting as ICMP echo requests or attempts to consume a lot of resources with such traffic; there are often better means of making someone's leased line and host machines be consumed than sending ICMP messages). Daniel ~~~~~~

2 1

Re: Ping flooding
by George Herbert 09 Jul '96

09 Jul '96

>We are currently undergoing a ping flood attack, though our upstream >provider has filtered icmp from the host so the flood is no longer >affecting our T1 line. > >The system administrator of the site that appears to be flooding us >doesn't believe his site is the source of the attack. He states that he >can't see the icmp packets, though I don't know how he is sniffing his >wire. > >My questions are these: > >Is it possible for someone to forged the source IP address of an icmp >packet? Yes, trivially. >If so, do they have to be in some routing proximity, or can they forge the >source address while they are connected from anywhere in the world? Anywhere that doesn't have outbound source filtering on their net connection... which is almost nobody. Back when I was at UC Berkeley in the late 80s, variations on this scheme were coded up and regularly used to interfere with other people's network game sessions (re-routing xtrek screen updates through Finland, similar stunts). The same basic concept can be done with any source IP addr and any ICMP command (or, basically, any IP data you want too). It is entirely possible that the information the other sysadmin has, that it's not coming from his net, is correct. If so, you have to go to your upstream and start tracing those packets back through the WAN connections to find out where they come from. This is doable but not at all trivial. However, it you see enough evidence to think it's someone trying to take you down maliciously and not just goofing off a bit, it is definitely worth doing the backtrack to find them. This is a classic denial of service attack which is a computer crime in most states and under federal computer crime laws if the attacker is in another state from you. -george william herbert gherbert(a)crl.com

1 0