April 1997 - Test - lists.nanog.org

Re: seeing the trees in the forest of confusion
by Sean Donelan 26 Apr '97

26 Apr '97

> The source was isolated within 60 minutes of the problem. > > The routes propagated for greater than 120 minutes after that > without the source contributing. (yes, the routes with an > originating AS path of 7007) > > This is the interesting problem that noone seems to focus on. This has happened before. I had a problem with routes incorrectly being advertised by another ISP(GI/MIDNET). The problem was quickly identified, and corrected. An old set of filters created by an engineer that had left the company was incorrectly configured. However this ISP had transit from Sprint. A Sprint router a couple of hops away kept advertising the bogus routes for two days before I could finally get someone in Sprint to believe a problem existed, and shoot the router. Why it took two days is a story in itself. > We did not expect it to hurt for 3 hours. It should have stopped > earlier. Why it didn't is the only interesting question left. Perhaps if you had shared the information more widely, someone who had seen a similar problem before could have helped. -- Sean Donelan, Data Research Associates, Inc, St. Louis, MO Affiliation given for identification not representation

1 0

Also, re: 7007
by Avi Freedman 26 Apr '97

26 Apr '97

Net Access, AS 4969, is now announcing: mae-netaxs#sho ip bgp reg 6082 BGP table version is 8459301, local router ID is 207.106.124.6 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal Origin codes: i - IGP, e - EGP, ? - incomplete Network Next Hop Metric LocPrf Weight Path *> 205.147.160.0/19 192.41.177.74 1 0 6082 ? *> 207.152.191.0 192.41.177.74 1 0 6082 ? *> 209.50.192.0/19 192.41.177.74 1 0 6082 ? mae-netaxs#sho ip access 68 Standard IP access list 68 permit 205.147.160.0, wildcard bits 0.0.31.255 permit 209.50.192.0, wildcard bits 0.0.31.255 permit 207.152.191.0, wildcard bits 0.0.0.255 permit 199.233.96.0, wildcard bits 0.0.0.255 deny any For ASN 6082 (MAI). This is a subset of routes to take care of some of their customers at another location. We're filtering their inbound routes and not announcing anyone's routes to them, FYI. Please don't filter _6082_. We're only announcing routes through distribute-lists, and MAI promises to be very careful with what they announce through 6082. MAI is still working with Bay at their data center to reach the point that they're comfortable re-advertising their routes directly. Ciscos have been offered to MAI, but MAI is still working with Bay. Avi

2 2

RE: Read this...Re: offtopic for NANOG - do not read
by Carl Oppedahl 26 Apr '97

26 Apr '97

At 03:50 PM 04/26/97 -0500, Jim Fleming wrote: >On Saturday, April 26, 1997 12:54 PM, Carl Oppedahl[SMTP:carl@oppedahl.com] wrote: >@ Thus, it seems likely that (barring a shift in control over .COM) NSI would >@ continue forever, carrying out its awful policy and putting innocent domain >@ name owners out of business. >@ >@ The only way that innocent domain name owners will have the cloud of NSI's >@ awful domain name policy lifted from their heads is if indeed the control >@ over .COM shifts elsewhere. So far as I can see, the most likely way for >@ that to happen is if IAHC's plans move forward and are put into place. > >People and companies can now register in >other domains than .COM. > >Why would a company want a .COM domain >when they can have something more expressive ? I guess you don't get it. Once a company has invested years in its business position in reliance on having a particular domain name, has invested years and energy and money getting people to put the company's URL into their bookmarks and getting people to set up links to the company's URL, then switching to some other domain name is out of the question. Similarly, if a company is an ISP with thousands or millions of customers whose email address ends with the company's domain name, then the company simply cannot change domain names. It is out of the question. Some of the companies I am thinking of signed up for their domain name *prior* to the start of NSI's five-year contract. Many of the companies I am thinking of signed up for their domain name prior to the July 28, 1995 start date of NSI's awful policy. Yet they are stuck with NSI's wacko handling of trademark challenges. It is all fine and good to talk about the domain name one *might* choose if starting a new business a year from now. But I am talking about present members of the Internet community whose businesses are subject to NSI's whim with respect to domain names. And whose businesses are subject to the truly scary prospect of NSF scuttling under the furniture in April of 1998, leaving NSI answering to no one, charging whatever it pleases for continued use of a domain name, and developing new and even worse ways of handling trademark challenges. For all these reasons, control of .COM has to get shifted away from NSI. And it is interesting but irrelevant to talk of other TLDs that may some day exist.

1 0

RE: NEWDOM: Re: offtopic for NANOG - do not read
by Carl Oppedahl 26 Apr '97

26 Apr '97

At 02:59 PM 04/26/97 -0500, Jim Fleming wrote: >On Saturday, April 26, 1997 11:49 AM, Carl Oppedahl[SMTP:carl@oppedahl.com] wrote: >@ At 01:46 PM 04/26/97 -0500, Karl Denninger wrote: >@ ><snip> >@ >@ >I'd like you to point out the major corporations and public universities who >@ >will do this. I'd also like you to immediately return that nice root server >@ >that NSI has paid for in part or whole, if you really believe this. >@ NSI has never paid for anything. NSI has doled out money collected by it >@ from NSF and from domain name owners. NSF (i.e. the US taxpayers) and the >@ COM, NET etc. domain name owners have paid for whatever it is you think NSI >@ has paid for. >Carl, >This is not quite true. >Paul Vixie has stated on the NANOG list that NSI >pays for his equipment. I have no doubt the check was from NSI. I am talking about where the money actually came from, and the answer is either (1) NSF (i.e. the US taxpayers) or (2) the domain name owners paying their $100 or $50 fees. >Paul is part of the Internet >Software Consortium (ISC) <http://www.ispc.org> >an IRS approved 501(c) company that accepts >donations so that Paul can write software that >supports the domain name system. > >Their web site says they have a $700,000/year budget. > >According to the IRS, you can get full-disclosure >on all of this. So what? Nobody doubts that the check Paul Vixie received was written by NSI -- if that's what he says, I believe him. No amount of full disclosure under 501(c)(3) is necessary to learn whose name was on the check. What I am talking about is the source of the money. NSI got it from you and me when we paid our $100 and our $50, or when they collected it from NSF (and thus from you and me).

1 0

Re: seeing the trees in the forest of confusion
by Russ Haynal 26 Apr '97

26 Apr '97

At 09:22 AM 4/26/97 -0500, Alan Hannan wrote: > This wasn't an Internet outage, this wasn't a catastrophe. It was > a brownout that sporadically hit providers at various strengths. > At least one NSP measured their backbone load go down by only 15% > during the incident. To go along with that... The MAE-EAST statistics (graph) also shows only a modest drop in traffic for a little over an hour... http://www.mfsdatanet.com/MAE/east.giga.overlay.html Russ __________________________________________________________________ Russ Haynal - Internet Consultant, Instructor, Speaker "Helping organizations gain the most benefit from the Internet" russ(a)navigators.com http://www.navigators.com 703-729-1757 ------------------------------------------------------------------ Author:"Internet; A Knowledge Odyssey" (Top-rated CD-ROM Tutorial) Available from MindQ Publishing: http://www.mindq.com

1 0

Re: Ascend GRF400
by Tony Li 26 Apr '97

26 Apr '97

> Correct me if I am wrong, but I thought the GRF-400 was an Ascend product > and the GRF-1600 is the NetStar GigaRouter, with some additional > developments. > > The GRF-400 is out there in networks like AGIS, NetRail, PSI, DOMINO and > Savvis. No word on any NetStar-cum-GRF1600 deployments. > > Am I right? My understanding is that the GRF-400 is just a 4 slot version of the GRF-1600, modulo cosmetic differences, branding etc. Certainly this is one of the easiest ways that one can take a crossbar switch and produce a cost reduced, bandwidth reduced midrange. Tony p.s. Do I hear a chainsaw in Minneapolis? ;-) ;-) ;-)

1 0

Now comes the news.
by Christian Nielsen 25 Apr '97

25 Apr '97

There are already 2 articles on the net about it http://www.news.com/News/Item/0,4,10083,00.html?latest http://www.wired.com/news/technology/story/3442.html and I am sure there are more to come. It seems too easy for one person/company to bring down the net. Yes, we all agree it was some kind of accident, one that wasn't supposed to happen BUT I have to ask, where were all the network people that should have caught this before it hit the internet full force. Yes, there is talk about having the RAs setup and to prevent this type of thing, but not much else. We put our stuff in the RA when we remember to but I don't think that many people look much at them anyways. Hopefully next time this can be stopped before long, like maybe within 10 - 20 mins. Maybe it is time for all companies, even the smaller ones, or those that use BGP to have and really USE a 24 hour noc. Where there are people there that really know about routers and BGP. Maybe as the internet grows, those making it grow need a more active role. Sorry, Voice mail doesn't cut it, or a busy number when the internet comes down.... Christian

1 0

RE: Wow, AS7007!
by Dave Van Allen 25 Apr '97

25 Apr '97

>-----Original Message----- >From: Stephen A Misel [SMTP:stevem@hway.net] >Sent: Friday, April 25, 1997 12:53 PM >To: nanog(a)merit.edu >Subject: Wow, AS7007! > >I happened to be in one of our 7505 routers this afternoon when POP -- all >of a sudden most of the internet disappeared! I immediately thought it was >me, but looked around and saw this AS7007 broadcasting MY routes! > >[...] > >Correct me if I'm wrong, but: > > (1) We're going to read about this in EVERY computer magazine, newspaper >and TV as "the end of the internet?" > >Probably. It's newsworthy in that it punctuates the statement "Nearly anyone >with a BGP router in hand can instantly core-dump the global routing tables" > > (2) Access lists by backbone providers *should* have prevented this. > >Mostly. An ISP, whether large or small that BGP's with customers can indeed >do distribute ACL's both on AS heard, and routes learned, including masks. >You can easily re-announce or announce only what you want, or not announce or >re-announce routes that are inconsistent with your policy or ACL's. > > (3) Does or does not the RADB and other routing registries (MCI's, etc) >prevent this? > >It helps, but all you need are a few ingress' that do not filter and you can >pollute enough of the core to hose it very nicely indeed. > >I bet this hole will be patched up real soon! > >I don't think so. I'm not sure that this is as much a "hole" as it is a >relationship and trust issue. Right now, when things go OK, the routing >policies on Net work pretty well. Unarguably, they need refining, but >all-in-all the Net still relies mostly on trust, as it has from the >beginning. If we simply take all trust away, then the current topology would >not work, and may not be able to be made to work quickly enough, without even >more disasters. > >This exact thing has happened before, and potentially will happen again >because all it can take is one typo under 'router bgp xxxxx' at the right >place, in the right network, and the Internet can go quickly to /dev/null. >This is the trust factor. We all rely on the fact that router-jocks won't >typo, will filter where appropriate, and will educate rookies prior to >whispering the enable passwd to them. > >A few things would help, IMO - All BGP should be authenticated, and all >neighbors should be ACL'd. > >Now after spending 4 hours announcing more specifics to cover the bogon >routes so we could play Internet today for a bit, it's time to be a >good-netcitizen and see if I can't re-CIDR myself. Then it's off to the >Scotch locker! :-) > > >Best regards, > >Dave Van Allen - You Tools Corporation/FASTNET(tm) >dave(a)fast.net (610)289-1100 http://www.fast.net >FASTNET - PA/NJ/DE Business Internet Solutions >

2 1

7007: FROM THE HORSE'S MOUTH
by Avi Freedman 25 Apr '97

25 Apr '97

MAI's view: Here's a status on what AS 7007 themselves saw and think happened. They're disconnected from the 'net now, so they can't post this themselves. 7007 is an ASN used by mai.net, MAI Network Services. They also use ASN 6082. Their topology at their new data center is: [asn 6082] Bay BLN [fddi] -> mae-east, where they peer with some and have mae-transit from Sprintlink [1239] [asn 7007] Bay BLN [t3] -> sprintlink [1239 1790] [t3] -> customer X And the two BLNs are connected to each other via 100bt. Customer X announced something like a full route table, according to MAI, and MAI was not filtering on either AS_PATH or by distribute_list (or whatever Bay calls them) on the routes. MAI apologizes for this, and acknowledges that if they had done this, the major problem could have been avoided. At about 11:30am (this is when we saw it), they started redistributing more specifics for many (thousands) of CIDR routes to Sprintlink [1239 1790] but the routes were announced - AND the routes were injected with an origin AS of 7007, hiding the AS_PATH the routes had (and the customer involved). MAI noticed this at about 11:45. MAI shut down the Sprintlink connection and the same thing happened again. 72,000 is the number of routes the Sprint router took before it melted, but the Bay might have been willing to generate even more than that. Anyway, Sprintlink saw the routes again from MAI (the de-aggregated specifics, with an AS_PATH of ^7007$). Then they rebooted the router and the 7007 router was not advertising the specifics or hiding AS_PATH info, but the damage was done, and the more specifics were still out there for some reason. At about 12:15 they shut off all of their routers; shortly after Sprintlink shut down the T3 for good measure. MAI does NOT think that they were distributing the BGP routes into an IGP and then readvertising them on that basis; they think there was/is a Bay BGP bug that caused this to happen. MAI's NOC # is 888 624 8700. Vincent Bono of MAI wishes to apologize for the trouble; a Bay tech is on the scene in DC working on the problem to make sure that when they come back up, nothing like this will happen again. An outside view of what happened: At 11:30, I noticed that we lost connectivity to the world. Some of our customers called our NOC - and our dual-homed customers said that they saw hundreds of routes from ASN 7007. One of the customers noticed that the ASN 7007 routes were stomping on their OSPF routes (because the BGP routes from _1239 1790 7007$ were more specific than some of their internal /23 OSPF routes). And ASN 7007 We saw about 60k routes in our core routers at the time, and saw thousands of routes from 7007 when we looked more carefully. We kept clearing sessions to filter 7007 but the routes kept popping back up: Sprint (of course), UUNET, and MCI all had them. Also, we had to advertise some of our customers more specifically because the dampened 7007 routes (because everyone else was clearing at the time) were screwing their connectivity. I know that some others did this, so tomorrow the tables will have a bunch of more specifics, I suppose. A note: If all more specific routes for a destination are dampened, exsting more general routes will not be looked at. If this behavior was changed, some of the blackholing that went on today would not have been possible. But this is a separate topic for discussion. I suppose that the immediate topic will be route filtering vs. AS_PATH filtering... Thanks, Avi Freedman (the original) Net Access (netaxs.net)

2 1

RE: Time frame..
by Wayne Bouchard 25 Apr '97

25 Apr '97

Based on my (now) mucked up traffic graphs, I'd say it started at about 8:30am PDT and I saw the last of it about 9:45. ---------------------------------------------------------------------- Wayne Bouchard GlobalCenter web(a)primenet.com Primenet Network Engineering Internet Solutions for (602) 416-6422 800-373-2499 x6422 Growing Businesses FAX: (602) 416-9422 http://www.primenet.com http://www.globalcenter.net ----------------------------------------------------------------------

2 1