- NANOG - lists.nanog.org

Re: MS explains
by Steven M. Bellovin 25 Jan '01

25 Jan '01

In message <E14LpXc-0000ET-00(a)roam.psg.com>, Randy Bush writes: >> I'd love to see a detailed description of what went wrong, and I hope >> that those in the know will be allowed to post it or present it in >> Atlanta. > >was it not don knuth's turing aware lecture which consisted of a detailed >analysys of some recent bugs in his code? > We all learn much more from failures than from successes. To quote Alfred E. Neuman, "Learn from the mistakes of others; you'll never live long enough to make them all yourself." --Steve Bellovin, http://www.research.att.com/~smb

1 0

RE: Microsoft spokesperson blames ICANN
by Tim Wolfe 25 Jan '01

25 Jan '01

Microsoft.com is again responding here. I had to point this out. On the main page (www.microsoft.com) under Today's News (which conveniently makes no mention of the fact that they've been offline for a day) They have this headline for a story, "Microsoft CEO Steve Ballmer explains how Microsoft software provides the business agility that customers need to stay ahead." Can you say irony? I bet Steve Ballmer can... --Tim ============================================= Timothy M. Wolfe CCNA, NSA Sr. Security Engineer tim(a)ignw.com InfoGroup Northwest 541.485.0957 x108 ============================================= -----Original Message----- From: woods(a)weird.com [mailto:woods@weird.com] Sent: Wednesday, January 24, 2001 3:01 PM To: nanog(a)merit.edu Subject: RE: Microsoft spokesperson blames ICANN [ On Wednesday, January 24, 2001 at 13:09:45 (-0800), Roeland Meyer wrote: ] > Subject: RE: Microsoft spokesperson blames ICANN > > From our efforts, it is not at all surprising that someone, at MSFT, munged > the DNS configuration, totally. Even their best guru could have done it, due > to the murky nature of the config. I suspect that there are less than 100 > ppl that could even have a clue, in this area, and they don't all have the > same pieces of clue. That's absolutely idiotic (of M$, that is !;-). Even more idiotic than putting all their nameservers in one basket, so to speak. I'd bet any high-school kid who had any experience whatsoever at installing Linux or FreeBSD could no doubt blow a real OS and a native BIND install onto any sufficiently capable set of four machines in about an hour or so and provided that someone could cough up at least a half-baked zone file from somewhere to load on them they'd all be online and answering to the registered nameserver IP numbers in no time flat. Certainly in less than what's apparently going to be at least 23 hours now! Heck I know a half dozen or more people around the world who would have put their dislike of M$ away for a short period and loaded a zone file or two on their own nameservers for M$ if only M$ could have managed to get the .COM zone updated with new delegations.... What ever happened in this community to asking the community for help when you're caught between a rock and a hard place? (Not that a company the size of M$ should have to ask for a handout -- they no doubt have significant IP connectivity in as many places around the world as almost anyone else!) MS has nothing and no-one to blame but their own stupidity and arrogance in this. Meanwhile they're so damn big and "important" to so many users that this outage is having both a direct and an indirect negative impact on a lot of ISPs around the world! "Hey! The Internet must be broken if I can't get to M$.COM!" > What's needed is for some(one/group) that has a though understanding of both > systems, at the design level, to sort out the bits. This is properly, a > development project, one that most system admins are unsuited for. > Unfortunately, it is left at the system admin level, IMHO. No, what's needed is for M$ to learn that they need to deploy software that's capable of the task even if it didn't come from a box and doesn't have their logo branded on it. Squishing things together that were never meant to be squished together is only going to cause a big mess. Err, has already caused a big mess, at least for M$ and those who deal with them! ;-) They'd also do well to learn a bit about network geography and just exactly how authoritative nameserver visibility from various locations on this wonderful Internet of ours can directly affect their bottom line! -- Greg A. Woods +1 416 218-0098 VE3TCP <gwoods(a)acm.org> <robohack!woods> Planix, Inc. <woods(a)planix.com>; Secrets of the Weird <woods(a)weird.com> PS: they're still down for the count from this neck of the woods! Interestingly one of them's completely unreachable from here now... $ host -C microsoft.com microsoft.com NS DNS4.CP.MSFT.NET Nameserver DNS4.CP.MSFT.NET not responding microsoft.com SOA record not found at DNS4.CP.MSFT.NET, try again microsoft.com NS DNS5.CP.MSFT.NET Nameserver DNS5.CP.MSFT.NET not responding microsoft.com SOA record not found at DNS5.CP.MSFT.NET, try again microsoft.com NS DNS7.CP.MSFT.NET Nameserver DNS7.CP.MSFT.NET not responding microsoft.com SOA record not found at DNS7.CP.MSFT.NET, try again microsoft.com NS DNS6.CP.MSFT.NET Nameserver DNS6.CP.MSFT.NET not reachable microsoft.com SOA record not found at DNS6.CP.MSFT.NET, try again

2 1

The unwashed and unaggregated
by Patrick Greenwell 25 Jan '01

25 Jan '01

What about using "sane" route aggregation(or lack thereof) as one of the determining factors in requests to receive additional address allocation from the IP registries? Good idea, bad idea?

2 1

RE: From Microsoft's site
by Roeland Meyer 25 Jan '01

25 Jan '01

> At 6:30 p.m. Tuesday (PST), a Microsoft technician made a > configuration > change to the routers on the edge of Microsoft's Domain Name Server > network. The DNS servers are used to connect domain names with numeric > IP addresses (e.g. 207.46.230.219) of the various servers and networks > that make up Microsoft's Web presence. Connect 22.5 hours worth of dots between these two events. > At approximately 5 p.m. Wednesday (PST), Microsoft removed the changes > to the router configuration and immediately saw a massive > improvement in the DNS network. Their management should be real embarrassed to take so long to back out the last-change.

4 3

Re: What is going on in the PAC NW (Microsoft Outage)
by Hank Nussbacher 25 Jan '01

25 Jan '01

At 11:35 24/01/01 -0800, Tony Rall wrote: > >Is there any information available about Microsoft and MS related >companies > >being unavailable > >I'm seeing about 50% packet loss to all 4 microsoft.com nameservers. I >can't get any dns responses from them. The following is all hypothesis: Reading the MS announcement that says "At 6:30 last night, a configuration change was made by a Microsoft technician to the routers on the edge of Microsoft's DNS network", and adding in the 50% packet loss you list above, it looks like the MS technician installed some sort of Cisco TCP/UDP rate-limit on some interface along the data path. -Hank

1 0

Labeling and naming
by Kurt Erik Lindqvist 25 Jan '01

25 Jan '01

For a project I am currently working on I stumbled upon the following. What is the best way to lable and name equipment? Although this applies to all equipment such as SDH ADMs, IP, ATM etc I realised that it seems to be hardest to find a sensible convention for IP equipment. Preferably I would like to find a convention that fits all, but I guess that is utopia. So, since list contains, PTTs, Telcos, ISPs and wannabees is there any good common scheme or pointers to something useful? - kurtis -

5 6

History: lengthy outages
by Sean Donelan 25 Jan '01

25 Jan '01

On Wed, 24 January 2001, jlewis(a)lewis.org wrote: > Somebody bitched a router config, and it took 22.5 hours to figure it out? > That's the sort of goof you might expect from a mom and pop ISP with a > hundred customers and virtually no IP clue. I'll be shocked if multiple > people (at multiple levels) aren't fired over this. Screwing up happens. > Taking this long to figure out what you (or even for others to figure out > what someone else) screwed up is just absolutely unbelievable. That's a bit unfair. There have been a number of lengthy outages. AS7007 router configuration problem: April 25 1997 lasted 2 hours AOL (ANS router configuration problem): Aug 7 1996 lasted 19 hours ATT frame-relay switch errors: April 13, 14 1998 lasted 26 hours BBN standard power failure: October 11, 1996 lasted about 12 hours(off and on) NETCOM router configuration error: June 20 1996 lasted 13 hours Sprint database problems: September 3 1996 lasted 5 hours NSI root server corruption (operational error): July 16 1997 lasted 4 hours PacBell configuration problems: January 30-31 1997 lasted 48 hours UUNET frame-relay problems: July 1 1997 lasted over 24 hours UUNET cisco/bay router problems: November 7 1997 lasted 5 hours Worldcom frame-relay switch errors: August 1999 lasted 9-10 days If I skipped your favorite provider, have no fear. More than likely they've also had (or will have) a lengthy outage at one time or another. One interesting thing I found was providers seem to have substantial outages shortly after announcing "100% Uptime." But somehow the net continues to stumble forward.

4 3

peer "sanity" filters - best practices?
by David P. Maynard 25 Jan '01

25 Jan '01

Several times over the past few years, the NANOG list has discussed the topic of if or how non-transit peer BGP announcements should be filtered. I searched through the archives, but couldn't find where anyone had published a summary of best practices for filtering announcements from peers that aren't one of the "top N" (for some debatable value of N) NSPs. What I am looking for is a set of filters that implement a sanity check to catch some of the recurring BGP config problems. For example, if SmallTime Internet suddenly starts announcing that they provide transit for UUNET, I don't want there to be any chance that I will start sending my UUNET-bound traffic to SmallTime. Some of the "rules" I can think of would be: - peers should not be announcing transit routes from the top N transit providers (implemented via AS filter-list blocking the top transit ASNs) - peers should not be announcing RFC1918 networks (implemented by prefix-list) - peers should not be announcing the same networks that "I" originate - peers should not be announcing a default route - peers should not announce more than n-thousand routes (implemented using the Cisco 'maximum-prefix' option) These rules would theoretically be applied to any peers who weren't part of the "top N." In practice, I'm thinking of peers that announce 1 to a few thousand prefixes. Is the idea even feasible? If so, what other rules make sense? One other option would be to limit announcements to /24 and shorter. Is there any agreement on the "top N" ASNs that "normal" peers shouldn't be announcing? Thanks. -dpm P.S. My interest is not just academic. These filters will get installed on one of my customers' networks. They have had a fair number of problems with peers announcing routes that they shouldn't and I am looking for ways to help reduce the problem. -- David P. Maynard, Flametree Corporation Network Infrastructure and Monitoring Solutions EMail: dpm(a)flametree.com, Tel: +1 512 670 4090, Fax: +1 512 670 4091 --

3 2

Re: What is going on in the PAC NW (Microsoft Outage)
by Steven M. Bellovin 25 Jan '01

25 Jan '01

In message <OFF5C3FF78.0778E224-ON882569DE.006B3A50@LocalDomain>, "Tony Rall" w rites: > > >>Is there any information available about Microsoft and MS related >companies >>being unavailable > >I'm seeing about 50% packet loss to all 4 microsoft.com nameservers. I >can't get any dns responses from them. > >As, I think, Greg Woods mentioned, it's a pretting weak setup putting all >of your nameservers on what appears to be the same network. > >I am amazed that this has taken so long to fix. Any chance their nameservers are being DDoSed? --Steve Bellovin, http://www.research.att.com/~smb

5 4

Re: Labeling and naming
by Michael Shields 25 Jan '01

25 Jan '01

In article <NEBBLMDAGLPOAGJMOFHAMEKCDBAA.karwas(a)ifxcorp.com>, "Przemyslaw Karwasiecki" <karwas(a)ifxcorp.com> wrote: > <2 letters of ISO country code> > http://www.bcpl.net/~jspath/isocodes.html > <3 letters of airport city code> > http://www.ufreight.com/faq/airport_code/airport_code_by_ac.html You don't need to rely on secondary compilations for these: http://www.din.de/gremien/nas/nabd/iso3166ma/ http://www.iata.org/codes/ -- Shields.

2 1