January 2001 - Test - lists.nanog.org

Re: History: lengthy outages
by Sean Donelan 25 Jan '01

25 Jan '01

On Thu, 25 January 2001, Randy Bush wrote: > >> UUNET cisco/bay router problems: November 7 1997 lasted 5 hours > > /vijay "remembering entirely too well" gill > > if this is 129/8, then i believe that major providers were down for almost > two days. That was a different one, but I didn't have that data readily available. The cisco/bay bug I was thinking of was when Cisco's kept reseting all their BGP sessions when they saw a particular announcement from Bay routers. There are a whole bunch of interesting problems I remember, but can't find the date. There was the Sprint "0.0.0.0" route problem. There was the 129/8 which happened shortly after AS7007. There was an ANS bug in the ENSSs which knocked them out. I tried not to include "malicious" acts, such as DDOS since I think they should be considered separately. Although I recognize often the underlying cause of the security issue may be related to operational error or bugs.

2 1

Incease in latency to TLD servers
by Sean Donelan 25 Jan '01

25 Jan '01

The Matrix (http://average.miq.net/Daily/markMM.html) is showing a remarkable rise in latency to many DNS servers. Has someone found a nifty new use for DNS servers?

2 1

Re: Microsoft spokesperson blames ICANN
by Patrick W. Gilmore 25 Jan '01

25 Jan '01

At 02:08 AM 1/25/2001 -0600, poptix(a)sleepybox.poptix.net wrote: > >www.microsoft.com is already a CNAME for www.microsoft.akadns.net, aka, >Akamai =) (Didn't I hear microsoft touting it's wonderful clustering/load >balancing features in W2k?) <simple statement of fact which everyone prolly already knows> Please note that if you cannot get a reply from the authoritative name server for [domain].[foo], you cannot be given the CNAME for a host. </simple statement of fact which everyone prolly already knows> (No comment on the reliability of Windows is intended.) > -poptix TTFN, patrick

2 1

RE: Tenure (was Re: MS explains)
by Scott Madley 25 Jan '01

25 Jan '01

I highly agree.. the Internet in purpose was founded for this dissemination of information, and gathering of knowledge. It's a shame that these days you normally can't find detailed evualuations of outage causes and resolutions unless behind the thick vail of an NDA. It's amazing some of the bugs that one finds in the course of work, that will probably never be brought into the light of the smaller networking public (small ISP's, small tech firms). =Scott On Thu, 25 January 2001, "Steven M. Bellovin" wrote: > In message <E14LpXc-0000ET-00(a)roam.psg.com>, Randy Bush writes: > >> I'd love to see a detailed description of what went wrong, and I hope > >> that those in the know will be allowed to post it or present it in > >> Atlanta. > > > >was it not don knuth's turing aware lecture which consisted of a detailed > >analysys of some recent bugs in his code? > > > We all learn much more from failures than from successes. To quote > Alfred E. Neuman, "Learn from the mistakes of others; you'll never live > long enough to make them all yourself." Donald Knuth has tenure. He can give all the lectures he wants about his greatest mistakes. Although all the "experts" say honesty is the best policy, the PR people at publically traded companies feel less is more. When the net was mostly operated and managed by academics, there was a reward in publishing the resultes of your mistakes. In today's net, you are likely to be punished or fired for publically discussing what happened. Ebay has never publically explained what went wrong with their equipment, Abovenet has never publically explained what went wrong with their router configuration, Worldcom has never publically explained what went wrong with their switches. Juniper and Cisco have been trading router interoperability bugs for a while, but they aren't publically discussed. Sometimes you might get a little information under NDA, but publish a "white paper?" Never! Admitly, there are exceptions. AT&T did release a fairly detailed analysis when they lost most of their 5ESSs switches due to a software bug. But it is so rare, we remember them because they are so rare.

1 0

Re: MS explains
by Steven M. Bellovin 25 Jan '01

25 Jan '01

In message <Pine.LNX.4.30.0101242038380.5951-100000(a)anime.net>, Dan Hollis writ es: > >On Wed, 24 Jan 2001, Eric A. Hall wrote: >> At 6:30 p.m. Tuesday (PST), a Microsoft technician made a configuration >> change to the routers on the edge of Microsoft's Domain Name Server >> network. >> At approximately 5 p.m. Wednesday (PST), Microsoft removed the changes to >> the router configuration and immediately saw a massive improvement in the >> DNS network. > >So basically, it took microsoft 23 hours to fix a router configuration. There's a story (possibly apocryphal) about the time that Steinmetz was called in as a consultant to repair problems with some massive piece of electrical machinery. After poking around for a while, some staring, and much thinking, he adjust one screw, and solved the problem. He then proceeded to write out a bill for $1000. The company was outraged. "$1000 for adjusting one screw? You're crazy!" Steinmetz agreed, took back the bill, and tore it up. He then wrote out a new bill: Adjust one screw $1 Knowing which screw to adjust $999 Remember the other half of Jim Duncan's post from last night: There were clearly some mistakes made, but it is also the case that there were a _lot_ of different things going on that contributed to the problem or complicated its resolution. He *worked* this problem; this is a first-hand statement, not conjecture by those who weren't there. Let me put forth a blatant generalization of my own: *all* major failures are due to complex causes. The proof is simple: if you're small and hence presumably clueless (the "mom and pop" ISPs another poster sneeringly referred to), your problems don't cause major failures for the rest of the net. If you're big (and hence presumably clueful), you solve the simple problems quickly and they don't become major failures. Finding and fixing *the* root cause is hard, when you're in the midst of a swamp full of other alligators, and you don't know which one is (currently) biting you in the rear. I'd love to see a detailed description of what went wrong, and I hope that those in the know will be allowed to post it or present it in Atlanta. But I'm willing to wager that it wasn't just (a) a single router configuration change, (b) brain-damage in Microsoft's DNS code, (c) malicious activity aimed at Microsoft, (d) RAMEN-induced misbehavior, or (e) any other single cause. --Steve Bellovin, http://www.research.att.com/~smb

3 2

Re: charitable organization for the shrinkage of the global routing table
by smd＠clock.org 25 Jan '01

25 Jan '01

So I have a bunch of replies, some of which suggest targetting the ISPs directly (phone bank of educated volunteers), some of which wondered why ISPs would care since non-aggregation is seen as good business sense in some cases. A couple replies asked if they could operate the cameras; one myopic confused soul suggested *I* model instead of Vint. And then there were a couple replies that indicated that we are *already* running into problems with some combination of numbers of routes, numbers of paths and numbers of implementation bugs (leaks in one implementation, pointer woes in another). That last point is serious. I humbly suggest that it's time to spam all the end users the NLANR cache researchers can identify as porn browsers with a simple message: Hot sexy babes want YOU to choose a provider who's a good aggregator! [With a URL pointing to someone's CIDR page, with an obligatory link to t&a or c&a or or s&m or s&g or ipv6 or whatever perverted stuff people spend alot of time reading about. Indeed, maybe we can enlist Betty Bowers and the folks at http://www.landoverbaptist.org so that we cover all the bases]. Sean.

1 0

RE: Microsoft spokesperson blames ICANN
by Roeland Meyer 25 Jan '01

25 Jan '01

Blaming it on ICANN, even indirectly, is about as clueless a move as can be imagined. ICANN has no direct authority over the root, the USG/DOC/NTIA has reserved that privilege for itself alone. They have let out the operations contract to NSI, who manages the a.root-servers.net and has been doing so for years. MHSC has been working with MS engineering and support staff, for the past few months, to integrate BIND8.2.2p7 servers with Win2K/DNS/AD. It isn't easy because of semantic inconsistency, radically diverse architecture concepts, and [above all] severe lack of documentation on MS part (as well as a few roaches in the SRV update stuff). >From our efforts, it is not at all surprising that someone, at MSFT, munged the DNS configuration, totally. Even their best guru could have done it, due to the murky nature of the config. I suspect that there are less than 100 ppl that could even have a clue, in this area, and they don't all have the same pieces of clue. Win2K DNS is self-consistent, BIND is self-consistent, they may be mutually consistent, but that has yet to be determined. MSFT works in a glasshouse with many of the panes painted-over, others are distorted. Similary, there is a tendency, among *nix folk, to discount anything MSFT (a mistake, IMHO). What's needed is for some(one/group) that has a though understanding of both systems, at the design level, to sort out the bits. This is properly, a development project, one that most system admins are unsuited for. Unfortunately, it is left at the system admin level, IMHO. -- ROELAND M.J. MEYER Managing Director Morgan Hill Software Company, Inc. TEL: +001 925 373 3954 FAX: +001 925 373 9781 http://www.mhsc.com mailto: rmeyer(a)mhsc.com > -----Original Message----- > From: Sean Donelan [mailto:sean@donelan.com] > Sent: Wednesday, January 24, 2001 11:40 AM > To: nanog(a)merit.edu > Subject: Microsoft spokesperson blames ICANN > > > > Microsoft appears to be blaming ICANN for the failure with Microft's > domain name servers (all located at the same place at Microsoft). > > Microsoft has yet to pin down the cause of the DNS error. "It can > be a system or human error, but somebody could also have done this > intentionally," De Jonge said. "We don't manage the DNS ourselves, > it is a system controlled by the Internet Corporation for Assigned > Names and Numbers (ICANN) with worldwide replicas." > > http://www.idg.net/ic_386962_1793_1-1681.html > > Microsoft gamers (users of www.zone.com) quickly came up with various > workarounds so they could continue playing. They were > posting HOSTS.TXT > files on various gamer bulletin boards overnight. In particular > "Asheron's Call" had several problems during the last week, including > banning a number of users and rolling back experience points due to > a game bug.

5 6

Re: Microsoft spokesperson blames ICANN
by Richard A. Steenbergen 25 Jan '01

25 Jan '01

On Wed, Jan 24, 2001 at 08:30:12PM -0500, Henry Yen wrote: > > try: http://secondary.easydns.com Super-redundant DNS hosting would make a good product. Akamai maybe? -- Richard A Steenbergen <ras(a)e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)

9 8

Tenure (was Re: MS explains)
by Sean Donelan 25 Jan '01

25 Jan '01

On Thu, 25 January 2001, "Steven M. Bellovin" wrote: > In message <E14LpXc-0000ET-00(a)roam.psg.com>, Randy Bush writes: > >> I'd love to see a detailed description of what went wrong, and I hope > >> that those in the know will be allowed to post it or present it in > >> Atlanta. > > > >was it not don knuth's turing aware lecture which consisted of a detailed > >analysys of some recent bugs in his code? > > > We all learn much more from failures than from successes. To quote > Alfred E. Neuman, "Learn from the mistakes of others; you'll never live > long enough to make them all yourself." Donald Knuth has tenure. He can give all the lectures he wants about his greatest mistakes. Although all the "experts" say honesty is the best policy, the PR people at publically traded companies feel less is more. When the net was mostly operated and managed by academics, there was a reward in publishing the resultes of your mistakes. In today's net, you are likely to be punished or fired for publically discussing what happened. Ebay has never publically explained what went wrong with their equipment, Abovenet has never publically explained what went wrong with their router configuration, Worldcom has never publically explained what went wrong with their switches. Juniper and Cisco have been trading router interoperability bugs for a while, but they aren't publically discussed. Sometimes you might get a little information under NDA, but publish a "white paper?" Never! Admitly, there are exceptions. AT&T did release a fairly detailed analysis when they lost most of their 5ESSs switches due to a software bug. But it is so rare, we remember them because they are so rare.

1 0

RE: OUTAGE: Earthlink - more nationwide difficulties
by Roeland Meyer 25 Jan '01

25 Jan '01

Oh yeah, they blamed it on PAIX. > -----Original Message----- > From: Roeland Meyer > Sent: Thursday, January 25, 2001 11:00 AM > To: 'Sean Donelan'; nanog(a)merit.edu > Subject: RE: OUTAGE: Earthlink - more nationwide difficulties > > > DSLnetworks (my lvrmr upstream) had outage (redback router) > from 2300 til 0900 this morning.<sheesh!> > > Can we say "PO'd moi"! > > > -----Original Message----- > > From: Sean Donelan [mailto:sean@donelan.com] > > Sent: Thursday, January 25, 2001 10:49 AM > > To: nanog(a)merit.edu > > Subject: OUTAGE: Earthlink - more nationwide difficulties > > > > > > > > Earthlink appears to have either a new, or continuing > routing problem > > affecting much of their network. > > > > > > >

1 0