In message <Pine.LNX.4.30.0101242038380.5951-100000@anime.net>, Dan Hollis writ es:

On Wed, 24 Jan 2001, Eric A. Hall wrote:

...

At 6:30 p.m. Tuesday (PST), a Microsoft technician made a configuration change to the routers on the edge of Microsoft's Domain Name Server network. At approximately 5 p.m. Wednesday (PST), Microsoft removed the changes to the router configuration and immediately saw a massive improvement in the DNS network.

So basically, it took microsoft 23 hours to fix a router configuration.

There's a story (possibly apocryphal) about the time that Steinmetz was called in as a consultant to repair problems with some massive piece of electrical machinery. After poking around for a while, some staring, and much thinking, he adjust one screw, and solved the problem. He then proceeded to write out a bill for $1000. The company was outraged. "$1000 for adjusting one screw? You're crazy!" Steinmetz agreed, took back the bill, and tore it up. He then wrote out a new bill: Adjust one screw $1 Knowing which screw to adjust $999 Remember the other half of Jim Duncan's post from last night: There were clearly some mistakes made, but it is also the case that there were a _lot_ of different things going on that contributed to the problem or complicated its resolution. He *worked* this problem; this is a first-hand statement, not conjecture by those who weren't there. Let me put forth a blatant generalization of my own: *all* major failures are due to complex causes. The proof is simple: if you're small and hence presumably clueless (the "mom and pop" ISPs another poster sneeringly referred to), your problems don't cause major failures for the rest of the net. If you're big (and hence presumably clueful), you solve the simple problems quickly and they don't become major failures. Finding and fixing *the* root cause is hard, when you're in the midst of a swamp full of other alligators, and you don't know which one is (currently) biting you in the rear. I'd love to see a detailed description of what went wrong, and I hope that those in the know will be allowed to post it or present it in Atlanta. But I'm willing to wager that it wasn't just (a) a single router configuration change, (b) brain-damage in Microsoft's DNS code, (c) malicious activity aimed at Microsoft, (d) RAMEN-induced misbehavior, or (e) any other single cause. --Steve Bellovin, http://www.research.att.com/~smb