January 2001 - Test - lists.nanog.org

Microsoft DNS outage
by Joseph T. Klein 25 Jan '01

25 Jan '01

It looks like the Microsoft name servers lack any sort of network diversity. If they had servers in geographically and network diverse location perhaps this outage would not have occurred. Diversity is the best protection against systemic failure. An important lesson is contained herein. -- Joseph T. Klein jtk(a)titania.net +1 414 915 7489 "Omnium rerum principia parva sunt." - Marcus Tullius Cicero

3 2

Microsoft Explains Site Access Issues
by Sean Donelan 25 Jan '01

25 Jan '01

Since Microsoft's lawyers object to quoting statements from their website, I'll paraphrase. You can read the original at http://www.microsoft.com/info/siteaccess.htm for however long Microsoft leaves that page up. Microsoft reports a Microsoft technician made a change to a router (brand unknown :-) configuration disrupting traffic to the subnet containing essentially all of Microsoft's name servers. Microsoft classified the incident as "operational error." Note: I'm happy Microsoft says operational error, and not operator error. While there are some true operator errors, there are usually other factors involved and the operator just happened to be the one found at the scene of the crash. Microsoft also says the problem was not with any of their or any third-parties products. Wow, I didn't think I picked up the start of the outage that fast, 6:30pm. I thought it would have been building for at least a few hours. After 22.5 hours, Microsoft removed the change from the router's configuration, and saw an improvement. Hopefully, I'll receive a response to my inquiry to the msnnoc last night soon.

1 0

Re: Microsoft spokesperson blames ICANN
by Steven M. Bellovin 25 Jan '01

25 Jan '01

In message <200101250230.VAA06726(a)rtp-msg-core-1.cisco.com>, Jim Duncan writes: > >Sean Donelan writes: >> Microsoft appears to be blaming ICANN for the failure with Microft's >> domain name servers (all located at the same place at Microsoft). >> >> Microsoft has yet to pin down the cause of the DNS error. "It can >> be a system or human error, but somebody could also have done this >> intentionally," De Jonge said. "We don't manage the DNS ourselves, >> it is a system controlled by the Internet Corporation for Assigned >> Names and Numbers (ICANN) with worldwide replicas." >> >> http://www.idg.net/ic_386962_1793_1-1681.html > >I have read that article many, many time today, trying to see how you >came to that conclusion and I don't get it. To reach that conclusion, >you've clearly quoted them out of the context of the larger article. >Even to reach that conclusion from the small part you quoted requires a >logical leap that is inappropriate, if not outright incorrect. Sorry, Jim; I think it's not that much of a stretch. They said that (a) it's a DNS problem, (b) they don't understand the cause, but (c) they don't manage the DNS, ICANN does. OK -- the problem is therefore in a piece they don't manage, so they're not at fault. But ICANN *does* manage it (or so the direct quote says). There's a decent implication there that the manager is at fault, though not (of course) a direct statement. I would also note that the article quotes De Jonge as saying "The *Internet's* Domain Name System (DNS) does does not return the correct response when it is queried for a Microsoft Web site" [emphasis added]. In other words, it's not *Microsoft's* DNS servers, it's the "Internet's". I know you worked hard on this, and I understand that at the time of this article, very little was understood about the root cause. (And I'm not at all surprised to hear that many different things contributed.) But that paragraph (and the additional sentence I quoted) are, at best, misleading, and can easily be read in the way that Sean read it. Maybe the guy was tired, maybe there was a language barrier, maybe the reporter misunderstood something (though there's a lot less scope for that in direct quotations). I read it the same way that Sean did. --Steve Bellovin, http://www.research.att.com/~smb

2 1

RE: Microsoft spokesperson blames ICANN
by Roeland Meyer 25 Jan '01

25 Jan '01

> Sorry, Jim; I think it's not that much of a stretch. They said that > (a) it's a DNS problem, (b) they don't understand the cause, but (c) > they don't manage the DNS, ICANN does. OK -- the problem is > therefore > in a piece they don't manage, so they're not at fault. But ICANN > *does* manage it (or so the direct quote says). There's a decent > implication there that the manager is at fault, though not > (of course) > a direct statement. I would also note that the article > quotes De Jonge > as saying "The *Internet's* Domain Name System (DNS) does > does not return > the correct response when it is queried for a Microsoft Web site" > [emphasis added]. In other words, it's not *Microsoft's* DNS > servers, > it's the "Internet's". You overlooked the prime facae error. ICANN most certainly does NOT manage the DNS. The US Department of Commerce does. That's actually been proven in court.

1 0

RE: Great job AT&T (re: Microsoft problems)
by Roeland Meyer 25 Jan '01

25 Jan '01

> From: Steve Sobol [mailto:sjsobol@NorthShoreTechnologies.net] > Sent: Wednesday, January 24, 2001 8:13 PM > > In response to what I said, > > > > topaz.nstc.com used to run Microsoft DNS. After a while, > having tried > > > repeatedly to get MSDNS to actually *serve* the zone files I > > > had set up, and having not been successful, I switched to Bind 8 > > > and have had no problems since. > > > > > > You might want to try Bind 8. > > Roeland Meyer replied: > > > The problem is that a PDC requires SRV updates on the zone. Only > > BIND-8.2.2p7 can do this and it's not clear that it runs > sufficiently well > > (consider that feature as beta verging on alpha, IMHO). > Certainly, the > > configuration is as clear as mud... way too much magic. > > That's only a problem if you're running a PDC on your LAN. > > I personally am not. However, I guarantee you that MSFT is and that their internal DNS is Win2K Active Directory DNS and that they have at least one PDC. In fact, they've bragged about how they finally were able to bring HotMail up on Win2Kdatacenter. This means that they HAD to deploy Win2K/DNS. > > My info says that MSFT only tested sufficiently with Win2K > DNS. Integration > > with BIND was not considered especially important. This, in > spite of the > > trip Vix made to NSFT, Summer '99 (where the SRV fixes came > from). It is > > supposed to work, but the config is a nightmare of > exceptions and special > > cases. > > But again, that's a Microsoft issue. I think we are all agreed that it is probably an MSFT issue. But, your suggestion, of MSFT deploying BIND8, is not workable. > I will give Microsoft points for making their DNS server > really easy to > configure. I will acknowledge not only that Bind is harder to > configure > for someone who doesn't know it, but has the problems Roeland > mentions. > Bind ain't perfect either, but it works, and it isn't produced by an > organization that thinks that Its Way is the Only Way(tm). I just deployed a root-server cluster, on COL Linux, bind-8.2.2p7. I am able to leave it with them because I ALSO installed WebMin-0.83, on a seprate IP addr, for administration. I just thank God that they haven't migrated to Win2K yet. > Microsoft has a large enough market share that I feel that it is their > obligation to make sure that their Internet products play nice with > the rest of the Net. Regardless of the fact that I dislike > their business > practices, if they'd just work harder towards that end, I'd > complain about > them a lot less than I now do. Agreed, whole heartedly! ref: HALLOWEEN I & II, for speculation of why they don't play nice.

1 0

Re: Monitoring highly redundant operations
by Henry Yen 25 Jan '01

25 Jan '01

On Wed, Jan 24, 2001 at 09:32:04AM -0600, Bill Fumerola wrote: > On Wed, Jan 24, 2001 at 08:09:19PM -0500, Henry Yen wrote: > > > > On Wed, Jan 24, 2001 at 02:31:20AM -0800, Sean Donelan wrote: > > > Not to pick on Dave, since I suspect he is going to have to face > > > the Microsoft PR department for re-indoctrination for speaking out > > > of turn, I'm glad to see someone from microsoft made an appearance. > > > > out of curiosity, how do you know he's really from microsoft, > > whether unofficial or not? > > (he might be one of those LIENUX ZeAlOtS.) > > > > perhaps he might face re-indoctrination over the mail client he > > is (apparently) using, as well as the mail server software he > > is (apparently) using: > > This is getting a little stupid now, don't you think(and if you don't > think so maybe you're one of those M1CR0S0FT Z34L0TS h0h0h0h0h0)? i hope you'll note that i just tacks onto sean donelan's thread, where he states '... suspect he is going to face ...', and i added an observation prefaced with '... he might face ...'. pick on sean, not me. also: > #1) He works for microsoft. I can personally attest to that. that's fine. i've been lurking on this list for a long time and don't remember seeing much traffic from this person. that someone else will vouch for him is the correct response. > #2) He seems to want to sincerely state that: > a) microsoft acknowledges the problem > b) microsoft is working on the problem > c) microsoft isn't globally affected by this > ... even on a possibly unofficial basis. Information is good. i only partially agree. it is a fact that many things from big companies (microsoft being only yet another big company) are guesses, projections, wet dreams, and/or FUD. especially since his response contained almost no information other than a)/b)/c), above, all not particularly useful to {na}nog-ers, IMHO. and topped off with "i wish i could disclose more but i can't". perhaps better not to say anything instead of seeding suspicions or raising expectations. > #3) Just because a person may work for one Evil Empire or another > doesn't mean that they are forced to use their operating system > or MUA. I work for a company that has a mail product that a > considerable amount of people use, and I don't use it. This is > an acceptable practice at a lot of companies. it's probably not a secret that i'm not microsoft's biggest ally. nevertheless, i find it curious that you appear to be implying that you think i said he works for an evil empire. as far as mail software is concerned, your statement is factually correct, but how often does it apply to companies who make/manufacture/distribute an e-mail product (especially *their own*), as it is in this case? > #4) He is as furthest from a linux zealot as you can possibly be. obviously, i was insensitive by not adding a smiley to the following characters: LIENUX ZeAlOtS i humbly apologize. you would be advised to note that i am, as far as i can remember, one of the only 2 (or 3) people who've defended microsoft in the role of aggrieved network operator (on nanog), at least until we all find out what really happened. if we get a complete, accurate, and verifiable (inasmuch as that's possible) report from microsoft backed up with additional personal insight from dave(a)sneakerz.org, then we'll all be richer for the experience. let's see if that happens... -- Henry Yen Aegis Information Systems, Inc. Senior Systems Programmer Hicksville, New York

1 0

Re: Microsoft spokesperson blames ICANN
by Jim Duncan 25 Jan '01

25 Jan '01

Sean Donelan writes: > Microsoft appears to be blaming ICANN for the failure with Microft's > domain name servers (all located at the same place at Microsoft). > > Microsoft has yet to pin down the cause of the DNS error. "It can > be a system or human error, but somebody could also have done this > intentionally," De Jonge said. "We don't manage the DNS ourselves, > it is a system controlled by the Internet Corporation for Assigned > Names and Numbers (ICANN) with worldwide replicas." > > http://www.idg.net/ic_386962_1793_1-1681.html I have read that article many, many time today, trying to see how you came to that conclusion and I don't get it. To reach that conclusion, you've clearly quoted them out of the context of the larger article. Even to reach that conclusion from the small part you quoted requires a logical leap that is inappropriate, if not outright incorrect. I personally worked on that case for nearly eight hours today, right up until the resolution. I can tell you it was a learning experience for everyone involved. There were clearly some mistakes made, but it is also the case that there were a _lot_ of different things going on that contributed to the problem or complicated its resolution. However, jumping to such a conclusion is out of line, especially with the clout and respect you have on this mailing list. It has inflamed the responses of dozens of list members who want nothing more than to smack back at Microsoft, regardless of the truth in their statements. Microsoft did *not* blame ICANN in that article. I wouldn't put it past a spokesperson, Microsoft or otherwise, to make that mistake, but that clearly wasn't the case here. Let's not complicate the issue by making the same mistakes in our own comments. Thanks. Jim -- Jim Duncan, Product Security Incident Manager, Cisco Systems, Inc. <http://www.cisco.com/warp/public/707/sec_incident_response.shtml> E-mail: <jnduncan(a)cisco.com> Phone(Direct/FAX): +1 919 392 6209

2 1

RE: Great job AT&T (re: Microsoft problems)
by Roeland Meyer 25 Jan '01

25 Jan '01

The problem is that a PDC requires SRV updates on the zone. Only BIND-8.2.2p7 can do this and it's not clear that it runs sufficiently well (consider that feature as beta verging on alpha, IMHO). Certainly, the configuration is as clear as mud... way too much magic. My info says that MSFT only tested sufficiently with Win2K DNS. Integration with BIND was not considered especially important. This, in spite of the trip Vix made to NSFT, Summer '99 (where the SRV fixes came from). It is supposed to work, but the config is a nightmare of exceptions and special cases. > From: Steven J. Sobol [mailto:sjsobol@NorthShoreTechnologies.net] > Sent: Wednesday, January 24, 2001 4:24 PM > > On Wed, 24 Jan 2001, Dave McKay wrote: > > > > I'll just point out a few lines I wrote. I have tried to > be polite about > > all of this. > > Microsoft's ITG is investigating this issue. I haven't > been clued in as > > of yet as to what is the main issue. > > I apolozise for not being able to disclose more information. > > Ok, I wasn't going to say this, but Microsoft has been absolutely > ridiculous about blaming other people, so I'm going to say > it; it's not > directed personally at you, it's directed at the morons who > wrote your DNS > software, and the even bigger morons who insist that > It's-Not-A-Microsoft-Problem-Because-Microsoft-Is-Perfect: > > topaz.nstc.com used to run Microsoft DNS. After a while, having tried > repeatedly to get MSDNS to actually *serve* the zone files I > had set up, > and having not been successful, I switched to Bind 8 and have had no > problems since. > > You might want to try Bind 8.

2 1

RE: Microsoft spokesperson blames ICANN
by Irwin Lazar 25 Jan '01

25 Jan '01

> > On Wed, Jan 24, 2001 at 08:30:12PM -0500, Henry Yen wrote: > > > try: http://secondary.easydns.com > > > > Super-redundant DNS hosting would make a good product. Akamai maybe? > It's already available. See: www.ultradns.com irwin

1 0

Re: Microsoft spokesperson blames ICANN
by Henry Yen 25 Jan '01

25 Jan '01

On Wed, Jan 24, 2001 at 05:34:26AM -0500, Christian Kuhtz wrote: > > SICK, verrrrry sick ;-] > > On Wed, Jan 24, 2001 at 04:24:52PM -0500, Henry Yen wrote: [ snip ] > > agreed that the benefits of Microsoft DNS (tm), powered by oops. nowadays, it wouldn't be called "Microsoft DNS (tm)" (although it sounds really nice, just kinda roollllllls off the tongue :); instead if would be called "Microsoft Active DNS (tm)" (if you are american, think about what M A DNS sounds like when you run-it-altogether :). -- Henry Yen Aegis Information Systems, Inc. Senior Systems Programmer Hicksville, New York

1 0