January 2001 - Test - lists.nanog.org

Estimated Time To Repair (was Re: History: lengthy outages)
by Sean Donelan 25 Jan '01

25 Jan '01

On Thu, 25 January 2001, Clayton Fiske wrote: > 1. I think it might be prudent to weed out the 2-5 hour outages here. > While that's still an excessively long time to recover a change that > should have been monitored and tested properly in the first place > (and still probably cause for firing in some shops), I can at least > conceive of it taking this amount of time. Too long, yes, but not > quite in the jaw-dropping category. I deliberately included them for a reason. Historically, when I look at a lot of network problems I notice an interesting coincidence. Outages involving "operator error" tended to take the longest time to fix, while outages involving equipment failure tend to be the shortest times. Complete hardware box failure (smoke makes debugging easy): 1 hour Power failure (utility, generator, etc): 3 hours External malicious attack (ddos, etc): 4 hours Fiber/Cable cut: 5 hours Electronic DCS failure: 18 hours Operator error: 1 business day (24-72 hours depending whether the operator made the change before leaving on a Friday night or a Tuesday night) Vendor software error: 2 business days (1 day to "escalate" the problem through customer channels, 1 day to actually get the fix, can take as long as 5 days if the problem happens after 3pm on Friday) Psychologists study why people have a difficult time recognizing their own mistakes. It is a very difficult problem. The problem is worse with "smart" people.

1 0

Re: traffic stats
by Ian A Finlay 25 Jan '01

25 Jan '01

On Thu, 25 Jan 2001, Troy Davis wrote: > http://west-boot.mfnx.net/traffic/sea/sea-nyc-oc48.html seems to show > that the traffic egressed Seattle to NYC - the outgoing jump lines up > with the incoming jump in both SJC -> SEA graphs. I wonder what caused that tremendous spike in traffic. If I am interpreting the graph correctly, it would appear that increases such as that don't occur very often. Looks like the traffic outbound increased more than 3 times for more than a few hours... Max Out: 1772.6 Mb/s (71.2%) Average Out: 282.5Mb/s (11.4%) -Ian

1 0

RE: Network diversity Software diversity
by Roeland Meyer 25 Jan '01

25 Jan '01

> From: woods(a)weird.com [mailto:woods@weird.com] > Sent: Wednesday, January 24, 2001 9:47 PM > > [ On , January 24, 2001 at 17:19:29 (-0800), Sean Donelan wrote: ] > > Subject: Network diversity Software diversity > > > > Using FreeBSD and BIND on *ALL* your name servers may be just as > > bad a practice as using Windows 2000 and Microsoft DNS on *ALL* > > your name servers. I still think NSI is taking a tremendous risk > > using identical servers for all their GTLD-servers, even though > > they are geographically distributed. > > Yeah, I was going to mention that, but I thought I'd already been > preaching too much to the converted! :-) Unless another name server, besides BIND8p7, can do SRV updates properly, I don't think it is possible to build that heterogenous software environment, when Win2K Active Directory is involved. In fact, even BIND8 has problems. It's only possible, with WinNT4, because WinNT4 doesn't have [very] many silly requirements and can live with a standard name server. > > You might try using UltraDNS on half your critical nameservers and > > BIND on the other half. And even using Solaris on some of the > > boxes and AIX or Linux, or NetBSD on the others. This is not because > > I think one or the other has a fatal flaw, but because software is > > a hard beast to manage. The idea behind diversity isn't you will > > never have an error. But the errors are unlikely to strike both > > servers at the same time. > > Therein lies the rub -- adding extra complexity to your systems also > makes them more difficult to manage, prone to error, and subject to > interoperational problems. > > Diversity of all forms definitely has its advantages, but it has its > costs too. The trick is to find a fair balance. :-) In this case, at this time, that is not possible under Win2K. It's the MSFT way or the highway ... However, if you think about it, this will definitely delay MANY Win2K Data Center migrations.

2 1

RE: MS explains
by Chris Davis 25 Jan '01

25 Jan '01

If Microsoft's problem was with routing to their DNS network, why did MX records resolve just fine? I'm afraid I don't understand that part. Is there policy routing or load balancing or something that can route MX requests to one set of dns servers while routing host requests to another set of DNS servers? As far as I can tell today, responses to my requests of the same DNS server for host records and MX records are returning from the same IP, 207.46.138.11, so it doesn't appear to me that different DNS servers are responding to different types of requests. -----Original Message----- From: Omachonu Ogali [mailto:missnglnk@informationwave.net] Sent: Thursday, January 25, 2001 8:25 AM To: Dan Hollis Cc: nanog(a)nanog.org Subject: Re: MS explains On Wed, Jan 24, 2001 at 08:40:19PM -0800, Dan Hollis wrote: > > On Wed, 24 Jan 2001, Eric A. Hall wrote: > > At 6:30 p.m. Tuesday (PST), a Microsoft technician made a configuration > > change to the routers on the edge of Microsoft's Domain Name Server > > network. > > At approximately 5 p.m. Wednesday (PST), Microsoft removed the changes to > > the router configuration and immediately saw a massive improvement in the > > DNS network. > > So basically, it took microsoft 23 hours to fix a router configuration. > > -Dan > > s/router configuration/default route/ -- Omachonu Ogali missnglnk(a)informationwave.net http://www.informationwave.net

2 2

MS explains
by Eric A. Hall 25 Jan '01

25 Jan '01

[anybody who shows up for a job interview with a resume saying "router ops for microsoft" should be viewed warily] http://www.microsoft.com/info/siteaccess.htm Microsoft Explains Site Access Issues On Tuesday evening and Wednesday, many Microsoft customers had difficulty accessing the company's Web sites. The cause has been determined, and the issue is resolved. At 6:30 p.m. Tuesday (PST), a Microsoft technician made a configuration change to the routers on the edge of Microsoft's Domain Name Server network. The DNS servers are used to connect domain names with numeric IP addresses (e.g. 207.46.230.219) of the various servers and networks that make up Microsoft's Web presence. The mistaken configuration change limited communication between DNS servers on the Internet and Microsoft's DNS servers. This limited communication caused many of Microsoft's sites to be unreachable (although they were actually still operational) to a large number of customers throughout last night and today. This was an operational error, and not the result of any issue with Microsoft or third-party products nor the security of our networks. Microsoft regrets any inconvenience caused to customers due to this issue. At approximately 5 p.m. Wednesday (PST), Microsoft removed the changes to the router configuration and immediately saw a massive improvement in the DNS network. All sites are currently available to customers. Again, Microsoft apologizes for the inconvenience. -- Eric A. Hall http://www.ehsco.com/ Internet Core Protocols http://www.oreilly.com/catalog/coreprot/

4 3

Monitoring highly redundant operations
by Sean Donelan 25 Jan '01

25 Jan '01

Not to pick on Dave, since I suspect he is going to have to face the Microsoft PR department for re-indoctrination for speaking out of turn, I'm glad to see someone from microsoft made an appearance. But he does raise an interesting problem. How do you know if your highly redudant, diverse, etc system has a problem. With an ordinary system its easy. It stops working. In a highly redudant system you can start losing critical components, but not be able to tell if your operation is in fact seriously compromised, because it continues to "work." As many of us have found out as we moved from simple networks to more complex networks, the network management is often much harder than the architecture of the network itself. Instead of relying on being notified when stuff "breaks" you have to actively monitor the state of your systems. Fairly frequently I see cases where the backup system failed, but no one knows about it until after the primary system also fails. >From the typical monitoring stations Dave sees, everything appears "normal." Yet, out in the real world there is a problem. Like most things its rarely a single thing that breaks, but chain of problems resulting in the final failure. So what should you be monitoring in addition to the typical graphs and logins to detect the problem seen by Microsot yesterday and today? On Wed, 24 January 2001, Dave McKay wrote: > Microsoft's ITG is investigating this issue. I haven't been clued in as > of yet as to what is the main issue. Hotmail's graphs and logins are > currently following the same trends as normal, they seem unaffected, > however this is not the case in all locations. DNS seems to be the > obvious choice for the blame. This is not the case in all areas, however. > At this point Microsoft is not willing to put the blame on anyone, or > any protocol for that matter. (Unless they already released a public > statement saying so, then who knows?) Anyway, the issues are being worked > on and service will be restored as soon as possible. I apolozise for not > being able to disclose more information. > > -- > Dave McKay > dave(a)sneakerz.org > Microsoft Global Network Architect

7 7

Re: Incease in latency to TLD servers
by Paul Vixie 25 Jan '01

25 Jan '01

http://f.root-servers.org/ looks normal to me, for whatever that's worth. additionally, http://f.root-servers.org/drew-pkts.html and http://f.root-servers.org/jason-pkts.html both show yesterday's microsoft.com problem clearly, but don't look whacky for today. (tuesday and friday show some reboots due to software upgrades.)

1 0

Re: Incease in latency to TLD servers
by Steven M. Bellovin 25 Jan '01

25 Jan '01

In message <0127E258EE29D3118A0F00609765B44847D0DC(a)dhcp-gateway.sitestream.net> , Karyn Ulriksen writes: > >Sean, > > The seven day map at http://average.miq.net/Weekly/markMM.html shows it >as even more dramatic. When I looked at the daily one, I wonder if perhaps >the demand of all those DNS servers re-querying for Microsoft was the >cause... but if that was the case, I assume (yuk!) we would have seen it >yesterday, too. The word I've heard is that there's a *large* amount of traffic at least one root server for assorted Microsoft domains. No idea what caused the other spikes... --Steve Bellovin, http://www.research.att.com/~smb

1 0

RE: Incease in latency to TLD servers
by Karyn Ulriksen 25 Jan '01

25 Jan '01

Sean, The seven day map at http://average.miq.net/Weekly/markMM.html shows it as even more dramatic. When I looked at the daily one, I wonder if perhaps the demand of all those DNS servers re-querying for Microsoft was the cause... but if that was the case, I assume (yuk!) we would have seen it yesterday, too. Have you ever seen spikes like this before? Any theories? K > -----Original Message----- > From: Sean Donelan [mailto:sean@donelan.com] > Sent: Thursday, January 25, 2001 1:19 PM > To: nanog(a)merit.edu > Subject: Incease in latency to TLD servers > > > > The Matrix (http://average.miq.net/Daily/markMM.html) is showing a > remarkable rise in latency to many DNS servers. Has someone found > a nifty new use for DNS servers? > > > >

2 1

RE: Incease in latency to TLD servers
by Scott Madley 25 Jan '01

25 Jan '01

As I try and normally stay away from the fluff that is Slashdot. I remember reading an article there some months ago about this. Further searching leads me to what you where talking about. Apparently it actually works. http://slashdot.org/articles/00/09/10/2230242.shtml -Scott -----Original Message----- From: M. David Leonard [mailto:mdl@equinox.shaysnet.com] Sent: Thursday, January 25, 2001 4:35 PM To: Sean Donelan Cc: nanog(a)merit.edu Subject: Re: Incease in latency to TLD servers Sean- Did I have a bad dream or wasn't someone trying to exploit some sort of tunneling via DNS connections to avoid firewalls blocking napster traffic? If I'm 'way off in left field just ignore me...... David Leonard ShaysNet On 25 Jan 2001, Sean Donelan wrote: > > The Matrix (http://average.miq.net/Daily/markMM.html) is showing a > remarkable rise in latency to many DNS servers. Has someone found > a nifty new use for DNS servers? > > > > >

1 0