- NANOG - lists.nanog.org

Re: From Microsoft's site
by Richard A. Steenbergen 26 Jan '01

26 Jan '01

Fri, Jan 26, 2001 at 01:14:34AM -0500, Greg A. Woods wrote: > > I haven't looked at how the routing advertisements for that /24 appear > out in the rest of the world, beyond what's registered at whois.ra.net, > but I doubt they've made separate advertisments for each IP# or some > subnets that would separate them, and even if they did I doubt such > advertisments coul even make it past the route filters of their peers. ... > So, how is it that ARIN let them get away with two entries for the same > damn server?!?!?!?!? Why do we continue to be so shocked by this? Do we really expect to find that microsoft has created an elaborate tunnel configuration and is really running 16 totally diverse highly redundant servers which are simply disguised as coming from a contiguous set of ips and a single router in traceroutes? Perhaps the real reason for their failure was a well planned attack based on knowledge purchased from corporate espionage which targetted key portions of their infrastructure in a massive DDoS effort? Much like the discussion of the clue of their network engineers, perhaps we need to stop looking for elaborate explanations and accept a more straight-forward and logical motivation. Incompetence, pure and simple. Have we ever seen anything good networking-related come out of MS yet? Why do we expect things to change now? Besides, if everyone knew what they were doing, the people with a clue would get paid a lot less. Sometimes a cigar is just a cigar. Can this thread please die now? Please? -- Richard A Steenbergen <ras(a)e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)

1 0

From Microsoft's site
by Jeremy Randall 26 Jan '01

26 Jan '01

http://www.microsoft.com/info/siteaccess.htm "Microsoft Explains Site Access Issues On Tuesday evening and Wednesday, many Microsoft customers had difficulty accessing the company's Web sites. The cause has been determined, and the issue is resolved. At 6:30 p.m. Tuesday (PST), a Microsoft technician made a configuration change to the routers on the edge of Microsoft's Domain Name Server network. The DNS servers are used to connect domain names with numeric IP addresses (e.g. 207.46.230.219) of the various servers and networks that make up Microsoft's Web presence. The mistaken configuration change limited communication between DNS servers on the Internet and Microsoft's DNS servers. This limited communication caused many of Microsoft's sites to be unreachable (although they were actually still operational) to a large number of customers throughout last night and today. This was an operational error, and not the result of any issue with Microsoft or third-party products nor the security of our networks. Microsoft regrets any inconvenience caused to customers due to this issue. At approximately 5 p.m. Wednesday (PST), Microsoft removed the changes to the router configuration and immediately saw a massive improvement in the DNS network. All sites are currently available to customers. Again, Microsoft apologizes for the inconvenience."

5 4

traffic stats
by Ian A Finlay 26 Jan '01

26 Jan '01

Hello, So I've followed this list for about a year now and find it to be very interesting. It's great to be able to read what's on the minds of all the intelligent folks that frequent this list. The reason I am posting is because I have been following the Microsoft outage(s) over the last several days. I just noticed that several news sources are reporting that Microsoft has stated that the problems on Thursday have to do with being hit with a denial-of-service attack. I then ran across the following: http://west-boot.mfnx.net/traffic/sjc2/sjc2-sea-oc48.html Looks like a nice spike on Tuesday. Anyway, I thought that was kind of interesting. Hope this post wasn't a waste of text :-) -Ian Ian Finlay

2 1

Re: Monitoring highly redundant operations
by Howard C. Berkowitz 26 Jan '01

26 Jan '01

>On Wed, 24 Jan 2001, Simon Lockhart wrote: > >> >> Indeed. We currently monitor each part of our operation from a monitoring >> station on our network. Under certain conditions, this can give us both >> false positives and false negatives: >> >Umm... Keynote? >(http://www.keynote.com) > >I find it truly amazing that people don't already diversely >monitor. Hell, have cronned pings running off your friend's cable modem >if that's all you can afford, but for christ's sake, a single box colo'd >in someone else's cage, or a shell at shells.com or nether.net really >isn't that expensive. > > >Fighting the war against bad networks, > >Matthew Devney >Teamsphere Interactive Might be interesting to define a set of basic monitoring functions that independent ISPs can run on each other and share results. Early warnings could go to a special email. _________________________________ FAQ, list archives, and subscription info: http://www.groupstudy.com/list/cisco.html Report misconduct and Nondisclosure violations to abuse(a)groupstudy.com

1 0

Estimated Time To Repair (was Re: History: lengthy outages)
by Sean Donelan 26 Jan '01

26 Jan '01

On Thu, 25 January 2001, Clayton Fiske wrote: > 1. I think it might be prudent to weed out the 2-5 hour outages here. > While that's still an excessively long time to recover a change that > should have been monitored and tested properly in the first place > (and still probably cause for firing in some shops), I can at least > conceive of it taking this amount of time. Too long, yes, but not > quite in the jaw-dropping category. I deliberately included them for a reason. Historically, when I look at a lot of network problems I notice an interesting coincidence. Outages involving "operator error" tended to take the longest time to fix, while outages involving equipment failure tend to be the shortest times. Complete hardware box failure (smoke makes debugging easy): 1 hour Power failure (utility, generator, etc): 3 hours External malicious attack (ddos, etc): 4 hours Fiber/Cable cut: 5 hours Electronic DCS failure: 18 hours Operator error: 1 business day (24-72 hours depending whether the operator made the change before leaving on a Friday night or a Tuesday night) Vendor software error: 2 business days (1 day to "escalate" the problem through customer channels, 1 day to actually get the fix, can take as long as 5 days if the problem happens after 3pm on Friday) Psychologists study why people have a difficult time recognizing their own mistakes. It is a very difficult problem. The problem is worse with "smart" people.

1 0

Re: traffic stats
by Ian A Finlay 26 Jan '01

26 Jan '01

On Thu, 25 Jan 2001, Troy Davis wrote: > http://west-boot.mfnx.net/traffic/sea/sea-nyc-oc48.html seems to show > that the traffic egressed Seattle to NYC - the outgoing jump lines up > with the incoming jump in both SJC -> SEA graphs. I wonder what caused that tremendous spike in traffic. If I am interpreting the graph correctly, it would appear that increases such as that don't occur very often. Looks like the traffic outbound increased more than 3 times for more than a few hours... Max Out: 1772.6 Mb/s (71.2%) Average Out: 282.5Mb/s (11.4%) -Ian

1 0

RE: Network diversity Software diversity
by Roeland Meyer 26 Jan '01

26 Jan '01

> From: woods(a)weird.com [mailto:woods@weird.com] > Sent: Wednesday, January 24, 2001 9:47 PM > > [ On , January 24, 2001 at 17:19:29 (-0800), Sean Donelan wrote: ] > > Subject: Network diversity Software diversity > > > > Using FreeBSD and BIND on *ALL* your name servers may be just as > > bad a practice as using Windows 2000 and Microsoft DNS on *ALL* > > your name servers. I still think NSI is taking a tremendous risk > > using identical servers for all their GTLD-servers, even though > > they are geographically distributed. > > Yeah, I was going to mention that, but I thought I'd already been > preaching too much to the converted! :-) Unless another name server, besides BIND8p7, can do SRV updates properly, I don't think it is possible to build that heterogenous software environment, when Win2K Active Directory is involved. In fact, even BIND8 has problems. It's only possible, with WinNT4, because WinNT4 doesn't have [very] many silly requirements and can live with a standard name server. > > You might try using UltraDNS on half your critical nameservers and > > BIND on the other half. And even using Solaris on some of the > > boxes and AIX or Linux, or NetBSD on the others. This is not because > > I think one or the other has a fatal flaw, but because software is > > a hard beast to manage. The idea behind diversity isn't you will > > never have an error. But the errors are unlikely to strike both > > servers at the same time. > > Therein lies the rub -- adding extra complexity to your systems also > makes them more difficult to manage, prone to error, and subject to > interoperational problems. > > Diversity of all forms definitely has its advantages, but it has its > costs too. The trick is to find a fair balance. :-) In this case, at this time, that is not possible under Win2K. It's the MSFT way or the highway ... However, if you think about it, this will definitely delay MANY Win2K Data Center migrations.

2 1

RE: MS explains
by Chris Davis 26 Jan '01

26 Jan '01

If Microsoft's problem was with routing to their DNS network, why did MX records resolve just fine? I'm afraid I don't understand that part. Is there policy routing or load balancing or something that can route MX requests to one set of dns servers while routing host requests to another set of DNS servers? As far as I can tell today, responses to my requests of the same DNS server for host records and MX records are returning from the same IP, 207.46.138.11, so it doesn't appear to me that different DNS servers are responding to different types of requests. -----Original Message----- From: Omachonu Ogali [mailto:missnglnk@informationwave.net] Sent: Thursday, January 25, 2001 8:25 AM To: Dan Hollis Cc: nanog(a)nanog.org Subject: Re: MS explains On Wed, Jan 24, 2001 at 08:40:19PM -0800, Dan Hollis wrote: > > On Wed, 24 Jan 2001, Eric A. Hall wrote: > > At 6:30 p.m. Tuesday (PST), a Microsoft technician made a configuration > > change to the routers on the edge of Microsoft's Domain Name Server > > network. > > At approximately 5 p.m. Wednesday (PST), Microsoft removed the changes to > > the router configuration and immediately saw a massive improvement in the > > DNS network. > > So basically, it took microsoft 23 hours to fix a router configuration. > > -Dan > > s/router configuration/default route/ -- Omachonu Ogali missnglnk(a)informationwave.net http://www.informationwave.net

2 2

MS explains
by Eric A. Hall 26 Jan '01

26 Jan '01

[anybody who shows up for a job interview with a resume saying "router ops for microsoft" should be viewed warily] http://www.microsoft.com/info/siteaccess.htm Microsoft Explains Site Access Issues On Tuesday evening and Wednesday, many Microsoft customers had difficulty accessing the company's Web sites. The cause has been determined, and the issue is resolved. At 6:30 p.m. Tuesday (PST), a Microsoft technician made a configuration change to the routers on the edge of Microsoft's Domain Name Server network. The DNS servers are used to connect domain names with numeric IP addresses (e.g. 207.46.230.219) of the various servers and networks that make up Microsoft's Web presence. The mistaken configuration change limited communication between DNS servers on the Internet and Microsoft's DNS servers. This limited communication caused many of Microsoft's sites to be unreachable (although they were actually still operational) to a large number of customers throughout last night and today. This was an operational error, and not the result of any issue with Microsoft or third-party products nor the security of our networks. Microsoft regrets any inconvenience caused to customers due to this issue. At approximately 5 p.m. Wednesday (PST), Microsoft removed the changes to the router configuration and immediately saw a massive improvement in the DNS network. All sites are currently available to customers. Again, Microsoft apologizes for the inconvenience. -- Eric A. Hall http://www.ehsco.com/ Internet Core Protocols http://www.oreilly.com/catalog/coreprot/

4 3

Monitoring highly redundant operations
by Sean Donelan 26 Jan '01

26 Jan '01

Not to pick on Dave, since I suspect he is going to have to face the Microsoft PR department for re-indoctrination for speaking out of turn, I'm glad to see someone from microsoft made an appearance. But he does raise an interesting problem. How do you know if your highly redudant, diverse, etc system has a problem. With an ordinary system its easy. It stops working. In a highly redudant system you can start losing critical components, but not be able to tell if your operation is in fact seriously compromised, because it continues to "work." As many of us have found out as we moved from simple networks to more complex networks, the network management is often much harder than the architecture of the network itself. Instead of relying on being notified when stuff "breaks" you have to actively monitor the state of your systems. Fairly frequently I see cases where the backup system failed, but no one knows about it until after the primary system also fails. >From the typical monitoring stations Dave sees, everything appears "normal." Yet, out in the real world there is a problem. Like most things its rarely a single thing that breaks, but chain of problems resulting in the final failure. So what should you be monitoring in addition to the typical graphs and logins to detect the problem seen by Microsot yesterday and today? On Wed, 24 January 2001, Dave McKay wrote: > Microsoft's ITG is investigating this issue. I haven't been clued in as > of yet as to what is the main issue. Hotmail's graphs and logins are > currently following the same trends as normal, they seem unaffected, > however this is not the case in all locations. DNS seems to be the > obvious choice for the blame. This is not the case in all areas, however. > At this point Microsoft is not willing to put the blame on anyone, or > any protocol for that matter. (Unless they already released a public > statement saying so, then who knows?) Anyway, the issues are being worked > on and service will be restored as soon as possible. I apolozise for not > being able to disclose more information. > > -- > Dave McKay > dave(a)sneakerz.org > Microsoft Global Network Architect

7 7