This was not a building wide problem; apparently it did serve to find the failed UPSes in a few places. The building management does these tests several times a year, and at least to the one colo where we had visibility to the building electrical service, we saw the same outages we always see during such a test. It is also worth noting, that I only saw one of about 60 BGP sessions across the Seattle-IX (which is in the building) reset at the time of the work. John -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Rick Kunkel Sent: Friday, January 26, 2007 11:26 PM To: chuck goolsbee Cc: nanog@merit.edu Subject: Re: Westin Seattle Outage? We too had probs. I saw only two outages, one around 8PM PDT and one around 9:45PM PDT. I called during the first one, and the people I talked to were obviously in a state, and I had trouble hearing anyone, as they were in an extremely loud part of the data center or something. From what I could understand through the noise of some really loud fans or a generator, there was a power test of some kind, and a generator flaked or something. I've requested more detailed info, but have yet to receive it.

From what I understand, it affected more than just one provider.

--Rick Kunkel On Fri, 26 Jan 2007, chuck goolsbee wrote:

...

We just saw one of our gig-e circuits to the Westin bounce three times and another just go flatline in the past hour.

Answering my own question I know, but the OnFiber/Qwest guys I spoke to informed me that they heard the Westin had some sort of backup power scheduled maintenance go wrong. That was the 3 bouncer. I still wouldn't mind independent verification of that.

The flatliner was XO, and it seems that it may not even touch the Westin (instead goes to 1000 Denny.) I still don't know what happened there. It did come back after 86 minutes of eerie silence. Anyone else with XO circuits see anything odd tonight?

--chuck

Note to XO NOC: Your hold music is *awful* and on a way too short loop. It is bad enough when it is bad music, but to hear it over and over and over... for well over an hour, is customer torture.