Anyone else lost power at Fisher Plaza this afternoon?
If anyone else lost power at Internap's Fisher Plaza facility in Seattle this afternoon and have any info related to exact duration or cause please hit me back off-list. Cheers, Jonathan
If anyone else lost power at Internap's Fisher Plaza facility in Seattle this afternoon and have any info related to exact duration or cause please hit me back off-list.
Isn't this like the 2nd or 3rd event at Fisher of late? Any answers Jonathan? Has Fisher owned up to the cause yet? I did not hear of any local grid issues (we're certainly NOT broiling up here like the rest of the country.) The *rumor* I heard was some sort of bypass switch failure. Genset(s) running, but no juice being delivered... then the UPS' ran dry. Outage lasted 45-90-some minutes depending on where in the facility your gear was. The above is all hearsay though, from some INAP customers I ran into yesterday. --chuck -- ----------------------------------------------------------------- Murphy's revenge: The more reliable you make a system, the longer it will take you to figure out what's wrong when it breaks. -- Sean Donelan on NANOG, Mon, 26 Nov 2001 06:28:22 -0500 (EST)
Hello: On 8/4/06 9:01 AM, "chuck goolsbee" <chucklist@forest.net> wrote:
If anyone else lost power at Internap's Fisher Plaza facility in Seattle this afternoon and have any info related to exact duration or cause please hit me back off-list.
Isn't this like the 2nd or 3rd event at Fisher of late?
It's the second event at Fisher Plaza - the last one happened near the end of May.
Any answers Jonathan? Has Fisher owned up to the cause yet? I did not hear of any local grid issues (we're certainly NOT broiling up here like the rest of the country.)
The *rumor* I heard was some sort of bypass switch failure. Genset(s) running, but no juice being delivered... then the UPS' ran dry. Outage lasted 45-90-some minutes depending on where in the facility your gear was.
It was a breaker in the main bypass from city power to the generators. The breaker failed to close so the generators happily fed power to nowhere. Then, everyone's UPS failed and down we/they went. The outage lasted approximately 26 minutes. The kicker is, this was the same component that failed in the first outage. I'm not sure if there's been an official announcement, per se, but I think Fisher Communications will have to report to the FCC because all of their radio and TV was down for much longer than the outage. Regards, Mike
Michael K. Smith wrote:
It was a breaker in the main bypass from city power to the generators. The breaker failed to close so the generators happily fed power to nowhere. Then, everyone's UPS failed and down we/they went. The outage lasted approximately 26 minutes.
Nobody checked to make sure that at least one of the UPSs showed a status of "ONLINE" instead of "ONBATTERY"? Were there no UPSs configured to alert during continued and extended PF? Surely people didn't just trust the sound/vibration of the running generator. -Jim P.
As far as I've heard there was not personnel manually interacting with the power system from the point that grid power was lost to when the UPS's finally gave out (it was a Sunday afternoon). I'm assuming that if someone was there engineers would have realized that the gen sets juice was going to /dev/null . I believe Fisher Plaza now has maintainence staff on-site 24x7 until all the parts are fixed. If anyone has a set of questions to ask colo providers about their power system best practices, or more simply background information about how facility power systems work for the 'smaller fish' here that have less exposure to running generators, etc... I'd greatly appreciate a reference. This event has reminded me that I can't be complacent and blindly hope that the people downstream from me (colo providers, etc...) are on their game 100%. Jim Popovitch wrote:
Michael K. Smith wrote:
It was a breaker in the main bypass from city power to the generators. The breaker failed to close so the generators happily fed power to nowhere. Then, everyone's UPS failed and down we/they went. The outage lasted approximately 26 minutes.
Nobody checked to make sure that at least one of the UPSs showed a status of "ONLINE" instead of "ONBATTERY"? Were there no UPSs configured to alert during continued and extended PF? Surely people didn't just trust the sound/vibration of the running generator.
-Jim P.
Hello Jim: On 8/4/06 9:30 AM, "Jim Popovitch" <jimpop@yahoo.com> wrote:
Michael K. Smith wrote:
It was a breaker in the main bypass from city power to the generators. The breaker failed to close so the generators happily fed power to nowhere. Then, everyone's UPS failed and down we/they went. The outage lasted approximately 26 minutes.
Nobody checked to make sure that at least one of the UPSs showed a status of "ONLINE" instead of "ONBATTERY"? Were there no UPSs configured to alert during continued and extended PF? Surely people didn't just trust the sound/vibration of the running generator.
-Jim P.
Indeed. The problem was there wasn't an engineer on site who could manually trip the breaker. They got onsite pretty quickly, but not quickly enough to trip the breaker in time to avoid an outage. So we watched the UPS drain all the way down which took about 24 minutes in our case. So close, yet so far. Regards, Mike
AFAIK, you don't need to have to have someone onsite to trip a breaker....if it doesn't do it automatically, there are a multitude of SCADA systems available to manuaully flip them on. Unless, of course, the electromechanical components that physically flip the breaker over have failed. Frank -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Michael K. Smith Sent: Friday, August 04, 2006 12:49 PM To: Jim Popovitch Cc: Nanog Subject: Re: Anyone else lost power at Fisher Plaza this afternoon? Hello Jim: On 8/4/06 9:30 AM, "Jim Popovitch" <jimpop@yahoo.com> wrote:
Michael K. Smith wrote:
It was a breaker in the main bypass from city power to the generators.
The
breaker failed to close so the generators happily fed power to nowhere. Then, everyone's UPS failed and down we/they went. The outage lasted approximately 26 minutes.
Nobody checked to make sure that at least one of the UPSs showed a status of "ONLINE" instead of "ONBATTERY"? Were there no UPSs configured to alert during continued and extended PF? Surely people didn't just trust the sound/vibration of the running generator.
-Jim P.
Indeed. The problem was there wasn't an engineer on site who could manually trip the breaker. They got onsite pretty quickly, but not quickly enough to trip the breaker in time to avoid an outage. So we watched the UPS drain all the way down which took about 24 minutes in our case. So close, yet so far. Regards, Mike
Hello Frank: On 8/5/06 1:20 PM, "Frank Bulk" <frnkblk@iname.com> wrote:
AFAIK, you don't need to have to have someone onsite to trip a breaker....if it doesn't do it automatically, there are a multitude of SCADA systems available to manuaully flip them on. Unless, of course, the electromechanical components that physically flip the breaker over have failed.
Frank
I would tend to agree; I think the issue, however, was the breaker had failed, so none of the systems in place were able to address it correctly. The manual failover is a list-ditch attempt should all others fail. Mike
participants (5)
-
chuck goolsbee -
Frank Bulk -
Jim Popovitch -
Jonathan Claybaugh -
Michael K. Smith