On Thu, Jun 30, 2005 at 03:52:00PM -0700, Pablo's Gmail wrote:
Does anyone have related and/or specific outage information to the Equinox IBX facility in Chicago that occurred this past weekend? I'm more curious of actual root cause of their power facilities, outage timelines, response times...etc.
Equinox outage in Chicago? Did the power to the treadmill go off while you were working out? :) Equinix actually did a pretty good job of providing detailed information about what went wrong and why (after the fact), and kept customers up to date regarding the resolutions and testing activity afterwards. If you are a customer you should have gotten a copy of the report by now. I'm not going to paste the actual contents of it here (since it has one of those pesky industry-standard confidential information tags on it), but you can probably get someone at Equinix to give you a copy. There are really only two things I can fault Equinix for regarding the handling of the incident: #1 They didn't get an announcement of the situation out until 4 hours after the failure. #2 They should have had better monitoring systems in place to realize that they were putting out dirty power, and taken action to shut it off rather than putting out 90V on 120V AC circuits and bouncing on/off every minute until utility power was restored. Nearly everyone I've talked to lost some equipment during this, in some cases millions of dollars worth of gear (thank god for insurance :P). The recovery process from the damaged hardware lasted much longer than the actual outage, at least for us and our vendors. Other than that, they did the best they could with the situation, followed up with corrective action and useful/detailed information after the incident was over, and actually delivered on SLA credits much faster than other colos in similar situations. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)