Power to some of the affected sections of the building has been restored via existing onsite generators. The central power risers cannot be connected to current generators in a timely manner due to excessive damage to the electrical switching equipment (and those generators may still be in standing water). These provide power to a number of colocated systems. Temporary generators are on order to be connected to the central risers, and the site expects that to be complete sometime late this evening. As best I can tell, there is still no utility power connected to any of the systems. The AC systems (chiller and crac) are currently not working. It is not clear to me whether these will be brought back on line when the temporary generators are available, but I am assuming so. It was pleasant to see the general positive attitude, sharing of information and offers of assistance that were made by representatives of the various tenants, customers and carriers that were on the scene. The usual suspects (companies and individuals) stepped up and took care of things, as they always seem to. --D On Fri, Jul 3, 2009 at 1:39 PM, Leo Bicknell <bicknell@ufp.org> wrote:
In a message written on Fri, Jul 03, 2009 at 03:22:14PM -0400, Sean Donelan wrote:
Are you better off with a single "tier 4" data center, multiple "tier 1" data centers, or something in between?
It depends entirely on your dependency on connectivity.
One extreme is something like a Central Office. Lots of cables from end-sites terminate in the building. Having a duplicate of the head end termination equipment on the opposite coast is darn near useless. If the building goes down, the users going through it go down. "Tier 4" is probably a good idea.
The other extreme is a pure content play (YouTube, Google Search). Users don't care which data center they hit (within reason), and indeed often don't know. You're better off having data centers spread out all over, both so you're more likely to only loose one at a time, but also so that the loss of one is relatively unimportant. Once you're already in this architecture, Tier 1 is generally cheaper.
There are two problems though. First, most folks don't fit neatly in one of these buckets. They have some ties to local infrastructure, and some items which are not tied. Latency as a performance penality is very subjective. A backup 1000 miles away is fine for many things, and very bad for some things.
Second, most folks don't have choices. It would be nice if most cities had three each Tier 1, 2, 3 and 4 data centers available so there was choice and competition but that's rare.
Very few companies consider these choices rationally; often because choices are made by different groups. I am amazed how many times inside of an ISP the folks deploying the DNS and mail servers are firewalled from the folks deploying the network, to the point where you have to get to the President to reach common management. This leads to them making choices in opposite directions that end up costing extra money the company, and often resulting in a much lower uptimes than expected. Having the network group deploy a single point of failure to the "Tier 4" data center the server guys required is, well, silly.
However, more important than all of this is testing your infrastructure. Would you feel comfortable walking into your data center and ripping the power cable out of some bit of equipment at random _right now_? If not, you have no faith your equipment will work in an outage.
-- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/
-- -- Darren Bolding -- -- darren@bolding.org --