Michael Dillon writes:

...

And the stories that the power guy I'm working with tells about foreign facilities, particularly in middle east war zones, are really scary...

...

We fundamentally do not have the facilities problem completely nailed down to the point that things will never drop. Level 4 datacenters can, and will, fail. Nothing you can do including just doing 48V DC for everything are truly foolproof solutions.

A single level 4 datacenter is a Single Point of Failure!

Two of those middle-eastern style facilities is... ? Has anyone actually kept track of all these data center failures over the years and done some statistical analysis on it? Maybe two half-baked data centers is better than one over the long run?

Remember that one 10-12 years ago in (Palo Alto, Mountainview?) where a lady in a car caused a backhoe driver to move out of the way which resulted in him cutting a gas line which resulted in the fire department evacuating the data center, cutting off electricity in the area, and forbidding the diesel generators to be switched on?

Santa Clara. I was working right outside the evacuation radius. Which exchange point was in the building? PB-NAP? CIX? I remember we had a net-dark event associated, but not which one. It was a bad day... The lesson, as you point out, is that geographical redundancy is sometimes necessary. This is as true for providers as for datacenter end-users... -george william herbert gherbert@retro.com