On Wed, 31 Aug 2005 15:47:43 +0200 Chris Gilbert <Chris@LainOS.org> wrote:
For those that don't know, directNIC as well as some other NOCs are located together in a high-rise in New Orleans.
Despite everyone's warnings (including my own!) some tough-as-nails guys from directNIC stayed behind to battle the hurricane and keep the networks online.
[snip] While the safety of our staff is paramount and most were safely evacuated, a small group of key personnel stayed behind to safeguard our data center and make sure that all of our services remained online and stable. During this time, while we spent a great deal of effort battling broken windows, incoming water, and flying debris, our hosting and registration services remained online and worked flawlessly. [/snip]
Link: http://www.directnic.com/katrina.php (Pictures too!)
It also mentions the power shortage, and that they are using diesel generated power.
I remember a week or two ago people were talking about building redundant datacenters, off-grid power, failure mitigation, etc.
I remember that after 9/11 the real network hits started about 3 days later, when the diesel generators started running out of fuel in the downtown telco hotels, and there was no way to physically get fuel trucks to their location. The equipment and generators were fine, they just ran out of fuel. If you look at the flooding in downtown New Orleans, it looks like this might happen again there. It makes me wonder whether part of disaster planning shouldn't be some sort of power triage, where if it looks like it's not going to be possible to get fuel to a datacenter after a systemwide power outage, instead of powering everything for a short time and then going dark, a subset is powered for weeks. Since I believe that air conditioning is a big part of the fuel expenditure, this might imply preplanning to the extent of grouping essential equipment together in a limited area that could be kept cool when everything else went dark.
I think if nothing else, this is at least a success story of building a NOC which can provide critical infrastructure that will survive major disasters.
-- Regards, Chris Gilbert
Regards Marshall Eubanks