I would hope that the data center engineers built and ran suite of tests to find failure points before the network infrastructure was put into production. That said, changes are made constantly to the infrastructure and it can become very difficult very quickly to know if the failovers are still going to work. This is one place where the power and network in a datacenter divulge. The power systems may take on additional load over the course of the life of the facility, but the transfer switches and generators do not get many changes made to them. Also, network infrastructure tests are not going to be zero impact if there is a config problem. Generator tests are much easier. You can start up the generator and do a load test. You can also load test the UPS systems as well. Then you can initiate your failover. Network tests are not going to be zero impact even if there isn't a problem. Let's say you wanted to power fail a edge router participating in BGP, it can take 30 seconds for that routers route to get withdrawn from the BGP tables of the world. The other problem is network failures always seem to come from "unexpected" issues. I always love it when I get an outage report from my ISP's or datacenter and they say an "unexpected issue" or "unforseen issue" caused the problem. Dylan -----Original Message----- From: Dan Snyder [mailto:sliplever@gmail.com] Sent: Monday, August 24, 2009 8:39 AM To: Ken Gilmour Cc: NANOG list Subject: Re: Data Center testing We have done power tests before and had no problem. I guess I am looking for someone who does testing of the network equipment outside of just power tests. We had an outage due to a configuration mistake that became apparent when a switch failed. It didn't cause a problem however when we did a power test for the whole data center. -Dan On Mon, Aug 24, 2009 at 9:31 AM, Ken Gilmour <ken.gilmour@gmail.com> wrote:
I know Peer1 in vancouver reguarly send out notifications of "non-impacting" generator load testing, like monthly. Also InterXion in Dublin, Ireland have occasionally sent me notification that there was a power outage of less than a minute however their backup successfully took the load.
I only remember one complete outage in Peer1 a few years ago... Never seen any outage in InterXion Dublin.
Also I don't ever remember any power failure at AiNet (Deepak will probably elaborate)
2009/8/24 Dan Snyder <sliplever@gmail.com>:
Does any one know of any data centers that do failure testing of their networking equipment regularly? I mean to verify that everything fails over properly after changes have been made over time. Is there any best practice guides for doing this?
Thanks, Dan