As prices fall, so does level of service. NANOGers all know providing uninterruptible power in the current evolving networks is hard as the communications infrastructure continues to decentralize. Providing non stop power for long term power failures with generators scattered all over the place is insanely hard.
Aside from human error, right now the weakest link in the net is the grid, and that is a link that isn't apparently getting any stronger.
It is interesting to note that one cause of the 6 hour Internet outage in Moscow was their belief that oin-site generators are not needed for telecom facilities. They felt that their AC power generation system was robust enough that providing 3 seperate power feeds into the M9 site contain MSK-IX was sufficient. And the power outage did show that the Russian power system architecture was indeed more robust than the American one. Their cascading failure was contained to a small area and power was restored faster than the August 2003 northeast failure. Perhaps there are lessons to be learned here by taking a closer look at what the Russian architecture is, what they expected to achieve with it, and what went wrong beyond the fact that old generators were not properly maintained or replaced. Is it possible to provide a municipal AC power grid for telecom facilities that can reliably power those facilities without recourse to on-site generators? After all, if there are working circuits into a site that means that there are undamaged paths which could be carrying AC current into the site. And if there are functioning gas lines into a site there are also undamaged paths which could be carrying AC current into a site. Note that I am assuming that at least one of the redundant power feeds into a site would be independent of the power grid, i.e. a municipal generator that supplies only telecom sites and data centers in a local area. --Michael Dillon