On Thu, 9 Apr 2009, Jared Mauch wrote:
That AT&T has stopped provisioning protection fiber for automatic restoral is mind boggling.
Only helps with N-1 breaks. Unfortunately, sometimes there are N+1 breaks. Check the NANOG archives, I believe there were 5 breaks in one day in the 1990's; and even in the last year there have been 2-4 breaks on some transoceanic cables at the same time. On the other hand, I've never heard a carrier complain about digging more fiber as long as someone is willing to pay for it. How much more is someone willing to pay to get more diversity? Not willing to pay for it? I guess that's an answer too.
That our crack (or on crack) govt contracting/emergency-preparedness staff didn't demand protected facilities for 911 is another mind boggling issue.
This costs $$$ and usually isn't a problem as there are other ways to communicate. The law-enforcement folks qualify for GETS so get priority on wired/PSTN. They can also get radio priority w/ WPS.
If you don't know the acronyms, see www.ncs.gov. GETS and WPS are good as long as the system is still connected. TSP and SHARES helps when the system becomes disconnected. Some carriers also have mutual aid pacts, and work with members of the mutual aid pact with spare facilities. Its better to sign up ahead time, rather than waiting until after the problem happens. Even though those tools are useful, also work on how to maintain your own self-sufficiency until help arrives. There will always be some prioritization of repair efforts. Although it had a big impact on some of the largest carriers in the region, especially for local services; its always interesting to see other stuff kept working. Not everything broke.
If you're an ISP, you may be able to obtain GETS or WPS for your engineers. This would allow you a better chance of getting a channel to respond to issues. This is a good test to see how your backup plans might work for communication in the case of a larger distaster (earthquake, or other).
Make sure you test the tools you have. The people I know with GETS cards are encouraged to test them regularly and verify they work. If someone has one, I'd be interested to know if it proved to be of value today.
It sucked, but its also an opportunity for ISPs to figure out better ways to do things. personal opinions only