Building Evacuation [Re: BBN outage]
At 09:06 PM 12-10-96 -0700, maillists wrote:
In hindsight, solutions to tolerate a given failure are very easy to come by but shouldn't be talked down. I did not blame BBN--shit happens--
One thing constructive I would note that I haven't seen yet is that while one can imagine many awful scenarios where your lovely NOC is absolutely leveled (*) by a horrendous accident, it is much more likely that your NOC will be forced to evacuate for some "trivial" reason. Perhaps asbestos. Perhaps a gas main leak in the street. Perhaps the fire alarms just keep going off. Perhaps the power went off and the fire marshall won't let anyone stay in the building without elevator power. It might very well have been the case that had BBN had a backup generator they might still have been forced to evacuate the building, if the fire dept was paying attention. You have to remember that it's harder to keep people in a given building than it is to safeguard the equipment there. You always have to be ready to send your NOC operators packing with a cell phone and a laptop for up to 48 hours, in case the building has to be evacuated for one of a dozen reasons. You should have a backup call center with an ACD set up in some other city or a contract with someone for a backup service. --Kent (*) My nightmare scenario back when I ran the NOC at BBN at 10 Moulton Street was a gasoline tanker turning over in the street out front. The gasoline runs down the stairwell into the sunken parking lot under the building. Someone exits the building into the parking lot and lights a cigarette ... Next moment my NOC and 300 people are ashes and there's a 60 foot crater where the building used to be.
On Mon, 14 Oct 1996, Kent W. England wrote:
One thing constructive I would note that I haven't seen yet is that while one can imagine many awful scenarios where your lovely NOC is absolutely leveled (*) by a horrendous accident, it is much more likely that your NOC will be forced to evacuate for some "trivial" reason. Perhaps asbestos. Perhaps a gas main leak in the street. Perhaps the fire alarms just keep going off. Perhaps the power went off and the fire marshall won't let anyone stay in the building without elevator power. It might very well have been the case that had BBN had a backup generator they might still have been forced to evacuate the building, if the fire dept was paying attention.
Ah, ya, but when I leave my POP I don't shut off the gear, so what if BBN staff had to leave (if there were any at that POP), the gear would be up and that is what counts.
You have to remember that it's harder to keep people in a given building than it is to safeguard the equipment there. You always have to be ready to send your NOC operators packing with a cell phone and a laptop for up to 48 hours, in case the building has to be evacuated for one of a dozen reasons. You should have a backup call center with an ACD set up in some other city or a contract with someone for a backup service.
Right, this is why we build a 2 or more NOCs, and have them all up all the time. Nathan Stratton CEO, NetRail, Inc. Tracking the future today! --------------------------------------------------------------------------- Phone (703)524-4800 NetRail, Inc. Fax (703)534-5033 2007 N. 15 St. Suite 5 Email sales@netrail.net Arlington, Va. 22201 WWW http://www.netrail.net/ Access: (703) 524-4802 guest --------------------------------------------------------------------------- "Therefore do not worry about tomorrow, for tomorrow will worry about itself. Each day has enough trouble of its own." Matthew 6:34
In message <3.0b36.32.19961014140811.006de8d0@mail.cts.com>, "Kent W. England" writes:
At 09:06 PM 12-10-96 -0700, maillists wrote:
In hindsight, solutions to tolerate a given failure are very easy to come by but shouldn't be talked down. I did not blame BBN--shit happens--
One thing constructive I would note that I haven't seen yet is that while one can imagine many awful scenarios where your lovely NOC is absolutely leveled (*) by a horrendous accident, it is much more likely that your NOC will be forced to evacuate for some "trivial" reason. Perhaps asbestos. Perhaps a gas main leak in the street. Perhaps the fire alarms just keep going off. Perhaps the power went off and the fire marshall won't let anyone stay in the building without elevator power. It might very well have been the case that had BBN had a backup generator they might still have been forced to evacuate the building, if the fire dept was paying attention.
You have to remember that it's harder to keep people in a given building than it is to safeguard the equipment there. You always have to be ready to send your NOC operators packing with a cell phone and a laptop for up to 48 hours, in case the building has to be evacuated for one of a dozen reasons. You should have a backup call center with an ACD set up in some other city or a contract with someone for a backup service.
--Kent
Kent, We have a backup NOC 1,200 miles away. All the tools are set up and kept running in two places and at least once a year Ann Arbor intentionally hands off NOC duties to Elmsford. The people issue is the main problem, with Elmsford not being as familiar with day to day NOC operations. Both NOCs are connected to two POPs in different states with alegedly diverse circuits. Neither NOC has generators but the last time the UPS ran out we had NOC service cut over in time. This is also why we have DNS servers in different states (and kerberos slave servers and other critical services). I don't think a cell phone and a laptop would cut it. Curtis
participants (3)
-
Curtis Villamizar -
Kent W. England -
Nathan Stratton