BBN (GTE) Suffers another major power problem.
The BBN Cambridge POP is "down and out" after a nearby manhole fire cut power to the building housing the POP. -Where is the generator backup?
The BBN Cambridge POP is "down and out" after a nearby manhole fire cut power to the building housing the POP.
-Where is the generator backup?
It's my understanding that BBN had emergency generator backup, but that it was insufficient to cover this outage. I'm pretty sure I saw that in one of their ticket emails last night, although it's not in the official ticket history. eric
I can't speak for BBN planet, but the power was out in Cambridge for a 2 hours. Anybody that can survive 2 hours running huge amounts of equipment must have their own power grid. :) Our little UPSes didn't survive... -Matthew At 07:39 AM 8/8/97 -0400, Eric Osborne wrote:
The BBN Cambridge POP is "down and out" after a nearby manhole fire cut power to the building housing the POP.
-Where is the generator backup?
It's my understanding that BBN had emergency generator backup, but that it was insufficient to cover this outage. I'm pretty sure I saw that in one of their ticket emails last night, although it's not in the official ticket history.
eric
Matthew White Network Administrator Channel 1 Communications 617.864.0100
On Fri, 8 Aug 1997, Matthew White wrote:
I can't speak for BBN planet, but the power was out in Cambridge for a 2 hours. Anybody that can survive 2 hours running huge amounts of equipment must have their own power grid. :)
Our little UPSes didn't survive...
-Matthew
Um, actually, it was out longer than that. The power was out from 1730 until roughly 2245, which was when the last of the Cambridge area outside of MIT got power back. MIT got power back at around ~2345 ......living in the People's Republic of Cambridge...... .....Matthew ----------------- M. F. Ringel Network Analysis BBN Planet Inc. ringel@bbnplanet.com
On Fri, 8 Aug 1997, Matthew F. Ringel wrote: |Um, actually, it was out longer than that. The power was out from 1730 |until roughly 2245, which was when the last of the Cambridge area outside of |MIT got power back. MIT got power back at around ~2345 So BBN couldn't last 5:15minutes?? What were you using for power backup? Some lousy AC UPS systems with 55minute runtime? Jonah
On Fri, 8 Aug 1997, Matthew White wrote:
I can't speak for BBN planet, but the power was out in Cambridge for a 2 hours. Anybody that can survive 2 hours running huge amounts of equipment must have their own power grid. :)
Our little UPSes didn't survive...
Well you don't want to design your power system so that your UPS will keep you through the outage. When you get to 250 KVA systems like we use for our POPs it is hard to get more then 30 min of uptime. All you want your UPS to do is keep you up until the generator gets you up to speed. Nathan Stratton President, CTO, NetRail,Inc. ------------------------------------------------------------------------ Phone (888)NetRail NetRail, Inc. Fax (404)522-1939 230 Peachtree Suite 500 WWW http://www.netrail.net/ Atlanta, GA 30303 ------------------------------------------------------------------------ "No king is saved by the size of his army; no warrior escapes by his great strength. - Psalm 33:16
At 07:39 AM 8/8/97 -0400, Eric Osborne wrote:
The BBN Cambridge POP is "down and out" after a nearby manhole fire cut power to the building housing the POP.
-Where is the generator backup?
It's my understanding that BBN had emergency generator backup, but that it was insufficient to cover this outage. I'm pretty sure I saw that in one of their ticket emails last night, although it's not in the official ticket history.
eric
Matthew White Network Administrator Channel 1 Communications 617.864.0100
On Fri, 8 Aug 1997, Nathan Stratton wrote: |On Fri, 8 Aug 1997, Matthew White wrote: | |> |> I can't speak for BBN planet, but the power was out in Cambridge for a 2 |> hours. |> Anybody that can survive 2 hours running huge amounts of equipment must |> have their own power grid. :) |> |> Our little UPSes didn't survive... | |Well you don't want to design your power system so that your UPS will keep |you through the outage. When you get to 250 KVA systems like we use for |our POPs it is hard to get more then 30 min of uptime. All you want your |UPS to do is keep you up until the generator gets you up to speed. Incorrect. Stop using AC equipment in your pops. Use DC equipment and get a _good_ DC Powerplant. Every carrier Class4/5 switchroom usually has 10-20,0000 AMP/hours of standby power. 1 DSC or Nortel switch sucks _quite_ a bit more power than even the largest of superpops. Every carrier has _at least_ 4hours of battery plant (most have 8-12). Relying on generators is a _bad_ idea. Its not hard to have 4-12hours of standby battery plant. Lucent/Lorain/Peco2 all make rather nice rectifiers, and C&D/Lucent/GDB all make some nice vented batteries. By going DC you also don't get hit with the inefficiencies of AC --> DC --> AC --> DC. You can bet that MCI/Sprint don't have a piece of AC equipment in their facilities and most likely are laughing their asses off right now. Jonah
On Fri, 8 Aug 1997, Jonah Yokubaitis wrote:
Incorrect. Stop using AC equipment in your pops. Use DC equipment and get a _good_ DC Powerplant. Every carrier Class4/5 switchroom usually has 10-20,0000 AMP/hours of standby power. 1 DSC or Nortel switch sucks _quite_ a bit more power than even the largest of superpops. Every carrier has _at least_ 4hours of battery plant (most have 8-12). Relying on generators is a _bad_ idea.
Ok, let me clarify. We have DC gear, most of our POPs also have vary large AC UPS systems because we also have 100 - 200 colo racks. Most (around 90% of our colo customers want AC, so we need a large AC as well as DC system.
Its not hard to have 4-12hours of standby battery plant. Lucent/Lorain/Peco2 all make rather nice rectifiers, and C&D/Lucent/GDB all make some nice vented batteries. By going DC you also don't get hit with the inefficiencies of AC --> DC --> AC --> DC.
You can bet that MCI/Sprint don't have a piece of AC equipment in their facilities and most likely are laughing their asses off right now.
True. Nathan Stratton President, CTO, NetRail,Inc. ------------------------------------------------------------------------ Phone (888)NetRail NetRail, Inc. Fax (404)522-1939 230 Peachtree Suite 500 WWW http://www.netrail.net/ Atlanta, GA 30303 ------------------------------------------------------------------------ "No king is saved by the size of his army; no warrior escapes by his great strength. - Psalm 33:16
Well, being a "telco", I can honestly say that DC is the way to go. Much more reliable power, less headache to buildout and easier to plan for expansion. Most "providers" want AC power in their bays because anyone can plug in an adapter.. so in essence anyone can rack and stack equipment. DC power actually takes a bit of knowledge about electricity.. so you don't fry your equipment or even better yet get 2nd degree burns on your hands when you don't ground yourself or your racks properly. Eric On Fri, 8 Aug 1997, Nathan Stratton wrote:
On Fri, 8 Aug 1997, Jonah Yokubaitis wrote:
Incorrect. Stop using AC equipment in your pops. Use DC equipment and get a _good_ DC Powerplant. Every carrier Class4/5 switchroom usually has 10-20,0000 AMP/hours of standby power. 1 DSC or Nortel switch sucks _quite_ a bit more power than even the largest of superpops. Every carrier has _at least_ 4hours of battery plant (most have 8-12). Relying on generators is a _bad_ idea.
Ok, let me clarify. We have DC gear, most of our POPs also have vary large AC UPS systems because we also have 100 - 200 colo racks. Most (around 90% of our colo customers want AC, so we need a large AC as well as DC system.
Its not hard to have 4-12hours of standby battery plant. Lucent/Lorain/Peco2 all make rather nice rectifiers, and C&D/Lucent/GDB all make some nice vented batteries. By going DC you also don't get hit with the inefficiencies of AC --> DC --> AC --> DC.
You can bet that MCI/Sprint don't have a piece of AC equipment in their facilities and most likely are laughing their asses off right now.
True.
Nathan Stratton President, CTO, NetRail,Inc. ------------------------------------------------------------------------ Phone (888)NetRail NetRail, Inc. Fax (404)522-1939 230 Peachtree Suite 500 WWW http://www.netrail.net/ Atlanta, GA 30303 ------------------------------------------------------------------------ "No king is saved by the size of his army; no warrior escapes by his great strength. - Psalm 33:16
_______________________________________________________ Eric D. Madison - Senior Network Engineer - ACSI - Advanced Data Services - ATM/IP Backbone Group 24 Hour NMC/NOC (800)291-7889 Email: noc@acsi.net
On Fri, 8 Aug 1997, Matthew White wrote:
Anybody that can survive 2 hours running huge amounts of equipment must have their own power grid. :)
Our little UPSes didn't survive...
Our solution to this was pretty simple: A gasoline generator and some heavy-duty cable :) No idea how that would scale (probably poorly), but at least it enabled us to get core services back online.
On Fri, 8 Aug 1997, Brian Tackett wrote:
Our solution to this was pretty simple: A gasoline generator and some heavy-duty cable :) No idea how that would scale (probably poorly), but at least it enabled us to get core services back online.
Try that with 480 volts 3 phase where you need 4 500 MCM cables for each phase. The nice thing is you can walk in the generator in winter to stay warm. Nathan Stratton President, CTO, NetRail,Inc. ------------------------------------------------------------------------ Phone (888)NetRail NetRail, Inc. Fax (404)522-1939 230 Peachtree Suite 500 WWW http://www.netrail.net/ Atlanta, GA 30303 ------------------------------------------------------------------------ "No king is saved by the size of his army; no warrior escapes by his great strength. - Psalm 33:16
On Fri, 8 Aug 1997, Matthew White wrote: | |I can't speak for BBN planet, but the power was out in Cambridge for a 2 |hours. |Anybody that can survive 2 hours running huge amounts of equipment must |have their own power grid. :) | |Our little UPSes didn't survive... Any major player in any telecom market that doesn't have at _least_ 4hours of battery plant is not worth its salt. -48VDC setups that are COMMON among carriers should be an absolute necessity for any large ISP/NSP as well. BBN not sizing their generator right was another huge mistake. You have to wonder who is asleep at the switch over there. Jonah
The BBN Cambridge POP is "down and out" after a nearby manhole fire cut power to the building housing the POP.
-Where is the generator backup?
It's my understanding that BBN had emergency generator backup, but that it was insufficient to cover this outage. I'm pretty sure I saw that in one of their ticket emails last night, although it's not in the official ticket history.
It's my understanding that a battery supply that lasts long enough to get the emergency generator running, along with a fuel contractor to keep the generator topped off, can last over rather long outages. -- Phil Howard KA9WGN +-------------------------------------------------------+ Linux Consultant | Linux installation, configuration, administration, | Milepost Services | monitoring, maintenance, and diagnostic services. | phil at milepost.com +-------------------------------------------------------+
On Fri, Aug 08, 1997 at 08:59:44AM -0500, Phil Howard wrote:
It's my understanding that BBN had emergency generator backup, but that it was insufficient to cover this outage. I'm pretty sure I saw that in one of their ticket emails last night, although it's not in the official ticket history.
It's my understanding that a battery supply that lasts long enough to get the emergency generator running, along with a fuel contractor to keep the generator topped off, can last over rather long outages.
The question is, what gets power? Assuming you're the size of BBN, it becomes.. er.. "physically impractical" to power everything with a generator. I know when I was at RPI, though they had a room full of batteries (3-5 minutes of power), and a huge generator, it was only enough to power the telephone system and mainframe. In fact, it couldn't handle the water cooler for the mainframe. But the point is, even though (I hope) BBN isn't using a power-hungry mainframe, enough routers, switches, NOC workstations, etc. will eventually add up to more kilowatts than you can supply yourself. A decision has to be made as to what stays on and what doesn't. -- = Christopher Masto = chris@netmonger.net = http://www.netmonger.net/ = = NetMonger Communications = finger for PGP key = $19.95/mo unlimited access = = Director of Operations = (516) 221-6664 = mailto:info@netmonger.net = v---(cut here)---v -- yourname@some.dumb.host.com "Keep in mind that anything Kibo says makes a great sig." -- Kibo ^---(cut here)---^
On Fri, 8 Aug 1997, Christopher Masto wrote:
The question is, what gets power? Assuming you're the size of BBN, it becomes.. er.. "physically impractical" to power everything with a generator. I know when I was at RPI, though they had a room full of batteries (3-5 minutes of power), and a huge generator, it was only enough to power the telephone system and mainframe. In fact, it couldn't handle the water cooler for the mainframe. But the point is, even though (I hope) BBN isn't using a power-hungry mainframe, enough routers, switches, NOC workstations, etc. will eventually add up to more kilowatts than you can supply yourself. A decision has to be made as to what stays on and what doesn't.
I do not agree at all, the question is what do you want to PAY to keep up. When we started we just provided generator feed just of our gear. After we lost power for +12 hours and our AIR units were off we learned that everything must be on generator. They make LARGE generators, we have a 750KVA in Atlanta, and are installing a 1000KVA in Chicago. It can be done, just not cheep. Nathan Stratton President, CTO, NetRail,Inc. ------------------------------------------------------------------------ Phone (888)NetRail NetRail, Inc. Fax (404)522-1939 230 Peachtree Suite 500 WWW http://www.netrail.net/ Atlanta, GA 30303 ------------------------------------------------------------------------ "No king is saved by the size of his army; no warrior escapes by his great strength. - Psalm 33:16
On Fri, Aug 08, 1997 at 02:15:35PM +0000, Nathan Stratton wrote:
I do not agree at all, the question is what do you want to PAY to keep up. When we started we just provided generator feed just of our gear. After we lost power for +12 hours and our AIR units were off we learned that everything must be on generator. They make LARGE generators, we have a 750KVA in Atlanta, and are installing a 1000KVA in Chicago. It can be done, just not cheep.
Before this gets too out-of-hand, let me just say "forget what I said before". It doesn't apply to BBN. Yes, they should have had adequate backup power. A clear case of posting before waking up on my part. -- = Christopher Masto = chris@netmonger.net = http://www.netmonger.net/ = = NetMonger Communications = finger for PGP key = $19.95/mo unlimited access = = Director of Operations = (516) 221-6664 = mailto:info@netmonger.net = v---(cut here)---v -- yourname@some.dumb.host.com "Keep in mind that anything Kibo says makes a great sig." -- Kibo ^---(cut here)---^
On Fri, 8 Aug 1997, Christopher Masto wrote:
The question is, what gets power? Assuming you're the size of BBN, it becomes.. er.. "physically impractical" to power everything with a generator. I know when I was at RPI, though they had a room full of batteries (3-5 minutes of power), and a huge generator, it was only enough to power the telephone system and mainframe. In fact, it couldn't handle the water cooler for the mainframe. But the point is, even though (I hope) BBN isn't using a power-hungry mainframe, enough routers, switches, NOC workstations, etc. will eventually add up to more kilowatts than you can supply yourself. A decision has to be made as to what stays on and what doesn't.
If you're in a 24x7, customer-driven market, you should have either: a) enough go-juice (via fuel contractor) to go until the grid is on-line OR b) smaller NOCs. There is no excuse for a large player like GTE to lose power for any extended period off time. (I can understand a brief, reboot long, outage if something went awry, but hours? No sir.) Rob Nelson President, INTERNOC (tm) the internetwork operating company, inc. +1.210.299.INOC / rnelson@internoc.com
participants (11)
-
Brian Tackett -
Christopher Masto -
Eric D. Madison -
Eric Osborne -
Jonah Yokubaitis -
Matthew F. Ringel -
Matthew White -
Nathan Stratton -
Patrick J. Chicas -
Phil Howard -
Robert T. Nelson