September 1996 - Test - lists.nanog.org

RE: power hit stories...
by Ry Jones (ECA) 09 Sep '96

09 Sep '96

At a company I used to work for, we had the badge reader/controller for the building in the security officer's office instead of the ops room. We had a failure (the Inauguration Day storm in the Pacific Northwest) that lasted longer than the UPS the security officer had put on his desktop. You guessed it... the building went into failsafe mode; only keys worked for access. None of the ops had keys and we were effectively locked out of the building. Somehow we got a hold of the security guy's home number and he came in and we put in an extension cord. Pretty stupid, but not as cool as your story. Related, but not to power, one Monday (holiday) the building went into Day Mode, which was quite a bit less secure than Night Mode. Someone had forgotten to enter the impending Holiday as a Night/Weekend mode day. The main doors, loading dock, and all internal doors were wide open all day until someone noticed none of them were locked. -- a-ryj(a)microsoft.com, rjones(a)wicker.com MSN WAN tools group http://www.wicker.com/ >---------- >From: Bill Sommerfeld[SMTP:sommerfeld@apollo.hp.com] >Sent: Sunday, September 08, 1996 5:21 PM >To: nanog(a)merit.edu >Subject: power hit stories... > >Reminds me of the power hit at MIT one summer Saturday in 1988. Ask >Jeff Schiller about it some time... > >Short version: > >MIT had just installed a 5ESS as its new PBX; they put in the usual >batteries for backup, and opted to not include a generator just for >the switch, because MIT had its own generator. > >All the multi-line phone sets were ISDN sets, powered from the phone >closets. The phone closets supposedly had emergency power (from the >generator) and (small) backup batteries. > >Anyhow, something bad happened somewhere in Cambridge (transformer >explosion or some such), with the result that most of town lost power. >For a long time. > >One might expect the generators to come on. > >Nope. They were off-line for maintainance. > >Phys plant had brought in portable generators on trailers. > >They didn't work either. > >oops. > >So, there were now three grades of power at MIT, and two of them >weren't working. > > - normal power (from the local power grid, now dead) > - emergency power (from the generator, now dead) > - switch power (from the switch's batteries, still working) > >Unfortunately, the main phys plant emergency number was on one of the >ISDN phones. As one might expect, that phone got a lot of calls after >the lights went out, and the phone went dead when its phone closet's >batteries went out.. > >Jeff was summoned to fix the switch so that the emergency number would >work again.. I happened to be nearby at the time and wound up getting >an impromptu tour of the switch. > >We noticed, on arrival, that the lights in the nearby room housing the >modem pool were evidently on switch power.. which was clearly a wiring >mistake. > >Jeff tried to get into the switch room with his key card, but some >part of the card reader/electronic lock setup which controlled access >to the door to the switch room was not on switch power, and it >wouldn't let us in. > >Our ears soon verified that the switch's burglar alarm, on the other >hand, *was* on switch power. > >Once we shut up the alarm, we discovered that switch console terminal >was *not* on switch power... > >Things went on like this for a while.. > >Moral: > >If there are multiple grades of power available, make *darn* sure that >nothing plugged into the highest grade depends on anything plugged >into a lower grade.. > > - Bill >

1 0

Re: The SWAMP
by Bradley Dunn 09 Sep '96

09 Sep '96

Interesting. Well, in that case the rest of the /24 they are in now would have no excuse not to renumber, now would they? :) How would the routes for these /32s get propagated? I assume you'd want them to be announced via BGP, so everyone would have to rewrite their filters to let those routes in. Right? -BD ---------- > From: Paul A Vixie <vixie(a)wisdom.home.vix.com> > To: nanog(a)merit.edu > Subject: Re: The SWAMP > Date: Sunday, September 08, 1996 7:12 PM > > We have considered assigning a /32 to each root name server and asking > everybody in the universe to carry these 10 or so routes in the interest > of complete and permanent provider independence. So rather than worry > about the /24's ... :-)

2 1

Re: MAE-East still no generator
by Eric Kozowski 09 Sep '96

09 Sep '96

>>>>>> "Avi" == Avi Freedman <freedman(a)netaxs.com> writes: > > Avi> Does anyone know of > Avi> inverters that take MFS's -48dc (with what I'm told is > Avi> 'positive-ground' power) and convert it to 110vac? > >Yes, such equipment is readily available. Even telcos have *some* AC >equipment! Check the yellow pages under Electronics; there's likely a >local company that carries that stuff surplus. Also, you can check >with your LXC to find out where they get equipment of that type. >This is from memory, but I believe Lorain makes such equipment. Yep, we use a Lorain inverter for our non -48VDC equipment. Works well. -- Eric Kozowski Senior Network Engineer eric(a)structured.net Structured Network Systems, Inc. (503)656-3235 FAX Better, Cheaper, Faster -- pick any two. (800)881-0962 Voice http://www.structured.net/ "Providing High Quality, Reliable Internet Service" -- 56k to DS1

1 0

power hit stories...
by Bill Sommerfeld 09 Sep '96

09 Sep '96

Reminds me of the power hit at MIT one summer Saturday in 1988. Ask Jeff Schiller about it some time... Short version: MIT had just installed a 5ESS as its new PBX; they put in the usual batteries for backup, and opted to not include a generator just for the switch, because MIT had its own generator. All the multi-line phone sets were ISDN sets, powered from the phone closets. The phone closets supposedly had emergency power (from the generator) and (small) backup batteries. Anyhow, something bad happened somewhere in Cambridge (transformer explosion or some such), with the result that most of town lost power. For a long time. One might expect the generators to come on. Nope. They were off-line for maintainance. Phys plant had brought in portable generators on trailers. They didn't work either. oops. So, there were now three grades of power at MIT, and two of them weren't working. - normal power (from the local power grid, now dead) - emergency power (from the generator, now dead) - switch power (from the switch's batteries, still working) Unfortunately, the main phys plant emergency number was on one of the ISDN phones. As one might expect, that phone got a lot of calls after the lights went out, and the phone went dead when its phone closet's batteries went out.. Jeff was summoned to fix the switch so that the emergency number would work again.. I happened to be nearby at the time and wound up getting an impromptu tour of the switch. We noticed, on arrival, that the lights in the nearby room housing the modem pool were evidently on switch power.. which was clearly a wiring mistake. Jeff tried to get into the switch room with his key card, but some part of the card reader/electronic lock setup which controlled access to the door to the switch room was not on switch power, and it wouldn't let us in. Our ears soon verified that the switch's burglar alarm, on the other hand, *was* on switch power. Once we shut up the alarm, we discovered that switch console terminal was *not* on switch power... Things went on like this for a while.. Moral: If there are multiple grades of power available, make *darn* sure that nothing plugged into the highest grade depends on anything plugged into a lower grade.. - Bill

1 0

Telehouse London
by mo＠uu.net 08 Sep '96

08 Sep '96

Recently went through a detailed review of the design of Telehouse Docklands. The UPS service is in fact based on house high-voltage DC. Inverters or converters are installed to meet customer requirements of voltage, frequency, etc. Batteries can carry the original full load for something like 18 hours, giving ample time to fix the generators (grin). The site was originally christened by some Japanese banks who wanted disaster recover for a building full of 3090s, which use 415 Hz power, btw. The mainframe biz didn't make it, so Telehouse started looking around for folks who wanted really killer facilities. They are *extremely* impressive. (better than some Mil-spec places!) The power system has 3 generators available at all times, only two of which are required to carry the original rated load (which was huge by current standards). There is space for a 4th if required. The building has 3 disjoint cooling loops with redundant pumps. The house power risers are huge floating buss-bars in 2 disjoint risers, each carries 2 independent rails ("A and B"). The building is Faraday-shielded because that far down the Thames, the ships go to full power on their radars and there are interesting EMI problems if you don't take care. The exterior glass, which is largely ornimental, is mylar-blast-coated. The recent Docklands bombing was right down the street and "nothing happened". The building AC is fed from 3 distinct locations on the power grid through disjoint entrances, and there are 4 entrances for fiber into the building, all going different directions. There is continuous video surveillance available for spaces, and there is a generator engineer on-site 24x365 (big problem if the generator doesn't start!). They have a run-tank rated for at least a week at full load. Self-contained drinking water and other needs for self-contained full support of the onsite staff. This is a very, very impresive building!!! Interestingly, if you build it like this ab initio, it is cheaper than doing much less after the fact by a LOT. -Mike O'Dell

2 1

Re: Optical bypass switches (Re: MAE-East still no generator)
by Dave Curado 08 Sep '96

08 Sep '96

> A thought I just had. How many folks have optical bypass switches on > their fddi connections? If a enough providers' equipment fail without > bypassing, the ring can get partitioned into different sections which > might explain some of this. Supposedly DRA's router is plugged into > the gigaswitch, but I don't have a gigaswitch manual to check what > happens with ring partitions. Perhaps we're getting off of the nanog topical area here, but... To the best of my knowledge, the fddi ring at mae-east is a concentrator type ring, a to m / b to m type of thing... instead of a fddi trunk type of ring with a to b and b to a. MFS may have a number of concentrators hanging off of the Gigaswitch. One of the nice things about a concentrated ring, if you will, is that individual ports can come up and down... the ring transitions, but when a node or nodes go down, the ring doesn't partition. One note of humor about optical bypasses... although I've never used them, I'm told they're less reliable than most fddi cards -- they end up breaking and causing more problems then if you lived without them. =-) davec

1 0

Re: MAE-East still no generator
by Justin W. Newton 08 Sep '96

08 Sep '96

At 10:55 AM 9/8/96 DST, Peter Lothberg wrote: > >So, can someone explain what the problem is? > >MAE-East is a telco facility (well...) but if I look at one of our routers; > >ICM-MAE-E uptime is 2 weeks, 4 days, 14 hours, 12 minutes >System restarted by reload at 14:40:57 EDT Tue Aug 20 1996 Actually my router there is running AC and is running with 11 weeks uptime (restarted by a relaod). I don't have any peering seesions that were up for more than three days though. Me not losing power doesn't help me if there's no way to talk to anyone else. Justin Newton Internet Architect Erol's Internet Services

1 0

Optical bypass switches (Re: MAE-East still no generator)
by Sean Donelan 08 Sep '96

08 Sep '96

>This is funny, I did loose my IBGP peers, maybe the MFS provided >IDSU is on AC power... -:) It wouldn't hurt to check. When doing a physical check of our facilities in the past, we've discovered MFS had installed some AC powered equipment on our rack. Other telco's have done the same thing. If you have an AC outlet too close to your rack, I've noticed even the telco techs don't like messing with DC power and plug things into the wall. A thought I just had. How many folks have optical bypass switches on their fddi connections? If a enough providers' equipment fail without bypassing, the ring can get partitioned into different sections which might explain some of this. Supposedly DRA's router is plugged into the gigaswitch, but I don't have a gigaswitch manual to check what happens with ring partitions. -- Sean Donelan, Data Research Associates, Inc, St. Louis, MO Affiliation given for identification not representation

1 0

RE: MAE-East still no generator
by Jim Browning 08 Sep '96

08 Sep '96

In some telco facilities that do have AC generator backup, there are a few seconds of down time while the generator kicks in. In these spaces, AC equipment will in all likelihood re-boot, while DC equipment stays up through the transition. -- Jim

4 3

Re: MAE-East still no generator
by Sean Donelan 08 Sep '96

08 Sep '96

>Just out of curiosity, does anyone have any personal experience with >losing utility-provided power more than, say, three times in a day? Plenty. When power goes out around here, its going to go up and down several more times. DRA is located between several organizations concerned with real money, Bridge Information handles real-time trading feeds, MasterCard International does credit cards and Western Union/AT&T does money orders. The standard operating procedure around here is three backup generators. Nothing can ruin your trading day faster than a power failure just at the moment you have your backup generator torn apart for maintenance. With 24 hour trading becoming the norm, all day is trading day. Two generators usually start immediately, with the third one held as a spare. DRA won't make the news. But if you hear MasterCard International is down, DRA will likely have suffered the same fate. P.S. a generator hint, don't forget to keep your fuel tanks warm. As several sites in the northeast found out a couple of winters ago, diesel fuel gels at very cold tempatures. -- Sean Donelan, Data Research Associates, Inc, St. Louis, MO Affiliation given for identification not representation

1 0