Speaking on Deep Background, the Press Secretary whispered:

We have about 10-15 minutes of UPS capacity.

I.e. -- long enough to: a) get metal spinning or b) realize "Oh SHIT" and start "sync sync sync halt" on every box.

The important thing with a generator is to make sure you test it regularly. Ours (a diesel at one POP and a natural gas at another) automatically run each week and alert us to any problems. Then, to test the full system (transfer switch, etc.), we "pull the handle" (shut off utility power) once a month and make sure that everything works as it is supposed to.

WhatHeSaid... You must test the things UNDER LOAD to be sure all is well. ISTM the SOP is 30min to an hour, but that can be done less frequently than starting and warmup. Further gotcha's: Diesel fuel is a Petri dish. Weird bugs grow in it. [Call Tom Ridge!] If you don't have the right additives, your filters SHALL clog on same when you most need same. Cooling systems/maint -- please recall the NYC telehouse {was it 60 Hudson?} where post 9/11 the generator went down on overheat because the radiator was clogged... Most Diesels have a "day tank" with underground storage for the real supply, and a transfer pump. It may sound trite, but be sure that pump is itself an emergency load, otherwise..... [Any inference that I may have in fact run into this issue myself on Somebody's installation is unproven rumor.] -- A host is a host from coast to coast.................wb8foz@nrk.com & no one will talk to a host that's close........[v].(301) 56-LINUX Unless the host (that isn't close).........................pob 1433 is busy, hung or dead....................................20915-1433

Cooling systems/maint -- please recall the NYC telehouse {was it 60 Hudson?} where post 9/11 the generator went down on overheat because the radiator was clogged...

That was 25 Broadway (Telehouse USA). The air intake filter for their generator clogged from all the WTC dust in the air. Pretty rare circumstance...

On Thu, 3 Apr 2003, Sean Donelan wrote:

...

That was 25 Broadway (Telehouse USA). The air intake filter for their generator clogged from all the WTC dust in the air. Pretty rare circumstance...

There were some initial reports that this was the case, but I have never seen the final report from Telehouse what the final analysis found. If you remember there were also reports about a water pump failure, raditor clogging, etc.

I know the clogged air filter was what we heard for 32 Old Slip. Since the telcos in that building (ATT, Focal, GNaps) only operate their own battery strings and not generators, they relied on the building generator. They didn't have any pull in getting maintenance folks back into the area to work on the genset. When power was restored initially it was not enough to handle cooling, so stepping out onto the 5th floor landing was like entering a blast furnace. At Telehouse, this is what I dug out of my archives, presumably from John Shields. Some things to note are that the initial failure was due to the facility running out of fuel. Ooops. Once refueled, the water pump supposedly died. If I recall correctly there were two other gensets running for the 6th floor, which at the time was basically unoccupied. It would have been a better design perhaps to tie both floors together and get N+1 redundancy on the gensets. But Telehouse has a poor track record with whomever does their electrical work. Before we moved to 25Bway, we went through a period where a transfer switch would fail during tests at 33 Whitehall. If I recall correctly this happened at least 5 times in one year. ----- To our 25 Broadway Clients: We'd like to bring you up to date regarding the power situation at 25 Broadway. As you are aware, the commercial power supply to 25 Broadway was lost at about 16:35 EDT Tuesday, September 11th, following the terrorist attack on NYC. With the cooperation of the NY City Office of Emergency Management and many others, we were able to replenish the diesel fuel. At approximately 15:30 Thursday, however, our critical power generator's water pump failed and power was lost to most of our customers' equipment on the 5th floor. We have aggressively pursued a two-pronged approach to restoring power: 1) TELEHOUSE prevailed upon Con Edison, the regional commercial electric power supplier, to deliver a portable generator to the site. The generator will be set up in the street adjacent to the building and connected by their electricians through a transformer than Con Ed delivered earlier today and wired directly into the building's bus. We expect to have power restored to your equipment between 15:30-17:00 EDT. 2) Although we were originally advised that we would have a new water pump shipped to the site for installation this morning, Caterpillar's representative in this region (H.O. Penn) was unable to locate one in the metropolitan area. We have arranged to have a pump forwarded from the warehouse in Pennsylvania via expedited shipping. The pump is expected to arrive in NYC by approximately 20:00 tonight with installation being planned for tomorrow. This will make available an additional level of backup beyond the generator provided by Con Edison. We understand that you have endured hardships as a result of this problem. We sincerely appreciate your patience and understanding as we, along with the rest of New York, work to overcome the stresses that this catastrophe has forced upon all. Our Customer Assistance Center (CAC) is staffed 24/7. Please feel free to contact us at [Confidential - Removed] at any time you have questions. Our email address [Confidential - Removed] should also be working again soon after the restoration of power.