On Thu, 3 Apr 2003, Sean Donelan wrote:
That was 25 Broadway (Telehouse USA). The air intake filter for their generator clogged from all the WTC dust in the air. Pretty rare circumstance...
There were some initial reports that this was the case, but I have never seen the final report from Telehouse what the final analysis found. If you remember there were also reports about a water pump failure, raditor clogging, etc.
I know the clogged air filter was what we heard for 32 Old Slip. Since the telcos in that building (ATT, Focal, GNaps) only operate their own battery strings and not generators, they relied on the building generator. They didn't have any pull in getting maintenance folks back into the area to work on the genset. When power was restored initially it was not enough to handle cooling, so stepping out onto the 5th floor landing was like entering a blast furnace. At Telehouse, this is what I dug out of my archives, presumably from John Shields. Some things to note are that the initial failure was due to the facility running out of fuel. Ooops. Once refueled, the water pump supposedly died. If I recall correctly there were two other gensets running for the 6th floor, which at the time was basically unoccupied. It would have been a better design perhaps to tie both floors together and get N+1 redundancy on the gensets. But Telehouse has a poor track record with whomever does their electrical work. Before we moved to 25Bway, we went through a period where a transfer switch would fail during tests at 33 Whitehall. If I recall correctly this happened at least 5 times in one year. ----- To our 25 Broadway Clients: We'd like to bring you up to date regarding the power situation at 25 Broadway. As you are aware, the commercial power supply to 25 Broadway was lost at about 16:35 EDT Tuesday, September 11th, following the terrorist attack on NYC. With the cooperation of the NY City Office of Emergency Management and many others, we were able to replenish the diesel fuel. At approximately 15:30 Thursday, however, our critical power generator's water pump failed and power was lost to most of our customers' equipment on the 5th floor. We have aggressively pursued a two-pronged approach to restoring power: 1) TELEHOUSE prevailed upon Con Edison, the regional commercial electric power supplier, to deliver a portable generator to the site. The generator will be set up in the street adjacent to the building and connected by their electricians through a transformer than Con Ed delivered earlier today and wired directly into the building's bus. We expect to have power restored to your equipment between 15:30-17:00 EDT. 2) Although we were originally advised that we would have a new water pump shipped to the site for installation this morning, Caterpillar's representative in this region (H.O. Penn) was unable to locate one in the metropolitan area. We have arranged to have a pump forwarded from the warehouse in Pennsylvania via expedited shipping. The pump is expected to arrive in NYC by approximately 20:00 tonight with installation being planned for tomorrow. This will make available an additional level of backup beyond the generator provided by Con Edison. We understand that you have endured hardships as a result of this problem. We sincerely appreciate your patience and understanding as we, along with the rest of New York, work to overcome the stresses that this catastrophe has forced upon all. Our Customer Assistance Center (CAC) is staffed 24/7. Please feel free to contact us at [Confidential - Removed] at any time you have questions. Our email address [Confidential - Removed] should also be working again soon after the restoration of power.