RE: [announce-all] NAC Maintenance this weekend (fwd)
Something interesting for all you operational folks... got this from a customer. ---------- Forwarded message ---------- Date: Fri, 10 Dec 1999 05:11:42 -0500 From: Protect_the_innocent <nobody@nowhere> To: 'Alex Rubenstein' <alex@nac.net> Subject: RE: [announce-all] NAC Maintenance this weekend We tested our Y2k power contingency plans just over two weeks ago... we didn't expect any downtime either. We moved up the date on our power systems and cut main power. Three generators failed to operate. Two threw their breakers due to the high load on them. The last was overwhelmed when the previous two cut out and threw an estimated 300 amp surge through the lines, blowing up hundreds of light bulbs, frying fax machines and pencil sharpeners along the way, jumping circuits in our industrial level surge protector and traveling through our "surge protected" lines to every desktop in the company and into the server room. After taking out over 100 monitors and almost 40 PCs, the surge proceeded to destroy our Server Room air conditioner, 4 huge APC UPS systems, 4 Compaq Proliant 6500s, 9 Compaq Proliant 3000s, a Compaq Proliant 1800 and both AS/400s. The surge then jumped the lines into the main power grid, blowing up two transformers, one of which fell on the IT manager's car (poetic justice) and cutting power on the entire block. The generator then proceeded to burst into flames, eventually blowing up all six generators and burning up seven cars. This all occurred within about 10 seconds and sent 38 people to the hospital, got the entire IT staff fired (and me hired as the new IT manager,) cost the company over $650,000 in equipment (not to mention the impending lawsuits), destroyed 8 cars and has cost the company two weeks (and running) of downtime. The resulting publicity got us on television in 5 states and caused our main competitor to announce that they will be closed for the week following New Years... Let's just say I'll make other plans for internet access for tonight and the weekend - just in case. -----Original Message----- From: Alex Rubenstein [mailto:alex@nac.net] Sent: Thursday, December 09, 1999 10:43 PM To: announce-all@nac.net; operations@nac.net Cc: network@nac.net; internal@nac.net Subject: [announce-all] NAC Maintenance this weekend ** This electronic communication is being originated from Net Access ** Corporation, your Internet Service Provider. In an ongoing effort to make our systems ready for the Year 2000 date rollover, we will be performing some tests and upgrades this weekend. On December 10th, 1999, at 5:00pm EST, we will be testing our Year 2000 power contingency plan. No downtime is expected during this test. On December 11th, 1999, as 2:00am EST, we will upgrading william.nac.net, a virtual web server. Expected downtime is four hours. Customers who have websites on this virtual web server will be out of service during this time period. If you have any questions, please call the Technical Assistance Centre at 973-590-5100.
When was the last time they tested their backup power *without* changing the date?
participants (3)
-
Alex Rubenstein
-
Randy Bush
-
Scott W Brim