FINAL: MFS Outage: 6:30PM PDT 4/18/97 - ~1:30AM 4/19/97
This final incident report is being sent at 1:14AM 4/19/97: UPDATE: (7:30PM PDT) Starting at approximately 6:30PM PDT, MFS's power unit on their fiber mux located at the Digital Internet Exchange is resetting every 30 seconds to 1 minute. All MFS fiber facilities are down during the power reset. This includes all MFS/Worldcom DS3 and T1 circuits entering the PAIX collocate facility. The IX Operations staff (Digital Internet Exchange Operations) believes that the problem is due to a bad rectifier on the fiber mux's power rack. We are currently waiting for MFS to provide a dispatch to fix this problem. UPDATE: (11:00PM PDT) At approximately 7:30PM, MFS arrived on site. We offered them clean power from our in-house UPS and rectifiers at this time, which would have prevented the rapid, regular resetting of their line cards. They were not interested in the power at this time. By 8:30PM, they were convinced their rectifier was the cause of their power loss. They still are not interested in our power. By 10:15, they had restored power to their rectifier. Many of their line cards, including an OC48, did not return back to an operational state, likely due to the rapid resetting previously. Again we offered our power, however now they definitely were convinced it was unnecessary as their rectifiers were functioning. Instead, they are forcing us to wait until next week. As of 11:00, the line cards are still not functioning. UPDATE: 12:10AM 4/19/97 As of 11:45PM, the MFS technicians decided to go ahead and begin cut over to our power so that at least the power is stable while they work on the line cards. They are currently working on this cutover. UPDATE: 1:14AM 4/19/97 By 12:15AM they changed their minds on the power cutover again. The electricians installed two redundant rectifiers and two additional batteries to the system, which should provide power until the cutover to the Digital Internet Exchange power next week. Also by this point they have determined that the problem with the line cards is due to the fact that they lost their configuration during the power fluctuations. They begin reprogramming them at this time. At 12:35AM, MFS customers begin reporting their services returning to an operational state. At 1:00AM, the MFS representative completes all programming. He says that the circuits will return to service, one by one, using a "time lapse" system over the next 30 to 35 minutes. By 1:35AM, all services should be restored. At 1:25AM, MFS left the site saying all circuits that were "called in" by their customers were on-line. If by 2:35AM your services are not restored, please contact MFS and give them your circuit I.D. We are available for any remote hands requirements in the facility at 1-800-209-9898. Thank you, -Jay Adelson ----------------------------------------------------------- Jay Adelson | Email: adelson@pa.dec.com Operations Manager | Voice: (415) 617-3607 Digital Internet Exchange | Fax: (415) 617-3233 ----------------------------------------------------------- Secure. Reliable. Connected.
Well done. -- Rusty --------------------------------------------------------------------- | MCI Communications | | Internet MCI rusty@mci.net | ---------------------| http://infopage.mci.net |---------------------
This final incident report is being sent at 1:14AM 4/19/97:
UPDATE: (7:30PM PDT)
Starting at approximately 6:30PM PDT, MFS's power unit on their fiber mux located at the Digital Internet Exchange is resetting every 30 seconds to 1 minute.
All MFS fiber facilities are down during the power reset. This includes all MFS/Worldcom DS3 and T1 circuits entering the PAIX collocate facility.
The IX Operations staff (Digital Internet Exchange Operations) believes that the problem is due to a bad rectifier on the fiber mux's power rack.
We are currently waiting for MFS to provide a dispatch to fix this problem.
UPDATE: (11:00PM PDT)
At approximately 7:30PM, MFS arrived on site. We offered them clean power from our in-house UPS and rectifiers at this time, which would have prevented the rapid, regular resetting of their line cards. They were not interested in the power at this time.
By 8:30PM, they were convinced their rectifier was the cause of their power loss. They still are not interested in our power.
By 10:15, they had restored power to their rectifier. Many of their line cards, including an OC48, did not return back to an operational state, likely due to the rapid resetting previously. Again we offered our power, however now they definitely were convinced it was unnecessary as their rectifiers were functioning. Instead, they are forcing us to wait until next week.
As of 11:00, the line cards are still not functioning.
UPDATE: 12:10AM 4/19/97
As of 11:45PM, the MFS technicians decided to go ahead and begin cut over to our power so that at least the power is stable while they work on the line cards. They are currently working on this cutover.
UPDATE: 1:14AM 4/19/97
By 12:15AM they changed their minds on the power cutover again. The electricians installed two redundant rectifiers and two additional batteries to the system, which should provide power until the cutover to the Digital Internet Exchange power next week. Also by this point they have determined that the problem with the line cards is due to the fact that they lost their configuration during the power fluctuations. They begin reprogramming them at this time.
At 12:35AM, MFS customers begin reporting their services returning to an operational state.
At 1:00AM, the MFS representative completes all programming. He says that the circuits will return to service, one by one, using a "time lapse" system over the next 30 to 35 minutes. By 1:35AM, all services should be restored.
At 1:25AM, MFS left the site saying all circuits that were "called in" by their customers were on-line.
If by 2:35AM your services are not restored, please contact MFS and give them your circuit I.D. We are available for any remote hands requirements in the facility at 1-800-209-9898. Thank you,
-Jay Adelson
----------------------------------------------------------- Jay Adelson | Email: adelson@pa.dec.com Operations Manager | Voice: (415) 617-3607 Digital Internet Exchange | Fax: (415) 617-3233 ----------------------------------------------------------- Secure. Reliable. Connected.
participants (2)
-
Jay Adelson -
Rusty Zickefoose