Not a famous operational issue, but in 2000, we had a major outage of our dialup modem pool. The owner of the building was re-skinning the outside using Styrofoam and stucco. A bunch of the Styrofoam had blocked the roof drains on the podium section of the building, immediately above our equipment room. A flash rainstorm filled the entire flat roof, and water came back in over the flashings, and poured directly in to our dialup modem pool through the hole in the concrete roof deck where the drain pipe protruded through. In retrospect, it was a monumentally stupid place to put our main modem pool, but we didn't realize what was above the drop ceiling - and that it was roof, not the other 11 floors of the building. 1 bay of 6 shelves of USR TC 1000 HiperDSPs were now very wet and blinking funny patterns on their LEDs. Fortunately, our vendor in Toronto (4 hour drive away) had stock of equipment that another customer kept delaying shipment on. They got their staff in, started un-boxing and, slotting cards. We spent a few hours tearing out the old gear and getting ready for replacements. We left Windsor, Ontario at around 12:00am - same time they left Toronto, heading towards us. We coordinated a meet at one of the rural exits along Highway 401 at a closed gas station at around 2am. Everything was going so well until a cop pulled up, and asked us what we were doing, as we were slinging modem chassis between the back of the vendor's SUV and our van... We calmly explained what happened. He looked between us a couple of times, shook his head and said "well, good luck with that", got back in his car and drove away. We had everything back online within 14 hours of the initial outage. At 02:37 PM 16/02/2021, John Kristoff wrote:
Friends,
I'd like to start a thread about the most famous and widespread Internet operational issues, outages or implementation incompatibilities you have seen.
Which examples would make up your top three?
To get things started, I'd suggest the AS 7007 event is perhaps the most notorious and likely to top many lists including mine. So if that is one for you I'm asking for just two more.
I'm particularly interested in this as the first step in developing a future NANOG session. I'd be particularly interested in any issues that also identify key individuals that might still be around and interested in participating in a retrospective. I already have someone that is willing to talk about AS 7007, which shouldn't be hard to guess who.
Thanks in advance for your suggestions,
John
-- Clayton Zekelman Managed Network Systems Inc. (MNSi) 3363 Tecumseh Rd. E Windsor, Ontario N8W 1H4 tel. 519-985-8410 fax. 519-985-8409