Normally I reference this as an example of terrible government bureaucracy, but in this case it's also how said bureaucracy can delay operational changes. I was a contractor for one of the many branches of the DoD in charge of the network at a moderate-sized site. I'd been there about 4 months, and it was my first job with FedGov. I was sent a pair of Cisco 6509-E routers, with all supervisors and blades needed, along with a small mountain of SFPs, to replace the non-E 6509s we had installed that were still using GBICs for their downlinks. These were the distro switches for approximately half the site. Problem was, we needed 84 new SC-LC fiber jumpers to replace the SC-SC we had in place for the existing switch - GBICs to SFPs remember. We hadn't received any with the shipment. So I reached out to the project manager to ask about getting the fiber jumpers. "Oh, that should be coming from the server farm folks, since it's being installed in a server farm." Okay, that seems stupid to me, but $FedGov, who knows. I tell him we're stalled out until we get those cables - we have the routers configured and ready to go, just need the jumpers, can he get them from the server farm folks? He'll do that. It took FIFTEEN MONTHS to hash out who was going to pay for and order the fiber jumpers. Any number of times as the months dragged on, I seriously considered ordering them on Amazon Prime using my corporate card. We had them installed a week and a half after we got them. Why that long? Because we had to completely reconfigure them, and after 15 months, the urgency just wasn't there. By the way, the project ended up buying them, not the server farm team. On Tue, Feb 16, 2021 at 2:38 PM John Kristoff <jtk@dataplane.org> wrote:
Friends,
I'd like to start a thread about the most famous and widespread Internet operational issues, outages or implementation incompatibilities you have seen.
Which examples would make up your top three?
To get things started, I'd suggest the AS 7007 event is perhaps the most notorious and likely to top many lists including mine. So if that is one for you I'm asking for just two more.
I'm particularly interested in this as the first step in developing a future NANOG session. I'd be particularly interested in any issues that also identify key individuals that might still be around and interested in participating in a retrospective. I already have someone that is willing to talk about AS 7007, which shouldn't be hard to guess who.
Thanks in advance for your suggestions,
John