I agree 100%. Now they need to figure out why bricking the management network stopped forwarding on the optical side. > (Forgive my top posting, not on my desktop as I’m out of town) Steven Naslund Chicago IL
Wild guess, based on my own experience as a NOC admin/head of operations at a large ISP - they have an automated deployment system for new firmware for a (mission critical) piece of backbone hardware.
They may have tested said firmware on a chassis with cards that did not exactly match the hardware they had in actual deployment (ie: card was older hw revision in deployed hardware), and while it worked fine there, it proceeded >shit the bed in the production.
Or, they missed a mandatory low level hardware firmware upgrade that has to be applied separately before the other main upgrade.
Kinda picturing in my mind that they staged all the updates, set a timer, staggered reboot, and after the first hit the fan, they couldn’t stop the rest as it fell apart as each upgraded unit fell on its own sword on reboot.
I’ve been bit by the ‘this card revision is not supported under this platform/release’ bug more often then I’d like to admit.
And, yes, my eyes did start to get glossy and hazy the more I read their explanation as well. It’s exactly the kind of useless post I’d write when I want to get (stupid) people off my back about a problem.