On Mon, Jul 11, 2022 at 9:01 AM Andrey Kostin <ankost@podolsk.ru> wrote:
It's hard to believe that a same time maintenance affecting so many
devices in the core network could be approved. Core networks are build
with redundancy, so that failures can't completely destroy the whole
network.

I think you might need to re-evaluate your assumption 
about how core networks are built.

A well-designed core network will have layers of redundancy 
built in, with easy isolation of fault layers, yes.

I've seen (and sometimes worked on) too many networks 
that didn't have enough budget for redundancy, and were 
built as a string of pearls, one router to the next; if any router 
in the string of pearls broke, the entire string of pearls would 
come crashing down, to abuse a metaphor just a bit too much.

Really well-thought out redundancy takes a design team that 
has enough experience and enough focused hours in the day 
to think through different failure modes and lay out the design 
ahead of time, before purchases get made.    Many real-world 
networks share the same engineers between design, deployment, 
and operation of the network--and in that model, operation and 
deployment always win over design when it comes time to allocate 
engineering hours.  Likeise, if you didn't have the luxury of being 
able to lay out the design ahead of time, before purchasing hardware 
and leasing facilities, you're likely doing the best you can with locations 
that were contracted before you came into the picture, using hardware 
that was decided on before you had an opportunity to suggest better 
alternatives. 

Taking it a step further, and thinking about the large Facebook outage, 
even if you did well in the design phase, and chose two different vendors, 
with hardware redundancy and site redundancy in your entire core 
network, did you also think about redundancy and diversity for the 
O&M side of the house?   Does each redundant data plane have a 
diverse control plane and management plane, or would an errant 
redistribution of BGP into IGP wipe out both data planes, and both 
hardware vendors at the same time?  Likewise, if a bad configuration 
push isolates your core network nodes from the "God box" that 
controls the device configurations, do you have redundancy in 
connectivity to that "God box" so that you can restore known-good 
configurations to your core network sites, or are you stuck dispatching 
engineers with laptops and USB sticks with configs on them to get 
back to a working condition again?

As you follow the control of core networks back up the chain, 
you ultimately realize that no network is truly redundant and 
diverse.  Every network ultimately comes back to a single point 
of failure, and the only distinction you can make is how far up the 
ladder you climb before you discover that single point of failure.

Thanks!

Matt