On 10/6/21 06:51, Hank Nussbacher wrote:
- "During one of these routine maintenance jobs, a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network"
Can anyone guess as to what command FB issued that would cause them to withdraw all those prefixes?
Hard to say, as it seems that the command was innocent enough, perhaps running a batch of other sub-commands to check port status, bandwidth utilization, MPLS-TE values, e.t.c. However, sounds like another unforeseen bug in the command ran other things, or the cascade process of how the sub-commands were ran caused unforeseen problems. We shall guess this one forever, as I doubt Facebook will go into that much detail. What I can tell you is that all the major content providers spend a lot of time, money and effort in automating both capacity planning, as well as capacity auditing. It's a bit more complex for them, because their variables aren't just links and utilization, but also locations, fibre availability, fibre pricing, capacity lease pricing, the presence of carrier-neutral data centres, the presence of exchange points, current vendor equipment models and pricing, projection of future fibre and capacity pricing, e.t.c. It's a totally different world from normal ISP-land.
- "it was not possible to access our data centers through our normal means because their networks were down, and second, the total loss of DNS broke many of the internal tools we’d normally use to investigate and resolve outages like this. Our primary and out-of-band network access was down..."
Does this mean that FB acknowledges that the loss of DNS broke their OOB access?
I need to put my thinking cap on, but not sure whether running DNS in the IGP would have been better in this instance. We run our Anycast DNS network in our IGP, mainly to always guarantee latency-based routing, but also to ensure that the failure of a higher-level protocol like BGP does not disconnect internal access that is needed for troubleshooting and repair. Given the IGP is a much more lower-level routing protocol, it's more likely (not impossible) that it would not go down with BGP. In the past, we have, indeed, had BGP issues that allowed us to maintain DNS access internally as the IGP was unaffected. The final statement from that report is interesting: "From here on out, our job is to strengthen our testing, drills, and overall resilience to make sure events like this happen as rarely as possible." ... which, in my rudimentary translation, means that: "There are no guarantees that our automation software will not poop cows again, but we hope that when that does happen, we shall be able to send our guys out to site much more quickly." ... which, to be fair, is totally understandable. These automation tools, especially in large networks such as BigContent, are significantly more fragile the more complex they get, and the more batch tasks they need to perform on various parts of a network of this size and scope. It's a pity these automation tools are all homegrown, and can't be bought "pre-packaged and pre-approved to never fail" from IT Software Store down the road. But it's the only way for networks of this capacity to operate, and the risk they always sit with for being that large. Mark.