On Sun, Nov 28, 2021 at 1:28 PM Dave Bell <me@geordish.org> wrote:
That facebook poorly managed their DNS to cause the recent disaster I don't want to wade into the middle of this argument, but has
On Sun, 28 Nov 2021 at 13:00, Masataka Ohta <mohta@necom830.hpcl.titech.ac.jp> wrote: there been more information about the recent facebook outage released that I missed? All I've read seems to say that the loss of connectivity to their DNS servers was a symptom, rather than the cause of the outage.
Hi Dave, You haven't missed anything. Facebook broke its backbone by mistake. Before they could restore it, the DNS records went stale and the now-stale servers withdrew themselves from the network as Facebook designed them to do when the records go stale. Unfortunately for Facebook, the internal servers did the same thing as the external ones. This broke the authentication system which in turn broke everything else, complicating their efforts to access the various systems including the ones they could have copied and pasted IP addresses from. But, to hear Masataka tell it, copy and paste hasn't been invented yet so we all type IP addresses by hand on our vt100 CRT terminals. Regards, Bill Herrin -- William Herrin bill@herrin.us https://bill.herrin.us/