It's making the general press this hour so of course you already know about it but my question is this: who peers with meta and have you seen BGP sessions drop or the like? Do you operate meta CDN nodes in your network? Are they screaming for help? This doesn't sound like it's a network layer problem but I'm curious. Cheers, -- jra -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
On Tue, Mar 05, 2024 at 11:06:11AM -0500, Jay Ashworth <jra@baylink.com> wrote a message of 124 lines which said:
This doesn't sound like it's a network layer problem but I'm curious.
We can see the start page, authentification fails with an error message. It does not look like a network issue.
This time it looks like AAA issue on Facebook side. All peerings are up and cdns are reachable Dominik Dobrowolski W dniu wt., 5.03.2024 o 17:14 Stephane Bortzmeyer <bortzmeyer@nic.fr> napisał(a):
On Tue, Mar 05, 2024 at 11:06:11AM -0500, Jay Ashworth <jra@baylink.com> wrote a message of 124 lines which said:
This doesn't sound like it's a network layer problem but I'm curious.
We can see the start page, authentification fails with an error message. It does not look like a network issue.
My first thought, I'll admit, is that they turned a LLM loose on their configuration controls, and it "naturally" put itself in route 666. I sure hope from whatever subbasement clueful admins are still in over there, they can put it back together. Otherwise, productivity might improve as users flee to Office 365 to get some work done. I keep worrying about KEYTRAP.... -- https://www.youtube.com/watch?v=N0Tmvv5jJKs Epik Mellon: Making the Internet Better Dave Täht CSO, LibreQos
They reported a major failure (fat finger source) on one of their DB clusters. -J On Tue, Mar 5, 2024 at 10:07 AM Jay Ashworth <jra@baylink.com> wrote:
It's making the general press this hour so of course you already know about it but my question is this: who peers with meta and have you seen BGP sessions drop or the like? Do you operate meta CDN nodes in your network? Are they screaming for help?
This doesn't sound like it's a network layer problem but I'm curious.
Cheers, -- jra -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Does meta keep a board somewhere to tell the world it’s down? From: NANOG <nanog-bounces+bkain1=ford.com@nanog.org> On Behalf Of Jay Ashworth Sent: Tuesday, March 05, 2024 11:06 AM To: nanog@nanog.org Subject: Meta outage WARNING: This message originated outside of Ford Motor Company. Use caution when opening attachments, clicking links, or responding. It's making the general press this hour so of course you already know about it but my question is this: who peers with meta and have you seen BGP sessions drop or the like? Do you operate meta CDN nodes in your network? Are they screaming for help? This doesn't sound like it's a network layer problem but I'm curious. Cheers, -- jra -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Yes: metastatus.com It isn't happy. On March 5, 2024 11:23:42 AM EST, "Kain, Becki (.)" <bkain1@ford.com> wrote:
Does meta keep a board somewhere to tell the world it’s down?
From: NANOG <nanog-bounces+bkain1=ford.com@nanog.org> On Behalf Of Jay Ashworth Sent: Tuesday, March 05, 2024 11:06 AM To: nanog@nanog.org Subject: Meta outage
WARNING: This message originated outside of Ford Motor Company. Use caution when opening attachments, clicking links, or responding.
It's making the general press this hour so of course you already know about it but my question is this: who peers with meta and have you seen BGP sessions drop or the like? Do you operate meta CDN nodes in your network? Are they screaming for help?
This doesn't sound like it's a network layer problem but I'm curious.
Cheers, -- jra -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Thank you From: Jay Ashworth <jra@baylink.com> Sent: Tuesday, March 05, 2024 11:25 AM To: Kain, Becki (.) <bkain1@ford.com>; nanog@nanog.org Subject: RE: Meta outage Yes: metastatus.com It isn't happy. On March 5, 2024 11:23:42 AM EST, "Kain, Becki (.)" <bkain1@ford.com<mailto:bkain1@ford.com>> wrote: Does meta keep a board somewhere to tell the world it’s down? From: NANOG <nanog-bounces+bkain1=ford.com@nanog.org<mailto:nanog-bounces+bkain1=ford.com@nanog.org>> On Behalf Of Jay Ashworth Sent: Tuesday, March 05, 2024 11:06 AM To: nanog@nanog.org<mailto:nanog@nanog.org> Subject: Meta outage WARNING: This message originated outside of Ford Motor Company. Use caution when opening attachments, clicking links, or responding. It's making the general press this hour so of course you already know about it but my question is this: who peers with meta and have you seen BGP sessions drop or the like? Do you operate meta CDN nodes in your network? Are they screaming for help? This doesn't sound like it's a network layer problem but I'm curious. Cheers, -- jra -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
see: Status and outages of Meta business products (metastatus.com)<https://metastatus.com/> ________________________________ From: NANOG <nanog-bounces+chris=kimball.cc@nanog.org> on behalf of Kain, Becki (.) via NANOG <nanog@nanog.org> Sent: Tuesday, March 5, 2024 11:23 AM To: Jay Ashworth <jra@baylink.com>; nanog@nanog.org <nanog@nanog.org> Subject: RE: Meta outage Does meta keep a board somewhere to tell the world it’s down? From: NANOG <nanog-bounces+bkain1=ford.com@nanog.org> On Behalf Of Jay Ashworth Sent: Tuesday, March 05, 2024 11:06 AM To: nanog@nanog.org Subject: Meta outage WARNING: This message originated outside of Ford Motor Company. Use caution when opening attachments, clicking links, or responding. It's making the general press this hour so of course you already know about it but my question is this: who peers with meta and have you seen BGP sessions drop or the like? Do you operate meta CDN nodes in your network? Are they screaming for help? This doesn't sound like it's a network layer problem but I'm curious. Cheers, -- jra -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
On Tue, Mar 5, 2024 at 10:38 AM Chris K <Chris@kimball.cc> wrote:
see: Status and outages of Meta business products (metastatus.com)
Things seem to be returning now, aside from the status page, which still appears blank. Interesting. -- Hunter Fuller (they) Router Jockey VBH M-1C +1 256 824 5331 Office of Information Technology The University of Alabama in Huntsville Network Engineering
On Tue, Mar 05, 2024 at 04:23:42PM +0000, Kain, Becki (.) via NANOG <nanog@nanog.org> wrote https://metastatus.com/ a message of 210 lines which said:
Does meta keep a board somewhere to tell the world it’s down?
https://m.facebook.com/nt/screen/?params=%7B%22note_id%22%3A10158791436142200%7D&path=%2Fnotes%2Fnote%2F - Jorge (mobile)
On Mar 5, 2024, at 10:25, Kain, Becki (.) via NANOG <nanog@nanog.org> wrote:
Does meta keep a board somewhere to tell the world it’s down?
From: NANOG <nanog-bounces+bkain1=ford.com@nanog.org> On Behalf Of Jay Ashworth Sent: Tuesday, March 05, 2024 11:06 AM To: nanog@nanog.org Subject: Meta outage
WARNING: This message originated outside of Ford Motor Company. Use caution when opening attachments, clicking links, or responding.
It's making the general press this hour so of course you already know about it but my question is this: who peers with meta and have you seen BGP sessions drop or the like? Do you operate meta CDN nodes in your network? Are they screaming for help?
This doesn't sound like it's a network layer problem but I'm curious.
Cheers, -- jra -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
That's from 2010. You can see if you look at the non-mobile version of the page (requires login though): https://www.facebook.com/notes/10158791436142200/ [image: image.png] On Tue, Mar 5, 2024 at 1:32 PM Jorge Amodio <jmamodio@gmail.com> wrote:
- Jorge (mobile)
On Mar 5, 2024, at 10:25, Kain, Becki (.) via NANOG <nanog@nanog.org> wrote:
Does meta keep a board somewhere to tell the world it’s down?
*From:* NANOG <nanog-bounces+bkain1=ford.com@nanog.org> *On Behalf Of *Jay Ashworth *Sent:* Tuesday, March 05, 2024 11:06 AM *To:* nanog@nanog.org *Subject:* Meta outage
WARNING: This message originated outside of Ford Motor Company. Use caution when opening attachments, clicking links, or responding.
It's making the general press this hour so of course you already know about it but my question is this: who peers with meta and have you seen BGP sessions drop or the like? Do you operate meta CDN nodes in your network? Are they screaming for help?
This doesn't sound like it's a network layer problem but I'm curious.
Cheers, -- jra
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
On Tue, 05 Mar 2024 11:06:11 -0500, Jay Ashworth <jra@baylink.com> wrote:
It's making the general press this hour so of course you already know about it but my question is this: who peers with meta and have you seen BGP sessions drop or the like? Do you operate meta CDN nodes in your network? Are they screaming for help?
This doesn't sound like it's a network layer problem but I'm curious.
What I found intriguing was that I was logged out by Google Docs at the same moment FB logged me out. Downdetector showed a number of other supposedly unrelated services with large outage report spikes at roughly the same time. mdr -- "There are no laws here, only agreements." -- Masahiko
On Tue, 05 Mar 2024 12:17 -0700, Michael Rathbun <mdr@tesp.com> wrote:
What I found intriguing was that I was logged out by Google Docs at the same moment FB logged me out. Downdetector showed a number of other supposedly unrelated services with large outage report spikes at roughly the same time.
I was logged out of Honeywell's Total Connect Comfort site (remote thermostat control) at the same time FB logged me out. I'm not using OAUTH logins anywhere. I had recently changed SSID and the thermostat wasn't accepting remote commands. I thought maybe the SSID change had broken it and had just deleted the device from the TCC site to try adding it back when I was logged out. It's funny timing because earlier today I was telling an employee how we don't like to have our maintenance overlap with vendor's maintenance/repair window because when something breaks while we are making changes we can't always tell who is at fault.
On Tue, 05 Mar 2024 12:17 -0700, Michael Rathbun <mdr@tesp.com> wrote:
What I found intriguing was that I was logged out by Google Docs at the same moment FB logged me out. Downdetector showed a number of other supposedly unrelated services with large outage report spikes at roughly the same time.
I was logged out of Honeywell's Total Connect Comfort site (remote thermostat control) at the same time FB logged me out. I'm not using OAUTH logins anywhere.
I had recently changed SSID and the thermostat wasn't accepting remote commands. I thought maybe the SSID change had broken it and had just deleted the device from the TCC site to try adding it back when I was logged out.
It's funny timing because earlier today I was telling an employee how we don't like to have our maintenance overlap with vendor's maintenance/repair window because when something breaks while we are making changes we can't always tell who is at fault.
Interesting. I have a script that makes a call to TCC every so often to read temperature data, and it failed from 8:50AM PST to 10:30AM PST. Ray
participants (12)
-
Chris K -
Dave Taht -
Dominik Dobrowolski -
Hunter Fuller -
Jay Ashworth -
Jorge Amodio -
Kain, Becki (.) -
Michael Rathbun -
Mike Lewinski -
Ray Van Dolson -
Ross Tajvar -
Stephane Bortzmeyer