So what happened at Facebook today ? I saw one article quoting Roland saying it was a route leak, but I haven't seen any other sources that aren't just quoting Roland. Usually there are a few independent posts out there by now. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com
On 14 Mar 2019, at 19:17, Mike Hammett wrote:
I saw one article quoting Roland saying it was a route leak, but I haven't seen any other sources that aren't just quoting Roland.
That was the result of a miscommunication; a clarification has been issued, FYI. -------------------------------------------- Roland Dobbins <roland.dobbins@netscout.com>
Do you have a link to the clarification? With the high jitter of news, all I'm finding is people parroting the original statement. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Roland Dobbins" <Roland.Dobbins@netscout.com> To: nanog@nanog.org Sent: Thursday, March 14, 2019 7:23:00 AM Subject: Re: FB? On 14 Mar 2019, at 19:17, Mike Hammett wrote:
I saw one article quoting Roland saying it was a route leak, but I haven't seen any other sources that aren't just quoting Roland.
That was the result of a miscommunication; a clarification has been issued, FYI. -------------------------------------------- Roland Dobbins <roland.dobbins@netscout.com>
So what happened yesterday? From: NANOG <nanog-bounces@nanog.org> On Behalf Of Mike Hammett Sent: Thursday, March 14, 2019 8:29 AM To: Roland Dobbins <Roland.Dobbins@netscout.com> Cc: nanog@nanog.org Subject: Re: FB? Do you have a link to the clarification? With the high jitter of news, all I'm finding is people parroting the original statement. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ________________________________ From: "Roland Dobbins" <Roland.Dobbins@netscout.com<mailto:Roland.Dobbins@netscout.com>> To: nanog@nanog.org<mailto:nanog@nanog.org> Sent: Thursday, March 14, 2019 7:23:00 AM Subject: Re: FB? On 14 Mar 2019, at 19:17, Mike Hammett wrote:
I saw one article quoting Roland saying it was a route leak, but I haven't seen any other sources that aren't just quoting Roland.
That was the result of a miscommunication; a clarification has been issued, FYI. -------------------------------------------- Roland Dobbins <roland.dobbins@netscout.com<mailto:roland.dobbins@netscout.com>>
I found this article <https://www.datacenterdynamics.com/news/facebook-instagram-whatsapp-suffer-global-outage/> but no real answers. On Thu, Mar 14, 2019 at 9:36 AM Kain, Rebecca (.) <bkain1@ford.com> wrote:
So what happened yesterday?
*From:* NANOG <nanog-bounces@nanog.org> *On Behalf Of *Mike Hammett *Sent:* Thursday, March 14, 2019 8:29 AM *To:* Roland Dobbins <Roland.Dobbins@netscout.com> *Cc:* nanog@nanog.org *Subject:* Re: FB?
Do you have a link to the clarification? With the high jitter of news, all I'm finding is people parroting the original statement.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
------------------------------
*From: *"Roland Dobbins" <Roland.Dobbins@netscout.com> *To: *nanog@nanog.org *Sent: *Thursday, March 14, 2019 7:23:00 AM *Subject: *Re: FB?
On 14 Mar 2019, at 19:17, Mike Hammett wrote:
I saw one article quoting Roland saying it was a route leak, but I haven't seen any other sources that aren't just quoting Roland.
That was the result of a miscommunication; a clarification has been issued, FYI.
-------------------------------------------- Roland Dobbins <roland.dobbins@netscout.com>
As much as I wanted to crack jokes because I cannot stand Facebook (the product), much love to all you FB engineers that went through (and are probably still going through) much hell. On Thu, Mar 14, 2019 at 9:58 AM Jason Suter <jsuter@tps.org> wrote:
I found this article <https://www.datacenterdynamics.com/news/facebook-instagram-whatsapp-suffer-global-outage/> but no real answers.
On Thu, Mar 14, 2019 at 9:36 AM Kain, Rebecca (.) <bkain1@ford.com> wrote:
So what happened yesterday?
*From:* NANOG <nanog-bounces@nanog.org> *On Behalf Of *Mike Hammett *Sent:* Thursday, March 14, 2019 8:29 AM *To:* Roland Dobbins <Roland.Dobbins@netscout.com> *Cc:* nanog@nanog.org *Subject:* Re: FB?
Do you have a link to the clarification? With the high jitter of news, all I'm finding is people parroting the original statement.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
------------------------------
*From: *"Roland Dobbins" <Roland.Dobbins@netscout.com> *To: *nanog@nanog.org *Sent: *Thursday, March 14, 2019 7:23:00 AM *Subject: *Re: FB?
On 14 Mar 2019, at 19:17, Mike Hammett wrote:
I saw one article quoting Roland saying it was a route leak, but I haven't seen any other sources that aren't just quoting Roland.
That was the result of a miscommunication; a clarification has been issued, FYI.
-------------------------------------------- Roland Dobbins <roland.dobbins@netscout.com>
Facebook pushed an update to their code that manages cookies, that had a rather severe bug in it that resulted in a large flood of requests to their database servers. To deal with this load, they had to prevent all writes and then slowly allow people back on. I saw the writeup for it last night but cannot seem to find it now! Grrr. Did I dream it? On Thu, Mar 14, 2019 at 8:42 AM Bryan Holloway <bryan@shout.net> wrote:
On 3/14/19 9:06 AM, Tom Beecher wrote:
As much as I wanted to crack jokes because I cannot stand Facebook (the product), much love to all you FB engineers that went through (and are probably still going through) much hell.
+1 on both counts.
We've all been there; no bueno.
That's a 2010 outage that someone dug out and was doing the rounds as a new one --srs ________________________________ From: NANOG <nanog-bounces@nanog.org> on behalf of cosmo <clinton.mielke@gmail.com> Sent: Thursday, March 14, 2019 9:50 PM To: Bryan Holloway Cc: nanog@nanog.org Subject: Re: FB? Facebook pushed an update to their code that manages cookies, that had a rather severe bug in it that resulted in a large flood of requests to their database servers. To deal with this load, they had to prevent all writes and then slowly allow people back on. I saw the writeup for it last night but cannot seem to find it now! Grrr. Did I dream it? On Thu, Mar 14, 2019 at 8:42 AM Bryan Holloway <bryan@shout.net<mailto:bryan@shout.net>> wrote: On 3/14/19 9:06 AM, Tom Beecher wrote:
As much as I wanted to crack jokes because I cannot stand Facebook (the product), much love to all you FB engineers that went through (and are probably still going through) much hell.
+1 on both counts. We've all been there; no bueno.
Looks like Google recently posted their post-mortem of their outage on the 12th https://status.cloud.google.com/incident/storage/19002 On Thu, Mar 14, 2019 at 1:21 PM Suresh Ramasubramanian <ops.lists@gmail.com> wrote:
That's a 2010 outage that someone dug out and was doing the rounds as a new one
--srs
------------------------------ *From:* NANOG <nanog-bounces@nanog.org> on behalf of cosmo < clinton.mielke@gmail.com> *Sent:* Thursday, March 14, 2019 9:50 PM *To:* Bryan Holloway *Cc:* nanog@nanog.org *Subject:* Re: FB?
Facebook pushed an update to their code that manages cookies, that had a rather severe bug in it that resulted in a large flood of requests to their database servers. To deal with this load, they had to prevent all writes and then slowly allow people back on.
I saw the writeup for it last night but cannot seem to find it now! Grrr. Did I dream it?
On Thu, Mar 14, 2019 at 8:42 AM Bryan Holloway <bryan@shout.net> wrote:
On 3/14/19 9:06 AM, Tom Beecher wrote:
As much as I wanted to crack jokes because I cannot stand Facebook (the product), much love to all you FB engineers that went through (and are probably still going through) much hell.
+1 on both counts.
We've all been there; no bueno.
The route-leak was something different that seems to have mainly hit west-Europe between 16:52 UTC to 17:08 UTC. There’s a few people in the *NOG communities still digging at the complete details of that right now, but it currently points to have originated from AS200020, impacting a few large upstreams for a short period of time. So unless this leak caused a catastrophic cascade in FB’s network somehow, it seems to be unrelated. It looked like a valid suspect because timing was very similar between the start of the FB outage and the leak. Jeroen Wunnink Sr. Manager - Integration Engineering www.gtt.net<http://www.gtt.net/> [id:image001.png@01D37331.D1301F60] From: NANOG <nanog-bounces@nanog.org> on behalf of "Kain, Rebecca (.)" <bkain1@ford.com> Date: Thursday, 14 March 2019 at 14:36 To: Mike Hammett <nanog@ics-il.net>, Roland Dobbins <Roland.Dobbins@netscout.com> Cc: "nanog@nanog.org" <nanog@nanog.org> Subject: RE: FB? So what happened yesterday? From: NANOG <nanog-bounces@nanog.org> On Behalf Of Mike Hammett Sent: Thursday, March 14, 2019 8:29 AM To: Roland Dobbins <Roland.Dobbins@netscout.com> Cc: nanog@nanog.org Subject: Re: FB? Do you have a link to the clarification? With the high jitter of news, all I'm finding is people parroting the original statement. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com<https://url.emailprotection.link/?b6x9-Fnx1LAfWyqRtESstZgT_vbCd3MONOqCVFZ0R_BHO80Ox-i-8zIm9qQ1soVeoZrxi8iA3iwJ_b5GpLUD9bw~~> Midwest-IX http://www.midwest-ix.com<https://url.emailprotection.link/?bTe_C0izNVY8kqDiMqBi3hZOjTZ-zNYEmkKhYlGBamvFdtMk4Ad_MAPQonzhIUmKh8G8FAwXEtjMYejM3PlLz6A~~> ________________________________ From: "Roland Dobbins" <Roland.Dobbins@netscout.com<mailto:Roland.Dobbins@netscout.com>> To: nanog@nanog.org<mailto:nanog@nanog.org> Sent: Thursday, March 14, 2019 7:23:00 AM Subject: Re: FB? On 14 Mar 2019, at 19:17, Mike Hammett wrote:
I saw one article quoting Roland saying it was a route leak, but I haven't seen any other sources that aren't just quoting Roland.
That was the result of a miscommunication; a clarification has been issued, FYI. -------------------------------------------- Roland Dobbins <roland.dobbins@netscout.com<mailto:roland.dobbins@netscout.com>>
On 14 Mar 2019, at 15:04, Jeroen Wunnink <Jeroen.Wunnink@gtt.net> wrote:
The route-leak was something different that seems to have mainly hit west-Europe between 16:52 UTC to 17:08 UTC. There’s a few people in the *NOG communities still digging at the complete details of that right now, but it currently points to have originated from AS200020, impacting a few large upstreams for a short period of time.
So unless this leak caused a catastrophic cascade in FB’s network somehow, it seems to be unrelated. It looked like a valid suspect because timing was very similar between the start of the FB outage and the leak.
Hello, I’m an engineer at AS200020. We indeed had a route leak yesterday between 16:45 and 17:07 UTC. This was caused by a mistake in the configuration of one of ou core routers by one of our engineers. We leaked full table to one of our transits, which is apparently accepted. We’ve seen some prefixes from a couple of AS numbers coming in. Facebook was not one of these AS’es. As far as I can tell it’s not related at all to the major incidents at Facebook. We are very sorry for any inconvenience people could have experienced. If you received complaints in the mentioned time period and want to know if it could be related, please let me know. Best regards, Pim van Stam NBIP-naWas / AS200020
I did see this article indicating they had somehow invalidated their cache in a botched deployment of changes - https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-o... On Thu, 14 Mar 2019 at 06:18, Mike Hammett <nanog@ics-il.net> wrote:
So what happened at Facebook today? I saw one article quoting Roland saying it was a route leak, but I haven't seen any other sources that aren't just quoting Roland. Usually there are a few independent posts out there by now.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
That’s old. By Robert Johnson on Thursday, September 23, 2010 at 7:29 PM Luke Ns From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Selphie Keller Sent: Thursday, March 14, 2019 4:06 PM To: Mike Hammett Cc: NANOG list Subject: Re: FB? I did see this article indicating they had somehow invalidated their cache in a botched deployment of changes - https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-o... On Thu, 14 Mar 2019 at 06:18, Mike Hammett <nanog@ics-il.net<mailto:nanog@ics-il.net>> wrote: So what happened at Facebook today? I saw one article quoting Roland saying it was a route leak, but I haven't seen any other sources that aren't just quoting Roland. Usually there are a few independent posts out there by now. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com
Yeah I just saw that date and that is odd, I got the link yesterday from somewhere and didn't notice the date was old. They do mention the configuration change issue in this one though that is dated today 14th - https://www.cbsnews.com/news/facebook-instagram-down-wednesday-facebook-blam...
From my understanding yesterday is they invalidated their world cache and all the traffic hit their backend quickly overwhelming their servers. It was strange that they didn't really talk about the issue just some brief messages on their twitter saying it wasn't ddos.
On Thu, 14 Mar 2019 at 15:08, Luke Guillory <lguillory@reservetele.com> wrote:
That’s old.
By Robert Johnson on Thursday, September 23, 2010 at 7:29 PM
Luke
Ns
*From:* NANOG [mailto:nanog-bounces@nanog.org] *On Behalf Of *Selphie Keller *Sent:* Thursday, March 14, 2019 4:06 PM *To:* Mike Hammett *Cc:* NANOG list *Subject:* Re: FB?
I did see this article indicating they had somehow invalidated their cache in a botched deployment of changes - https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-o...
On Thu, 14 Mar 2019 at 06:18, Mike Hammett <nanog@ics-il.net> wrote:
So what happened at Facebook today? I saw one article quoting Roland saying it was a route leak, but I haven't seen any other sources that aren't just quoting Roland. Usually there are a few independent posts out there by now.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_facebook_status_1106229690069442560&d=DwIGaQ&c=n6-cguzQvX_tUIrZOS_4Og&r=r4NBNYp4yEcJxC11Po5I-w&m=IHR1veHNjVYVktL31OQ_tgBUNHO5Uf3ACrvIVAW5cho&s=zrKUWVShQdFllKTGbJE5kITG87q7KNJHo0bD6aETBBk&e= From: NANOG <nanog-bounces@nanog.org> On Behalf Of Luke Guillory Sent: Thursday, March 14, 2019 2:09 PM To: Selphie Keller <selphie.keller@gmail.com>; Mike Hammett <nanog@ics-il.net> Cc: NANOG list <nanog@nanog.org> Subject: RE: FB? That’s old. By Robert Johnson on Thursday, September 23, 2010 at 7:29 PM Luke Ns From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Selphie Keller Sent: Thursday, March 14, 2019 4:06 PM To: Mike Hammett Cc: NANOG list Subject: Re: FB? I did see this article indicating they had somehow invalidated their cache in a botched deployment of changes - https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_notes_facebook-2Dengineering_more-2Ddetails-2Don-2Dtodays-2Doutage_431441338919_&d=DwIGaQ&c=n6-cguzQvX_tUIrZOS_4Og&r=r4NBNYp4yEcJxC11Po5I-w&m=IHR1veHNjVYVktL31OQ_tgBUNHO5Uf3ACrvIVAW5cho&s=1EmIo8GEivgILxC4jZzEdBpYWt5R9CZ5cXhtr6i55rc&e=<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_notes_facebook-2Dengineering_more-2Ddetails-2Don-2Dtodays-2Doutage_431441338919_&d=DwMGaQ&c=n6-cguzQvX_tUIrZOS_4Og&r=WoGou9bjN14EvLKS6DHxfMEG6f2_bRhXNpedbbFoYDk&m=ceoMZY1q59Sow7WeapJsoNJW9M6RtvZkKXfu4rtpYcY&s=ryKQO6pOvAkwCIA0hQDtc2YA6zayXLUHgF1cQYftw6s&e=> On Thu, 14 Mar 2019 at 06:18, Mike Hammett <nanog@ics-il.net<mailto:nanog@ics-il.net>> wrote: So what happened at Facebook today? I saw one article quoting Roland saying it was a route leak, but I haven't seen any other sources that aren't just quoting Roland. Usually there are a few independent posts out there by now. ----- Mike Hammett Intelligent Computing Solutions https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ics-2Dil.com&d=DwIGaQ&c=n6-cguzQvX_tUIrZOS_4Og&r=r4NBNYp4yEcJxC11Po5I-w&m=IHR1veHNjVYVktL31OQ_tgBUNHO5Uf3ACrvIVAW5cho&s=EAZZC6r_-2rdFCKgq9XpQy30F7OH79M6sZPNvXq0FPA&e=<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ics-2Dil.com&d=DwMGaQ&c=n6-cguzQvX_tUIrZOS_4Og&r=WoGou9bjN14EvLKS6DHxfMEG6f2_bRhXNpedbbFoYDk&m=ceoMZY1q59Sow7WeapJsoNJW9M6RtvZkKXfu4rtpYcY&s=mqEeLmJWERmxCoZUEBneCbzo6adoCrpBOoe6fg-HoIk&e=> Midwest-IX https://urldefense.proofpoint.com/v2/url?u=http-3A__www.midwest-2Dix.com&d=DwIGaQ&c=n6-cguzQvX_tUIrZOS_4Og&r=r4NBNYp4yEcJxC11Po5I-w&m=IHR1veHNjVYVktL31OQ_tgBUNHO5Uf3ACrvIVAW5cho&s=fEoBWpTXgY7eXczzc8vo7VHbvopKqDWk6Xz2XYutL0k&e=<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.midwest-2Dix.com&d=DwMGaQ&c=n6-cguzQvX_tUIrZOS_4Og&r=WoGou9bjN14EvLKS6DHxfMEG6f2_bRhXNpedbbFoYDk&m=ceoMZY1q59Sow7WeapJsoNJW9M6RtvZkKXfu4rtpYcY&s=F-6Dzwg1bC3hEWn0_nXjswSlbrgbBSq3DLpcpMtdh-Q&e=>
No one looks at dates on Facebook posts. On Thu, Mar 14, 2019, 17:10 Luke Guillory <lguillory@reservetele.com> wrote:
That’s old.
By Robert Johnson on Thursday, September 23, 2010 at 7:29 PM
Luke
Ns
*From:* NANOG [mailto:nanog-bounces@nanog.org] *On Behalf Of *Selphie Keller *Sent:* Thursday, March 14, 2019 4:06 PM *To:* Mike Hammett *Cc:* NANOG list *Subject:* Re: FB?
I did see this article indicating they had somehow invalidated their cache in a botched deployment of changes - https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-o...
On Thu, 14 Mar 2019 at 06:18, Mike Hammett <nanog@ics-il.net> wrote:
So what happened at Facebook today? I saw one article quoting Roland saying it was a route leak, but I haven't seen any other sources that aren't just quoting Roland. Usually there are a few independent posts out there by now.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
Ah-ha, that is indeed the write-up I saw. 8 years old! On Thu, Mar 14, 2019, 2:07 PM Selphie Keller <selphie.keller@gmail.com> wrote:
I did see this article indicating they had somehow invalidated their cache in a botched deployment of changes - https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-o...
On Thu, 14 Mar 2019 at 06:18, Mike Hammett <nanog@ics-il.net> wrote:
So what happened at Facebook today? I saw one article quoting Roland saying it was a route leak, but I haven't seen any other sources that aren't just quoting Roland. Usually there are a few independent posts out there by now.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
The date on that is 2010. On Thu, Mar 14, 2019 at 2:07 PM Selphie Keller <selphie.keller@gmail.com> wrote:
I did see this article indicating they had somehow invalidated their cache in a botched deployment of changes - https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-o...
On Thu, 14 Mar 2019 at 06:18, Mike Hammett <nanog@ics-il.net> wrote:
So what happened at Facebook today? I saw one article quoting Roland saying it was a route leak, but I haven't seen any other sources that aren't just quoting Roland. Usually there are a few independent posts out there by now.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
-- Jeff Shultz Central Office Technician SCTC (503) 769-2125 Go Big Ask for Gig -- Like us on Social Media for News, Promotions, and other information!! <https://www.facebook.com/SCTCWEB/> <https://www.instagram.com/sctc_503/> <https://www.yelp.com/biz/sctc-stayton-3> <https://www.youtube.com/c/sctcvideos> _**** This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. ****_
Yes, evidently someone screenshotted it and it was making the rounds on social media this morning, sans the date. So now back to other theories. On Thu, Mar 14, 2019 at 2:16 PM Jeff Shultz <jeffshultz@sctcweb.com> wrote:
The date on that is 2010.
On Thu, Mar 14, 2019 at 2:07 PM Selphie Keller <selphie.keller@gmail.com> wrote:
I did see this article indicating they had somehow invalidated their
cache in a botched deployment of changes - https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-o...
On Thu, 14 Mar 2019 at 06:18, Mike Hammett <nanog@ics-il.net> wrote:
So what happened at Facebook today? I saw one article quoting Roland
saying it was a route leak, but I haven't seen any other sources that aren't just quoting Roland. Usually there are a few independent posts out there by now.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
-- Jeff Shultz Central Office Technician SCTC (503) 769-2125 Go Big Ask for Gig
-- Like us on Social Media for News, Promotions, and other information!!
<https://www.facebook.com/SCTCWEB/> <https://www.instagram.com/sctc_503/> <https://www.yelp.com/biz/sctc-stayton-3> <https://www.youtube.com/c/sctcvideos>
_**** This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. ****_
The cache invalidation thing is incorrect according to an Facebook SWE I talked to. He wouldn't tell me what it actually was though, basically saying "you have to know our infrastructure to understand and I can't tell you that." On Thu, Mar 14, 2019, 5:28 PM cosmo <clinton.mielke@gmail.com> wrote:
Yes, evidently someone screenshotted it and it was making the rounds on social media this morning, sans the date.
So now back to other theories.
On Thu, Mar 14, 2019 at 2:16 PM Jeff Shultz <jeffshultz@sctcweb.com> wrote:
The date on that is 2010.
On Thu, Mar 14, 2019 at 2:07 PM Selphie Keller <selphie.keller@gmail.com> wrote:
I did see this article indicating they had somehow invalidated their
cache in a botched deployment of changes - https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-o...
On Thu, 14 Mar 2019 at 06:18, Mike Hammett <nanog@ics-il.net> wrote:
So what happened at Facebook today? I saw one article quoting Roland
saying it was a route leak, but I haven't seen any other sources that aren't just quoting Roland. Usually there are a few independent posts out there by now.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
-- Jeff Shultz Central Office Technician SCTC (503) 769-2125 Go Big Ask for Gig
-- Like us on Social Media for News, Promotions, and other information!!
<https://www.facebook.com/SCTCWEB/> <https://www.instagram.com/sctc_503/> <https://www.yelp.com/biz/sctc-stayton-3> <https://www.youtube.com/c/sctcvideos>
_**** This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. ****_
participants (16)
-
Bryan Holloway -
cosmo -
Dobbins, Roland -
Jason Suter -
Jeff Shultz -
Jeroen Wunnink -
Kain, Rebecca (.) -
Luke Guillory -
Mike Hammett -
Pim van Stam -
Ray Van Dolson -
Robert Webb -
Ross Tajvar -
Selphie Keller -
Suresh Ramasubramanian -
Tom Beecher