This is what looked happened:

There was a large scale BGP 'leak' incident causing about 20k prefixes for 2400 network (ASNs) to be rerouted through AS396531 (a steel plant) and then on to its transit provider: Verizon (AS701) Start time: 10:34:21 (UTC) End time: 12:37  (UTC)
All ASpaths had the following in common:
701 396531 33154


33154 (DQECOM ) is an ISP providing transit to 396531.
396531 is by the looks of it a steel plant. dual homed to 701 and 33154.
701 is verizon and accepted by the looks of it all BGP announcements from 396531

What appears to have happened is that 33154  those routes were propagated to 396531, which then send them to Verizon and voila... there is the full leak at work.
(DQECOM  runs a BGP optimizer (https://www.noction.com/clients/dqe , thanks Job for pointing that out, more below)

As a result traffic for 20k prefixes or so was now rerouted through verizon and 396531 (the steel plant)

We've seen numerous incidents like this in the past
lessons learned:
1) if you do use a BGP optimizer, please FILTER!
2) Verizon... filter your customers, please!


Since the BGP optimizer introduces new more specific routes, a lot of traffic for high traffic destinations would have been rerouted through that path, which would have been congested, causing the outages.
There were many cloudflare prefixes affected, but also folks like Amazon, Akamai, Facebook, Apple, Linode etc.

here's one example for Amazon - CloudFront : 52.84.32.0/22. Normally announced as a 52.84.32.0/21 but during the incident as a /22 (remember more specifics always win)
https://stat.ripe.net/52.84.32.0%2F22#tabId=routing&routing_bgplay.ignoreReannouncements=false&routing_bgplay.resource=52.84.32.0/22&routing_bgplay.starttime=1561337999&routing_bgplay.endtime=1561377599&routing_bgplay.rrcs=0,1,2,5,6,7,10,11,13,14,15,16,18,20&routing_bgplay.instant=null&routing_bgplay.type=bgp

RPKI would have worked here (assuming you're strict with the max length)!


Cheers
 Andree


My secret spy satellite informs me that Dmitry Sherman wrote On 2019-06-24, 3:55 AM:
Hello are there any issues with CloudFlare services now?

Dmitry Sherman
dmitry@interhost.net
Interhost Networks Ltd
Web: http://www.interhost.co.il
fb: https://www.facebook.com/InterhostIL
Office: (+972)-(0)74-7029881 Fax: (+972)-(0)53-7976157