This is what looked happened: There was a large scale BGP 'leak' incident causing about 20k prefixes for 2400 network (ASNs) to be rerouted through AS396531 (a steel plant) and then on to its transit provider: Verizon (AS701) Start time: 10:34:21 (UTC) End time: 12:37 (UTC) All ASpaths had the following in common: 701 396531 33154 33154 (DQECOM ) is an ISP providing transit to 396531. 396531 is by the looks of it a steel plant. dual homed to 701 and 33154. 701 is verizon and accepted by the looks of it all BGP announcements from 396531 What appears to have happened is that 33154 those routes were propagated to 396531, which then send them to Verizon and voila... there is the full leak at work. (DQECOM runs a BGP optimizer (https://www.noction.com/clients/dqe , thanks Job for pointing that out, more below) As a result traffic for 20k prefixes or so was now rerouted through verizon and 396531 (the steel plant) We've seen numerous incidents like this in the past lessons learned: 1) if you do use a BGP optimizer, please FILTER! 2) Verizon... filter your customers, please! Since the BGP optimizer introduces new more specific routes, a lot of traffic for high traffic destinations would have been rerouted through that path, which would have been congested, causing the outages. There were many cloudflare prefixes affected, but also folks like Amazon, Akamai, Facebook, Apple, Linode etc. here's one example for Amazon - CloudFront : 52.84.32.0/22. Normally announced as a 52.84.32.0/21 but during the incident as a /22 (remember more specifics always win) https://stat.ripe.net/52.84.32.0%2F22#tabId=routing&routing_bgplay.ignoreReannouncements=false&routing_bgplay.resource=52.84.32.0/22&routing_bgplay.starttime=1561337999&routing_bgplay.endtime=1561377599&routing_bgplay.rrcs=0,1,2,5,6,7,10,11,13,14,15,16,18,20&routing_bgplay.instant=null&routing_bgplay.type=bgp RPKI would have worked here (assuming you're strict with the max length)! Cheers Andree My secret spy satellite informs me that Dmitry Sherman wrote On 2019-06-24, 3:55 AM:
Hello are there any issues with CloudFlare services now?
Dmitry Sherman dmitry@interhost.net Interhost Networks Ltd Web: http://www.interhost.co.il fb: https://www.facebook.com/InterhostIL Office: (+972)-(0)74-7029881 Fax: (+972)-(0)53-7976157