Verizon 701 Route leak?
Anyone from Verizon want to comment on a possible route leak on the 25th 10:30 PM EDT? Saw route table jump up to 737629 routes last night from 701. Marcus Josephson mjosephson@inap.com<mailto:mjosephson@inap.com> * www.inap.com<http://www.inap.com> INAP (r) connectivity | colocation | managed hosting | cloud One Ravinia Drive * Suite 1300 * Atlanta * GA * 30346
Hello, Do you mean this one ? https://dyn.com/blog/large-bgp-leak-by-google-disrupts-internet-in-japan/ https://bgpmon.net/bgp-leak-causing-internet-outages-in-japan-and-beyond/ https://mobile.twitter.com/bgpmon/status/901565301048803329 Best regards.
Le 25 août 2017 à 08:58, Marcus Josephson <mjosephson@inap.com> a écrit :
Anyone from Verizon want to comment on a possible route leak on the 25th 10:30 PM EDT? Saw route table jump up to 737629 routes last night from 701.
Marcus Josephson
mjosephson@inap.com<mailto:mjosephson@inap.com> * www.inap.com<http://www.inap.com>
INAP (r) connectivity | colocation | managed hosting | cloud
One Ravinia Drive * Suite 1300 * Atlanta * GA * 30346
Damn you Google.. yup. Thanks for links. -Marcus From: Youssef Bengelloun-Zahr [mailto:bengelly@gmail.com] Sent: Monday, August 28, 2017 11:47 AM To: Marcus Josephson <mjosephson@inap.com> Cc: nanog@nanog.org Subject: Re: Verizon 701 Route leak? Hello, Do you mean this one ? https://dyn.com/blog/large-bgp-leak-by-google-disrupts-internet-in-japan/ https://bgpmon.net/bgp-leak-causing-internet-outages-in-japan-and-beyond/ https://mobile.twitter.com/bgpmon/status/901565301048803329 Best regards. Le 25 août 2017 à 08:58, Marcus Josephson <mjosephson@inap.com<mailto:mjosephson@inap.com>> a écrit : Anyone from Verizon want to comment on a possible route leak on the 25th 10:30 PM EDT? Saw route table jump up to 737629 routes last night from 701. Marcus Josephson mjosephson@inap.com<mailto:mjosephson@inap.com><mailto:mjosephson@inap.com> * www.inap.com<http://www.inap.com><http://www.inap.com> INAP (r) connectivity | colocation | managed hosting | cloud One Ravinia Drive * Suite 1300 * Atlanta * GA * 30346
On Mon, Aug 28, 2017 at 03:48:44PM +0000, someone wrote:
Damn you Google.. yup.
I am not sure it is fair to say "damn you Google", because accidents happen (be it through human error or software defects). All of us have entered commands at some point and subsequently https://media.giphy.com/media/bI5BEfwbdVPcA/giphy.gif Software defects are a real risk as well: Major BGP implementations have had bugs which caused NO_EXPORT functionality to malfunction, or bugs where random pieces of your configuration are ignored (for instance the part of the configuration which contains a prefix-filter). Allegedly the famous AS 7007 incident was also caused by a software defect. The key concept here is that, since we know accidents an happen, we ought to look out for each other and impose multiple layers of protection for our mutual benefit. Mechanisms that come to mind are maximum prefix limits and coarse as-path filters. At NANOG67 I described a few of these tricks https://www.youtube.com/watch?v=CSLpWBrHy10 / https://www.nanog.org/sites/default/files/Snijders_Everyday_Practical_Bgp.pd... To further reduce the risk surface, it may be good to ask your BGP vendor for specific behaviour applicable to EBGP sessions: ask for RFC 8212 support. RFC 8212 defines that when you configure an EBGP session without any routing policy associated with that neighbor, no routes will be exchanged until it is explicitly instructed to do so. Finally, it may be worthwhile exploring if we can standardize and promote maximum prefix limits applied on the the _sending_ side. This way you protect your neighbor (and the Internet at large) by self-destructing when you inadvertently announce more than what you'd expect to announce. BIRD has this functionality http://bird.network.cz/?get_doc&f=bird-3.html#proto-export-limit however I am not aware of other implementations. Feedback welcome! Kind regards, Job
On 28/08/17 18:34, Job Snijders wrote:
Finally, it may be worthwhile exploring if we can standardize and promote maximum prefix limits applied on the the _sending_ side. This way you protect your neighbor (and the Internet at large) by self-destructing when you inadvertently announce more than what you'd expect to announce. BIRD has this functionality http://bird.network.cz/?get_doc&f=bird-3.html#proto-export-limit however I am not aware of other implementations. Feedback welcome!
Having just dug up the reference for some strange reason... Back at NANOG38 (2006) Tom Scholl mentioned in a talk on max prefix: "Perhaps maximum-prefix outbound? (Suggested by Eric Bell years ago)" https://www.nanog.org/meetings/nanog38/presentations/scholl-maxpfx.pdf Notably Juniper does now have prefix-export-limit, but only for readvertisement into IS-IS or OSPF: https://www.juniper.net/documentation/en_US/junos/topics/reference/configura...
Damn you Google.. yup. Thanks for links. A public post-mortem would be highly appreciated (from all parties).
there has been more press hysteria on this than actual packet droppage. goog fat fingered or otherwise misannounced a numer of large consumer isp's prefixes. the leak was for aybe 20 minutes. almost no one over here noticed. but the press, isoc, ... said "japan knocked off the internet." take that as a calibration of the press, isoc, ... randy
Good use-case for https://datatracker.ietf.org/doc/draft-ietf-grow-bmp-adj-rib-out and snapshot auditing before and after changes. Leak didn't last long but it could have been caught within milliseconds verses minutes via oh sh** alarms. --Tim On 8/29/17, 6:46 PM, "NANOG on behalf of Randy Bush" <nanog-bounces@nanog.org on behalf of randy@psg.com> wrote: >> Damn you Google.. yup. Thanks for links. > A public post-mortem would be highly appreciated (from all parties). there has been more press hysteria on this than actual packet droppage. goog fat fingered or otherwise misannounced a numer of large consumer isp's prefixes. the leak was for aybe 20 minutes. almost no one over here noticed. but the press, isoc, ... said "japan knocked off the internet." take that as a calibration of the press, isoc, ... randy
Good use-case for https://datatracker.ietf.org/doc/draft-ietf-grow-bmp-adj-rib-out and snapshot auditing before and after changes. Leak didn't last long but it could have been caught within milliseconds verses minutes via oh sh** alarms.
[ i happen to like bmp, but ... ] if the sender did not have the automation or the mops to not leak in the first place, how well will they apply post hoc detection and repair? if the receiver did not filter, and an tier-1 as-path filter would have sufficed in this case, how well do you think they will be at applying post hoc detection and repair? this was an easily preventable ops failure. but what we will do is go to idr and grow and invent 42 more hacks, kinda like ipv6 transition mechanisms. </snark> randy
participants (7)
-
Job Snijders
-
Julien Goodwin
-
Marcus Josephson
-
Mikael Abrahamsson
-
Randy Bush
-
Tim Evens (tievens)
-
Youssef Bengelloun-Zahr