On Mon, Jun 24, 2019 at 08:18:27AM -0400, Tom Paseka via NANOG wrote:
a Verizon downstream BGP customer is leaking the full table, and some more specific from us and many other providers.
It appears that one of the implicated ASNs, AS 33154 "DQE Communications LLC" is listed as customer on Noction's website: https://www.noction.com/clients/dqe I suspect AS 33154's customer AS 396531 turned up a new circuit with Verizon, but didn't have routing policies to prevent sending routes from 33154 to 701 and vice versa, or their router didn't have support for RFC 8212. I'd like to point everyone to an op-ed I wrote on the topic of "BGP optimizers": https://seclists.org/nanog/2017/Aug/318 So in summary, I believe the following happened: - 33154 generated fake more-specifics, which are not visible in the DFZ - 33154 announces those fake more-specifics to at least one customer (396531) - this customer (396531) propagated to to another upstream provider (701) - it appears that 701 did not sufficient prefix filtering, or a maximum-prefix limit While it is easy to point at the alleged BGP optimizer as the root cause, I do think we now have observed a cascading catastrophic failure both in process and technologies. Here are some recommendations that all of us can apply, that may have helped dampen the negative effects: - deploy RPKI based BGP Origin validation (with invalid == reject) - apply maximum prefix limits on all EBGP sessions - ask your router vendor to comply with RFC 8212 ('default deny') - turn off your 'BGP optimizers' I suspect we, collectively, suffered significant financial damage in this incident. Kind regards, Job