Re: BGP Optimizers (Was: Validating possible BGP MITM attack)
Hi Job, I believe your disclaimer makes a lot of sense. From our perspective using more specifics is one of the options to make BGP follow the optimized path instead of the « natural » path. We used to be doing more specifics because with the same prefix being announced, we were simply not getting a best route announced back to the optimiser. Since the adoption of BGP ADD-PATH, our solution does not need to use more specifics to maintain a full collection of the routers BGP table. (In addition, it has actually never been a strong a requirement due to the use of other SNMP collection processes.) Therefore LOCAL_PREF is the option we advise and implement. The examples you mention confirm the issues are mainly due to poorly configured networks where routes are leaked out although they shouldn’t be. Adequate routers are able to filter out prefixes based on attributes like communities, which we set by default. We’ve had an instance of such an issue with one of our customers a few years ago; it turned out to be mistaken CLI commands that the engineer gave to the router. Our XCA software service and platform has hundreds of ASs running for years and none are making any noise. Another point of discussion is the fact that transit and large content providers actually accept thousands of routes coming from anywhere, there is a lot of room for optimization. And I know how much you personally try to contribute to enhance this. There actually is a reason for operating BGP optimizers. The BGP protocol, while robust and scalable, doesn't know anything about link capacity, doesn’t apply performance analytics and can easily drive links into saturation, introducing packet loss. Also, it is not aware of commercial agreements like CDR, generating costs that could be prevented. It also, of course, ignores the performance of available paths. All of the above actually impacts customer traffic and business performance. Since a few years we see our Customers take more care of quality and capacity management… and stop relying on BGP « blindly ». Most transit providers like to explain that their service are premium and that’s the reason why their prices are premium. But when you look at actual performance measurements, some premium providers are actually just behind the cheaper ones. I’m in RIPE 76 tomorrow, I’ll be more than happy to discuss this topic further with you. Kind regards, François (I’m a product engineer at Border 6 - Expereo, a BGP optimization software company.) François DEVIENNE Mobile: +33.651.937.927 E-mail: francois.devienne@expereo.com<mailto:francois.devienne@expereo.com> BORDER 6 S.A.S. - EXPEREO On Aug 31, 2017 (35), at 22:06, Job Snijders <job@ntt.net<mailto:job@ntt.net>> wrote: Dear all, disclaimer: [ The following is targetted at the context where a BGP optimizer generates BGP announcement that are ordinarily not seen in the Default-Free Zone. The OP indicated they announce a /23, and were unpleasantly surprised to see two unauthorized announcements for /24 more-specifics pop up in their alerting system. No permission was granted to create and announce these more-specifics. The AS_PATH for those /24 announcements was entirely fabricated. Original thread https://mailman.nanog.org/pipermail/nanog/2017-August/092124.html ] On Thu, Aug 31, 2017 at 11:13:18AM -0700, Andy Litzinger wrote: Presuming it was a route optimizer and the issue was ongoing, what would be the suggested course of action? I strongly recommend to turn off those BGP optimizers, glue the ports shut, burn the hardware, and salt the grounds on which the BGP optimizer sales people walked. It is extremely irresponsible behavior to use software that generates _fake_ BGP more-specifics for the purpose of traffic engineering. You simply cannot expect that those more-specifics will never escape into the global DFZ! Relying on NO_EXPORT is not sufficient: we regularly see software bugs related to NO_EXPORT, and community-squashing configuration mistakes happen all the time. Consider the following: if you leak your own internal more-specifics, at least you are the legitimate destination. (You may suffer from suboptimal routing, but it isn't guaranteed downtime.) However if you generate fake more-specifics for prefixes belonging to OTHER organisations, you essentially are complicit in BGP hijacking. If those fake more-specifics accidentally leak into the DFZ, you are bringing down the actual owner of such prefixes, and depriving people from access to the Internet. Example case: https://mailman.nanog.org/pipermail/nanog/2013-January/054846.html reach out to those 2 AS owners and see if they could stop it? Yes, absolutely! And if everyone of the affected parties of this localized hijack leak (or should we say 'victims') reaches out to the wrongdoers, they contribute peer pressure to rectify the situation. Just make sure you assign blame to the correct party. :) Or is it something I just have to live with as a traffic engineering solution they are using and mark the alerts as false positives? No, this is not something we should just accept. The Default-Free Zone is a shared resource. Anyone using "BGP optimizers" is not only risking their own online presence, but also everyone else's. Using a BGP optimizer is essentially trading a degree of risk to society for the purpose of saving a few bucks or milliseconds. It is basically saying: "The optimizer helps me, but may hurt others, and I am fine with that". An analogy: We don't want transporters of radioactive material to cut corners by using non-existant roads to bring the spent nuclear fuels from A to B faster, nor do we want them to rely on non-robust shielding that works "most of the time". We share the BGP DFZ environment, 'BGP optimizers' are downright pollution contributors. Kind regards, Job ps. If you want to do BGP optimization: don't have the BGP optimizer generate fake BGP announcements, but focus only on manipulating non-transitive attributes (like LOCAL_PREF) on real announcements you actually received from others. Or Perhaps BGP optimizers shouldn't even talk BGP but rather push freshly generated TE-optimized routing-policy to your edge boxes. Or perhaps the 'BGP optimizer' vendors should come to IETF to figure out a safe way (maybe a new AFI/SAFI to manipulate real announcements)... there are ways to do this, without risk to others! p.s. providing a publicly available BGP looking glasses will contribute to proving your innocence in cases like these. Since in many cases the AS_PATH is a complete fabrication, we need to manually check every AS in the AS_PATH to see whether the AS carries the fake more-specific. A public looking glass speeds up this fault-finding process. If you don't want to host a webinterface yourself, please consider sending a BGP feed to the Route Views Project or RIPE RIS, or for something queryable in a real-time fashion the NLNOG RING Looking Glass http://lg.ring.nlnog.net/
Dear Francois, On Thu, May 17, 2018 at 10:14:19AM +0000, Francois Devienne wrote:
The examples you mention confirm the issues are mainly due to poorly configured networks where routes are leaked out although they shouldn’t be. Adequate routers are able to filter out prefixes based on attributes like communities, which we set by default.
Question: is your implementation setting NO_EXPORT by default, or some other communities? Not all BGP "Optimizers" set NO_EXPORT by default... in that context I am not sure it is fair to say "the network is poorly configured" - when an "optimizer" doesn't set NO_EXPORT it is the "optimizer" that is comes with poor defaults. And there is another challenge: NO_EXPORT does not always work correctly. Software defects happen, and are unpredictable. All major routing platforms have seen bugs where for one reason or another NO_EXPORT does not (or no longer) work correctly. Heavy reliance on NO_EXPORT can be a weakness in network architecture.
There actually is a reason for operating BGP optimizers. The BGP protocol, while robust and scalable, doesn't know anything about link capacity, doesn’t apply performance analytics and can easily drive links into saturation, introducing packet loss. Also, it is not aware of commercial agreements like CDR, generating costs that could be prevented. It also, of course, ignores the performance of available paths. All of the above actually impacts customer traffic and business performance. Since a few years we see our Customers take more care of quality and capacity management… and stop relying on BGP « blindly ».
You are correct that there is a need (and a market) for BGP optimizers. BGP is terrible for routing around problematic parts of the topology if we look at metrics such as latency or packetloss. In my opinion, what the "optimizer" vendors _should_ have done (years ago), is go to IETF to the IDR working group and help standardize a safe and robust way to extend the BGP protocol to allow for better traffic engineering. Think along the lines what flowspec is for firewalls, perhaps there should be a "traffic engineering spec (tespec)" for routers. This should happen in a different Subsequent AFI to avoid the extremely risky behaviour we now see in the Unicast SAFI. I have no problem with software that automatically interacts with routers to manipulate LOCALPREF, there is no issue if software supresses more-specifics, prepends of the own ASN is no issue either, but what _is_ an issue is any software that generates fake routes and distributes those fake routes in the IPv4/IPv6 Unicast AFI/SAFI on boxes connected to the BGP DFZ. To phrase and summarize the isuse, using the terminology you provided: The "optimized" paths MUST NEVER be distributed in the same AFI/SAFI as the "natural" paths. Overloading the "natural" AFI/SAFI with fake generated more-specifics, which only exist for the purpose of outbound traffic engineering (even with communities) is a very dangerous thing. Yes, this will be a bit of work, but that is the cost of doing business.
(I’m a product engineer at Border 6 - Expereo, a BGP optimization software company.)
I appreciate your outreach in this public forum and the disclosure. Kind regards, Job
participants (2)
-
Francois Devienne
-
Job Snijders