Performance metrics used in commercial BGP route optimizers
Hi, I'm doing a research on BGP route optimisation and the performance metrics used by commercial route optimizer appliances to select better path to a prefix. I would appreciate any information about the performance metrics used in commercial BGP route optimizers, white papers or any other document that describes how these metrics are measured and collected by commercial route optimizers. Thanks -- Dimeji Fayomi
The answers which you seek would be considered secret sauce to these vendors. But you can start at running MTRs through a VRF per carrier only containing a default route, and looking at the results. Ryan On Tue, Jul 16, 2019 at 6:11 AM -0700, "Dimeji Fayomi" <oof1@students.waikato.ac.nz<mailto:oof1@students.waikato.ac.nz>> wrote: Hi, I'm doing a research on BGP route optimisation and the performance metrics used by commercial route optimizer appliances to select better path to a prefix. I would appreciate any information about the performance metrics used in commercial BGP route optimizers, white papers or any other document that describes how these metrics are measured and collected by commercial route optimizers. Thanks -- Dimeji Fayomi
The most important metric for a BGP optimizer is how much it physically weighs. That way you'll know if you can carry it to the trash pile yourself, or need to get help so you don't hurt your back. :) On Tue, Jul 16, 2019 at 9:21 AM Ryan Hamel <Ryan.Hamel@quadranet.com> wrote:
The answers which you seek would be considered secret sauce to these vendors.
But you can start at running MTRs through a VRF per carrier only containing a default route, and looking at the results.
Ryan
On Tue, Jul 16, 2019 at 6:11 AM -0700, "Dimeji Fayomi" < oof1@students.waikato.ac.nz> wrote:
Hi,
I'm doing a research on BGP route optimisation and the performance metrics used by commercial route optimizer appliances to select better path to a prefix.
I would appreciate any information about the performance metrics used in commercial BGP route optimizers, white papers or any other document that describes how these metrics are measured and collected by commercial route optimizers.
Thanks -- Dimeji Fayomi
Tom Beecher wrote : The most important metric for a BGP optimizer is how much it physically weighs. That way you'll know if you can carry it to the trash pile yourself, or need to get help so you don't hurt your back.
Please dispose of it in an environment friendly way. In the city I live and many other ones we have programs to get rid of such garbage in a proper way. https://www.roseville.ca.us/residents/utility_exploration_center/e-waste Stop the pollution of the routing table and the planet !
Mark Tinka wrote : So I got this from BGPmon, earlier today:
Oh yeah, a /32 sourced from a private ASN without no-export. Michel TSI Disclaimer: This message and any files or text attached to it are intended only for the recipients named above and contain information that may be confidential or privileged. If you are not the intended recipient, you must not forward, copy, use or otherwise disclose this communication or the information contained herein. In the event you have received this message in error, please notify the sender immediately by replying to this message, and then delete all copies of it from your system. Thank you!...
On Tue, Jul 16, 2019, 4:11 PM Dimeji Fayomi <oof1@students.waikato.ac.nz> wrote:
I'm doing a research on BGP route optimisation and the performance metrics used by commercial route optimizer appliances to select better path to a prefix.
You may have discovered that already during your research, but just in case: basically, using those optimizers at full throttle is a bad practice and is generally discouraged. A research into the deep-juju of BGP optimization is roughly equivalent to a research about how alcohol may make you a faster driver. I.e. it's fine in academy but you certainly may want to emphasize security considerations in your paper. -- Töma
Most of which are bunk if you and your upstream have appropriate filters. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Töma Gavrichenkov" <ximaera@gmail.com> To: "Dimeji Fayomi" <oof1@students.waikato.ac.nz> Cc: "NANOG" <nanog@nanog.org> Sent: Tuesday, July 16, 2019 8:30:37 AM Subject: Re: Performance metrics used in commercial BGP route optimizers On Tue, Jul 16, 2019, 4:11 PM Dimeji Fayomi < oof1@students.waikato.ac.nz > wrote: I'm doing a research on BGP route optimisation and the performance metrics used by commercial route optimizer appliances to select better path to a prefix. You may have discovered that already during your research, but just in case: basically, using those optimizers at full throttle is a bad practice and is generally discouraged. A research into the deep-juju of BGP optimization is roughly equivalent to a research about how alcohol may make you a faster driver. I.e. it's fine in academy but you certainly may want to emphasize security considerations in your paper. -- Töma <blockquote> </blockquote>
On Tue, Jul 16, 2019, 5:49 PM Mike Hammett <nanog@ics-il.net> wrote:
Most of which are bunk if you and your upstream have appropriate filters.
True, and, while we're at it, it's okay to drink and drive a car if the manufacturer has built enough driver assistance systems in it. -- Töma
More like do whatever you want in your own house as long as you don't infringe upon others. The argument against route optimizers (assuming appropriate ingress\egress filters) is a religious one and should be treated as such. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Töma Gavrichenkov" <ximaera@gmail.com> To: "Mike Hammett" <nanog@ics-il.net> Cc: "NANOG" <nanog@nanog.org>, "Dimeji Fayomi" <oof1@students.waikato.ac.nz> Sent: Tuesday, July 16, 2019 9:53:46 AM Subject: Re: Performance metrics used in commercial BGP route optimizers On Tue, Jul 16, 2019, 5:49 PM Mike Hammett < nanog@ics-il.net > wrote: Most of which are bunk if you and your upstream have appropriate filters. True, and, while we're at it, it's okay to drink and drive a car if the manufacturer has built enough driver assistance systems in it. -- Töma
So I got this from BGPmon, earlier today: ***** You received this email because you are subscribed to BGPmon.net. For more details about these updates please visit: https://portal.bgpmon.net/myalerts.php ==================================================================== RPKI Validation Failed (Code: 9) ==================================================================== Your prefix: 105.16.0.0/12: Prefix Description: In case of abuse, please contact abuse@seacom.mu Update time: 2019-07-16 15:45 (UTC) Detected by #peers: 1 Detected prefix: 105.26.205.62/32 Announced by: AS65075 (-Private Use AS-) Upstream AS: AS53089 (IDC TELECOM LTDA EPP, BR) ASpath: 53089 65075 Alert details: https://portal.bgpmon.net/alerts.php?details&alert_id=89182454 Mark as false alert: https://portal.bgpmon.net/fp.php?aid=89182454 RPKI Status: ROA validation failed: Invalid Prefix-Length + Invalid Origin ASN, expected 37100 -------------------------------------------------------------- *for questions regarding the change code or other question, please see: https://portal.bgpmon.net/faq.php Latest BGPmon news: http://bgpmon.net/blog/ * Popular Destinations rerouted to Russia * Today’s BGP leak in Brazil * BGP leak causing Internet outages in Japan and beyond. . ***** See why we like (not!) BGP optimizers so much? Mark.
On Tue, Jul 16, 2019 at 3:33 PM Mike Hammett <nanog@ics-il.net> wrote:
More like do whatever you want in your own house as long as you don't infringe upon others.
That's where the rub is; when using "BGP optimisers" to influence public Internet routing, you cannot guarantee you won't infringe upon others.
The argument against route optimizers (assuming appropriate ingress\egress filters) is a religious one and should be treated as such.
The argument against "BGP optimizers" is that we *cannot* assume appropriate ingress or egress filters. It appears to me like fallacy to suggest a line of reasoning ala "if you do things correctly, things won't go wrong". Clearly we've observed many times over that things *do* go wrong. Some examples: almost every year one of the major BGP vendors has a serious bug related to the functionality to NO_EXPORT in some release. Also, routinely we observe there are software defects that cause a device to behave different (read 'leak') than how the operator had explicitly configured the device. These are facts, not religious statements. Perhaps in a bug-free world there is room for dangerous activities, but there is no such thing as bug-free. And I haven't even covered the human error angle. We must robustly architect our networks to mitigate or dampen the negative effects of issues at all layers of the stack. I consider it wholly inappropriate to write-off the countless hours spend dealing with fallout from "BGP optimizers" and the significant financial damages we've sustained as "religious arguments". Kind regards, Job
Job Snijders wrote on 16/07/2019 18:41:
I consider it wholly inappropriate to write-off the countless hours spend dealing with fallout from "BGP optimizers" and the significant financial damages we've sustained as "religious arguments".
it would be interesting to see research into the financial losses experienced by people and organisations across the internet caused by routing outages relating to bgp optimisers. Nick
Nowhere near the number as an engineer fat fingering a route. There are ISPs that accept routes all the way to /32 or /128, for traffic engineering with ease, and/or RTBH. Ryan -----Original Message----- From: NANOG <nanog-bounces@nanog.org> On Behalf Of Nick Hilliard Sent: Tuesday, July 16, 2019 11:04 AM To: Job Snijders <job@instituut.net> Cc: NANOG <nanog@nanog.org> Subject: Re: Performance metrics used in commercial BGP route optimizers Job Snijders wrote on 16/07/2019 18:41:
I consider it wholly inappropriate to write-off the countless hours spend dealing with fallout from "BGP optimizers" and the significant financial damages we've sustained as "religious arguments".
it would be interesting to see research into the financial losses experienced by people and organisations across the internet caused by routing outages relating to bgp optimisers. Nick
On Tue, Jul 16, 2019 at 6:10 PM Ryan Hamel <Ryan.Hamel@quadranet.com> wrote:
Nowhere near the number as an engineer fat fingering a route.
How are you able to make that assertion?
There are ISPs that accept routes all the way to /32 or /128, for traffic engineering with ease, and/or RTBH.
This strikes me as a bit of a red herring. Aren't the damaging effects of "BGP optimisers" *amplified* (not caused!) by ISPs who accept "all routes"? An ISP accepting incorrect routing information still is a step below entities actively generating and distributing incorrect routing information. Kind regards, Job
Oh I don't know about that. There's been a pile of high profile incidents which have been associated with "BGP optimisers", affecting connectivity to huge chunks of the internet, world-wide. It's not unusual for a single incident to have widespread or even global effect, and what with the Internet playing such an important part of the world's economies, it's hard not to be curious about the overall financial impact of this sort of thing. Nick Ryan Hamel wrote on 16/07/2019 19:10:
Nowhere near the number as an engineer fat fingering a route. There are ISPs that accept routes all the way to /32 or /128, for traffic engineering with ease, and/or RTBH.
Ryan
-----Original Message----- From: NANOG <nanog-bounces@nanog.org> On Behalf Of Nick Hilliard Sent: Tuesday, July 16, 2019 11:04 AM To: Job Snijders <job@instituut.net> Cc: NANOG <nanog@nanog.org> Subject: Re: Performance metrics used in commercial BGP route optimizers
Job Snijders wrote on 16/07/2019 18:41:
I consider it wholly inappropriate to write-off the countless hours spend dealing with fallout from "BGP optimizers" and the significant financial damages we've sustained as "religious arguments".
it would be interesting to see research into the financial losses experienced by people and organisations across the internet caused by routing outages relating to bgp optimisers.
Nick
On Tue, Jul 16, 2019 at 5:22 PM Nick Hilliard <nick@foobar.org> wrote:
Oh I don't know about that. There's been a pile of high profile incidents which have been associated with "BGP optimisers", affecting connectivity to huge chunks of the internet, world-wide.
How many, exactly and with a pointer/reference, have been 'not an optimizer' ? I almost made the same post as Mr Hammett ~4 messages (his messages) back made: "Err, all of these problems are possible without an optimizer", except I don't really believe that it's helpful: "All my friends failed out, but I just got D's..." isn't really selling this as a great plan. I suppose if your (not nick's) story is: "Well, I know what I'm doing, my upstreams filter me, I'd NEVER leak..." ok, but really no, not ok... someone 'not you' will eventually operate part of your network and fail where you hadn't. -chris
All of the same tragedy can happen without BGP optimizers, and does. BGP optimizers only harm the global Internet when route filters don't do their job. (Un)Fortunately, many other things also harm the global Internet when route filters don't do their job. Things other than BGP optimizers harm the global Internet more frequently via the same vector (lack of proper route filters). A given set of bugs are unlikely to affect both Optimizer edge egress filters and upstream ingress filters. If so, the Internet as a whole has much graver things to worry about. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Job Snijders" <job@instituut.net> To: "Mike Hammett" <nanog@ics-il.net> Cc: "Töma Gavrichenkov" <ximaera@gmail.com>, "NANOG" <nanog@nanog.org> Sent: Tuesday, July 16, 2019 12:41:13 PM Subject: Re: Performance metrics used in commercial BGP route optimizers On Tue, Jul 16, 2019 at 3:33 PM Mike Hammett < nanog@ics-il.net > wrote: More like do whatever you want in your own house as long as you don't infringe upon others. That's where the rub is; when using "BGP optimisers" to influence public Internet routing, you cannot guarantee you won't infringe upon others. <blockquote> The argument against route optimizers (assuming appropriate ingress\egress filters) is a religious one and should be treated as such. </blockquote> The argument against "BGP optimizers" is that we *cannot* assume appropriate ingress or egress filters. It appears to me like fallacy to suggest a line of reasoning ala "if you do things correctly, things won't go wrong". Clearly we've observed many times over that things *do* go wrong. Some examples: almost every year one of the major BGP vendors has a serious bug related to the functionality to NO_EXPORT in some release. Also, routinely we observe there are software defects that cause a device to behave different (read 'leak') than how the operator had explicitly configured the device. These are facts, not religious statements. Perhaps in a bug-free world there is room for dangerous activities, but there is no such thing as bug-free. And I haven't even covered the human error angle. We must robustly architect our networks to mitigate or dampen the negative effects of issues at all layers of the stack. I consider it wholly inappropriate to write-off the countless hours spend dealing with fallout from "BGP optimizers" and the significant financial damages we've sustained as "religious arguments". Kind regards, Job
Peace, On Tue, Jul 16, 2019, 9:24 PM Mike Hammett <nanog@ics-il.net> wrote:
BGP optimizers only harm the global Internet when route filters don't do their job. (Un)Fortunately, many other things also harm the global Internet when route filters don't do their job.
That is correct; however, there are potentially harmful things that cannot easily be avoided (so we accept the risk), and optimizers are not among those. E.g. there are quite a number of things which could cause an airplane to crash yet are still allowed onboard; but this alone isn't an argument convincing enough to allow handguns there. -- Töma
On Tue, Jul 16, 2019 at 01:24:11PM -0500, Mike Hammett wrote:
All of the same tragedy can happen without BGP optimizers, and does.
I disagree. You are skipping over crucial distinction we should make between common 'route leaks' (incorrect propagation of valid routing information), and the poison that is 'bgp optimiser hijacks' (propagating of invalid/nonexistent routing information). In the first case, a simple leak of existing real routing information, you'll often see that the outcomes of the leak have a longer AS_PATH, and that the leaking ASN has an actual path towards the destination. In the best case the leaked routes are ignored because they don't become the best path, in the worst case anyone using those leaked paths suffers from congestion. In the second case, leaked routes that came from a so-called 'bgp optimiser', during the leak there is no forwarding path to the actual destination. The packets circulate in a loop and never arrive at the intended destination. This is hard downtime for the affected prefixes. We also often see that the AS_PATH is entirely fabricated by "BGP optimisers", further increasing the risk of the hijacked route announcements being used.
BGP optimizers only harm the global Internet when route filters don't do their job. (Un)Fortunately, many other things also harm the global Internet when route filters don't do their job. Things other than BGP optimizers harm the global Internet more frequently via the same vector (lack of proper route filters).
A given set of bugs are unlikely to affect both Optimizer edge egress filters and upstream ingress filters. If so, the Internet as a whole has much graver things to worry about.
I believe it is a fallacy to state that "because other things can harm the Internet" it would be somehow become OK to use a BGP optimiser. It is not, it is extremely dangerous for those networks whose prefixes are being 'optimised' (née hijacked). Every day we see negative effects as a result from "bgp optimizers". We also have observed that some of the 'bgp optimizers' have consciously chosen to not apply even the most basic of harm reduction methods, see https://twitter.com/JobSnijders/status/1143205986787831819 We can't stop people from deploying this type of software, the Internet simply doesn't provide that kind of regulatory environment, but one should be fully aware of the terrible risks involved when doing so. Networks should be cognizant of peers they suspect are using such software to steer traffic. Kind regards, Job
On 16/07/2019 20:41, Job Snijders wrote:
On Tue, Jul 16, 2019 at 3:33 PM Mike Hammett <nanog@ics-il.net <mailto:nanog@ics-il.net>> wrote:
More like do whatever you want in your own house as long as you don't infringe upon others.
That's where the rub is; when using "BGP optimisers" to influence public Internet routing, you cannot guarantee you won't infringe upon others.
The argument against route optimizers (assuming appropriate ingress\egress filters) is a religious one and should be treated as such.
There is a difference between BGP optimizers and route optimizers. When was the last time you heard a complain about Akamai screwing up the global routing table over the past 12 years: https://www.akamai.com/us/en/about/news/press/2007-press/akamai-introduces-a... https://developer.akamai.com/legacy/learn/Optimization/SureRoute.html -Hank
On Wed, Jul 17, 2019 at 12:38 AM Hank Nussbacher <hank@efes.iucc.ac.il> wrote:
On 16/07/2019 20:41, Job Snijders wrote:
On Tue, Jul 16, 2019 at 3:33 PM Mike Hammett <nanog@ics-il.net> wrote:
More like do whatever you want in your own house as long as you don't infringe upon others.
That's where the rub is; when using "BGP optimisers" to influence public Internet routing, you cannot guarantee you won't infringe upon others.
The argument against route optimizers (assuming appropriate ingress\egress filters) is a religious one and should be treated as such.
There is a difference between BGP optimizers and route optimizers. When was the last time you heard a complain about Akamai screwing up the global routing table over the past 12 years:
https://www.akamai.com/us/en/about/news/press/2007-press/akamai-introduces-a...
https://developer.akamai.com/legacy/learn/Optimization/SureRoute.html
-Hank
Along these same lines I'd like to point out that nearly all or possibly even all incidents in recent memory are attributable to a single product whereas there has been at least one other product on the market for something like 15+ years that AFAIK has not had a single incident associated with it (and also does not create more specific prefixes as part of its operation). So is it really that one product is spoiling the market for the rest here or are they all bad? -- [stillwaxin@gmail.com ~]$ cat .signature cat: .signature: No such file or directory [stillwaxin@gmail.com ~]$
I was attracted to BGP route optimizers for latency/jitter reduction and partial black hole detection scenarios. Our traffic is low enough in volume that we aren't playing the circuit commit game, but rather optimizing the path to VoIP customers who don't care that provider Y in path X-Y-Z had a fiber cut causing issues with their phone service. They are a piece of the SDN and automation fun. Hopefully the vendors will devise ways of dealing with traffic load balancing without splitting prefixes.Or maybe RPKI will become more ubiquitous and leaks won't be as painful. Similar to how DNSSEC led many ISPs to remove their DNS redirecting "search services". On Wed, Jul 17, 2019 at 10:02 AM Michael Still <stillwaxin@gmail.com> wrote:
On Wed, Jul 17, 2019 at 12:38 AM Hank Nussbacher <hank@efes.iucc.ac.il> wrote:
On 16/07/2019 20:41, Job Snijders wrote:
On Tue, Jul 16, 2019 at 3:33 PM Mike Hammett <nanog@ics-il.net> wrote:
More like do whatever you want in your own house as long as you don't
infringe upon others.
That's where the rub is; when using "BGP optimisers" to influence public
Internet routing, you cannot guarantee you won't infringe upon others.
The argument against route optimizers (assuming appropriate
ingress\egress filters) is a religious one and should be treated as such.
There is a difference between BGP optimizers and route optimizers. When
was the last time you heard a complain about Akamai screwing up the global routing table over the past 12 years:
https://www.akamai.com/us/en/about/news/press/2007-press/akamai-introduces-a...
https://developer.akamai.com/legacy/learn/Optimization/SureRoute.html
-Hank
Along these same lines I'd like to point out that nearly all or possibly even all incidents in recent memory are attributable to a single product whereas there has been at least one other product on the market for something like 15+ years that AFAIK has not had a single incident associated with it (and also does not create more specific prefixes as part of its operation). So is it really that one product is spoiling the market for the rest here or are they all bad?
-- [stillwaxin@gmail.com ~]$ cat .signature cat: .signature: No such file or directory [stillwaxin@gmail.com ~]$
You can achieve performance based routing using latency/jitter and partial blackhole detection as your metrics without resorting to prefix splitting - adjust local preferences on received routes, don’t install received routes, add static routes, change MPLS label if doing EPE etc. I see very few, if any, use cases for prefix splitting that could not be accommodated for via other mechanisms that do not leave a network in a “locked and loaded” dangerous state. But it’s a free world and market economy (at least in the NA part of NANOG) so people will use this stuff unfortunately. But take a look at the Twitter link Job supplied earlier where a vendor confirmed they are insecure mode of operation by default. Why??? Being good corporate citizens the model should be secure by default and you can remove the safety if you know what you’re doing (ideally you couldn’t remove the safety at all, but see above comment around free market). Handing people software with the safety removed and assuming they won’t pull the trigger is reckless business conduct IMO. But at the end of the day, it’s all about the almighty dollar and if a customer can be gained where the path is resistance is less giving them an insecure product by default, typically because the customer doesn’t have the technical knowledge to understand what they’re actually doing, then so be it. Sent from my iPhone On Jul 17, 2019, at 2:53 PM, Jared Geiger <jared@compuwizz.net<mailto:jared@compuwizz.net>> wrote: I was attracted to BGP route optimizers for latency/jitter reduction and partial black hole detection scenarios. Our traffic is low enough in volume that we aren't playing the circuit commit game, but rather optimizing the path to VoIP customers who don't care that provider Y in path X-Y-Z had a fiber cut causing issues with their phone service. They are a piece of the SDN and automation fun. Hopefully the vendors will devise ways of dealing with traffic load balancing without splitting prefixes.Or maybe RPKI will become more ubiquitous and leaks won't be as painful. Similar to how DNSSEC led many ISPs to remove their DNS redirecting "search services". On Wed, Jul 17, 2019 at 10:02 AM Michael Still <stillwaxin@gmail.com<mailto:stillwaxin@gmail.com>> wrote: On Wed, Jul 17, 2019 at 12:38 AM Hank Nussbacher <hank@efes.iucc.ac.il<mailto:hank@efes.iucc.ac.il>> wrote:
On 16/07/2019 20:41, Job Snijders wrote:
On Tue, Jul 16, 2019 at 3:33 PM Mike Hammett <nanog@ics-il.net<mailto:nanog@ics-il.net>> wrote:
More like do whatever you want in your own house as long as you don't infringe upon others.
That's where the rub is; when using "BGP optimisers" to influence public Internet routing, you cannot guarantee you won't infringe upon others.
The argument against route optimizers (assuming appropriate ingress\egress filters) is a religious one and should be treated as such.
There is a difference between BGP optimizers and route optimizers. When was the last time you heard a complain about Akamai screwing up the global routing table over the past 12 years:
https://www.akamai.com/us/en/about/news/press/2007-press/akamai-introduces-a...
https://developer.akamai.com/legacy/learn/Optimization/SureRoute.html
-Hank
Along these same lines I'd like to point out that nearly all or possibly even all incidents in recent memory are attributable to a single product whereas there has been at least one other product on the market for something like 15+ years that AFAIK has not had a single incident associated with it (and also does not create more specific prefixes as part of its operation). So is it really that one product is spoiling the market for the rest here or are they all bad? -- [stillwaxin@gmail.com<mailto:stillwaxin@gmail.com> ~]$ cat .signature cat: .signature: No such file or directory [stillwaxin@gmail.com<mailto:stillwaxin@gmail.com> ~]$
On Tue, Jul 16, 2019, 6:29 PM Mike Hammett <nanog@ics-il.net> wrote:
assuming appropriate ingress\egress filters
This assumption itself is a good start for the aforementioned "security considerations" chapter, b/c this is the assumption most of us make but only few routinely check. -- Töma
participants (14)
-
Christopher Morrow
-
Dimeji Fayomi
-
Hank Nussbacher
-
Jared Geiger
-
Job Snijders
-
Mark Tinka
-
Michael Still
-
Michel Py
-
Mike Hammett
-
Nick Hilliard
-
Nikolas Geyer
-
Ryan Hamel
-
Tom Beecher
-
Töma Gavrichenkov