A survey on BGP MRAI timer values in practice
Hi NANOG, This is Shahrooz, a fourth-year CS Ph.D. student at the University of Massachusetts Amherst working under the supervision of prof. Arun Venkataramani. We often read that the Internet (i.e. BGP) has a long convergence delay. But why is it so slow? And can we (researchers) do anything about it? Please help us out to find out by answering our short anonymous survey (<10 minutes). This survey aims at finding the best current practices on the Internet about MRAI/"delay out" timer values. We expect the findings to increase the understanding of the perceived BGP convergence on the Internet, which could then help researchers to design better solutions for BGP long convergence delay. Survey URL: https://forms.gle/VNRpU2MzRU8DX1o57 We expect the questionnaire to be filled out by network operators whose job relates to BGP operations. It has a total of 6 questions and should take less than 10 minutes to answer. A summary of the aggregate results will be published as a part of a scientific article later (hopefully :) this year. Thank you so much in advance, and we look forward to read your responses! We would be also extremely grateful if you could forward this email to any operator you might know who may not read NANOG. Best, Shahrooz
On Mon, 7 Jun 2021 at 19:32, <shahrooz@cs.umass.edu> wrote:
We often read that the Internet (i.e. BGP) has a long convergence delay. But why is it so slow? And can we (researchers) do anything about it?
Create business incentives to improve it. This is a non-technical problem, we've long had technical tools to make it fast, there just isn't incentive to make it fast. Customers are not asking operators for better convergence speeds.
Please help us out to find out by answering our short anonymous survey (<10 minutes).
Can you tell me what have you done so far? What are the default MRAI values for each AFI/SAFI for IOS, IOS-XR, Junos, SROS, VRP and EOS? Then people responding don't have to check what their NOS does, they can refer to your table and tell the default value, since this is what
99% will be using.
Now your survey has built-in selection bias, people who answer it are people who know what it is and who are concerned about it and have changed it, this is not a representative group and you will start your work with very bad data. -- ++ytti
+1 to Saku's concerns - I simply ignored the survey because I wasn't sure what MRAI was, and I wasn't sure what my values would be. But I have time to be interested right now, so a-spelunking I go... The term "MRAI" does not appear anywhere in Arista's or Extreme's documentation. Nor does this timer interval appear in any BGP-related "show" output, on any of my platforms, that I can see. I've found out that "out-delay" (not "delay out") is synonymous with MRAI and seems widely used. I found "out-delay" in an Arista technote, and now I know how to override it on Aristas, and the default is zero (0). Unfortunately, "show" commands on EOS will only show you the current out-delay if it is non-zero, which makes reporting it a bit difficult. Extreme's MLXe platform doesn't appear to support an out-delay/MRAI knob at all, at least as far as I can tell. I know there are several other current and former MLXe operators here, maybe one of them will know? Based on my limited history with NANOG-L, I guess your initial email might have been seen by perhaps 20 people who immediately knew what you were talking about, and 2000 who didn't. (I don't actually know subscriber numbers, that's totally a WAG. And maybe more people have touched out-delay knobs than I think. Dunno.) <snide but hopefully not too much> This illustrates the gap between academia and industry - academics can research a narrow topic, and come at issues from a theoretic standpoint using unfamiliar terminology. As a practitioner, I get told to add carrier X as a peer by end-of-day Friday, and "just make it work". I wasn't even able to understand your initial question, because I haven't spent a semester understanding the intricacies of BGP propagation - I just know it usually works, knobs exist that I shouldn't fiddle with, and that's good enough for my job. </snide> If your work results in actionable recommendations such as "don't use BGP out-delay timers to mitigate XYZ in circumstance LMNO, do ABC instead", that's fantastic. Please keep us advised, and do post aggregated survey results here once you close the survey. I am specifically interested in the answer to "Have you ever had to adjust BGP out-delay with any of your peers, and why?" It would be great if we could derive that answer from the survey results, but anecdotal replies here would also be helpful. All you larger(-than-me) network operators out there: when would I need to use out-delay? Why? What does it accomplish? Good luck in reformulating your survey to get better engagement, -Adam Adam Thompson Consultant, Infrastructure Services [1593169877849] 100 - 135 Innovation Drive Winnipeg, MB, R3T 6A8 (204) 977-6824 or 1-800-430-6404 (MB only) athompson@merlin.mb.ca<mailto:athompson@merlin.mb.ca> www.merlin.mb.ca<http://www.merlin.mb.ca/> ________________________________ From: NANOG <nanog-bounces+athompson=merlin.mb.ca@nanog.org> on behalf of Saku Ytti <saku@ytti.fi> Sent: June 8, 2021 01:06 To: shahrooz@cs.umass.edu <shahrooz@cs.umass.edu> Cc: nanog list <nanog@nanog.org>; Arun Venkataramani <arun@cs.umass.edu> Subject: Re: A survey on BGP MRAI timer values in practice On Mon, 7 Jun 2021 at 19:32, <shahrooz@cs.umass.edu> wrote:
We often read that the Internet (i.e. BGP) has a long convergence delay. But why is it so slow? And can we (researchers) do anything about it?
Create business incentives to improve it. This is a non-technical problem, we've long had technical tools to make it fast, there just isn't incentive to make it fast. Customers are not asking operators for better convergence speeds.
Please help us out to find out by answering our short anonymous survey (<10 minutes).
Can you tell me what have you done so far? What are the default MRAI values for each AFI/SAFI for IOS, IOS-XR, Junos, SROS, VRP and EOS? Then people responding don't have to check what their NOS does, they can refer to your table and tell the default value, since this is what
99% will be using.
Now your survey has built-in selection bias, people who answer it are people who know what it is and who are concerned about it and have changed it, this is not a representative group and you will start your work with very bad data. -- ++ytti
On Wed, 9 Jun 2021 at 01:18, Adam Thompson <athompson@merlin.mb.ca> wrote: If your work results in actionable recommendations such as "don't use BGP
out-delay timers to mitigate XYZ in circumstance LMNO, do ABC instead", that's fantastic. Please keep us advised, and do post aggregated survey results here once you close the survey.
What is actionable? What is the goal? The question as OP presented contains some assumptions a) better convergence is needed b) MRAI is important part of the solution space Neither are provable. We already know how to make DFZ convergence really fast (or at least orders of magnitude faster than it is), that information exists, but that isn't deployed because customers are not asking for it, so providers are not aware that there is room for improvements. Things don't optimise to be as good as they can be, things optimise to be as bad as the market allows them to be. And the market accepts the DFZ convergence. If you do decide to optimise for DFZ convergence, without commercial pressure, you will risk lower availability, because you'll be using configuration less tested by other customers and everyone knows how terrible quality every NOS is. Everyone finds novel bugs, in the same damn protocols we've ran +20 years. It's like running Windows and Linux and regularly finding out listing files in a directory breaks your service, year after year after year. For those who are interested in better convergence - change your interface down reporting to 0 (there may be delay before interface down is reported to system, so that optical protection works without causing outage) - use 'add-path' or at least 'best-external' in iBGP, so that you always have backup eBGP route immediately available once best is invalidated (normally you have lot of delay to find next best, once you lose your best eBGP) - tie your route validity to IGP, so you can invalidate your BGP the moment IGP disappears - ensure IGP converges fast (another topic) - set MRAI to 0 - use PIC edge - ensure your BGP NLRI can be as large as MTU allows - ensure your convergence isn't bottle necked by slow peer in group - ensure you are not dropping received TCP packets on punt path - ensure your fast external fallover works (eBGP down, on int down) this is quite easy to break - then ensure everyone else in the DFZ does the same thing But from a business POV, don't do any of this, you will have more bugs and lower availability and your customers will be less happy.
I *am *specifically interested in the answer to "Have you ever had to adjust BGP out-delay with any of your peers, and why?" It would be great if we could derive that answer from the survey results, but anecdotal replies here would also be helpful. All you larger(-than-me) network operators out there: when would I need to use out-delay? Why? What does it accomplish?
Good luck in reformulating your survey to get better engagement, -Adam
*Adam Thompson* Consultant, Infrastructure Services [image: 1593169877849] 100 - 135 Innovation Drive Winnipeg, MB, R3T 6A8 (204) 977-6824 or 1-800-430-6404 (MB only) athompson@merlin.mb.ca www.merlin.mb.ca
------------------------------ *From:* NANOG <nanog-bounces+athompson=merlin.mb.ca@nanog.org> on behalf of Saku Ytti <saku@ytti.fi> *Sent:* June 8, 2021 01:06 *To:* shahrooz@cs.umass.edu <shahrooz@cs.umass.edu> *Cc:* nanog list <nanog@nanog.org>; Arun Venkataramani <arun@cs.umass.edu> *Subject:* Re: A survey on BGP MRAI timer values in practice
On Mon, 7 Jun 2021 at 19:32, <shahrooz@cs.umass.edu> wrote:
We often read that the Internet (i.e. BGP) has a long convergence delay. But why is it so slow? And can we (researchers) do anything about it?
Create business incentives to improve it. This is a non-technical problem, we've long had technical tools to make it fast, there just isn't incentive to make it fast. Customers are not asking operators for better convergence speeds.
Please help us out to find out by answering our short anonymous survey (<10 minutes).
Can you tell me what have you done so far? What are the default MRAI values for each AFI/SAFI for IOS, IOS-XR, Junos, SROS, VRP and EOS? Then people responding don't have to check what their NOS does, they can refer to your table and tell the default value, since this is what
99% will be using.
Now your survey has built-in selection bias, people who answer it are people who know what it is and who are concerned about it and have changed it, this is not a representative group and you will start your work with very bad data.
-- ++ytti
-- ++ytti
We already know how to make DFZ convergence really fast (or at least orders of magnitude faster than it is), that information exists, but that isn't deployed because customers are not asking for it, so providers are not aware that there is room for improvements.
are we confident that in the global context, not just within an isp, there is negligible, well acceptable, oscillation? randy --- randy@psg.com `gpg --locate-external-keys --auto-key-locate wkd randy@psg.com` signatures are back, thanks to dmarc header butchery
On Wed, 9 Jun 2021 at 20:43, Randy Bush <randy@psg.com> wrote:
are we confident that in the global context, not just within an isp, there is negligible, well acceptable, oscillation?
I don't understand the question, but the way I read the question it may be unanswerable even if I did understand it. As the reader would self-define negligible and well acceptable and answer yes/no based on the definition they used, which might be different to the definition writer intended. -- ++ytti
On 6/10/21 08:26, Saku Ytti wrote:
I don't understand the question, but the way I read the question it may be unanswerable even if I did understand it. As the reader would self-define negligible and well acceptable and answer yes/no based on the definition they used, which might be different to the definition writer intended.
It's possible we've become accustom to a slow, global BGP, due to a perception of fragility (and complexity), which favours stability over speed. I suppose the size of the current BGP and the nature of the FSM it lends itself to does some to account for those perceptions. At a per-ISP level, it is not impossible to speed up (i)BGP convergence. On a global scale, taking the least common denominator to allow for all manner of network we don't know about, allowing the ship a wide turn in BGP waters, at least on a perceptive level, seems like an unsigned social agreement amongst autonomous systems. Ultimately, I feel we aren't talking enough about this, and hopefully, this thread gets us to that point. Mark.
My question at this point is, what slow global convergence? When I (or any of my downstreams) adjusts a prefix, I nearly always see global propagation in well under 60 seconds. Among networks where I can check, at least. I understand it could be technically possible to see near-instantaneous global convergence in more like <5sec, but... on a global scale, neither I nor my customers care about the difference between 5s and 60s. Do other people need <60s propagation? -Adam ________________________________ From: NANOG <nanog-bounces+athompson=merlin.mb.ca@nanog.org> on behalf of Mark Tinka <mark@tinka.africa> Sent: June 10, 2021 02:36 To: nanog@nanog.org <nanog@nanog.org> Subject: Re: A survey on BGP MRAI timer values in practice On 6/10/21 08:26, Saku Ytti wrote:
I don't understand the question, but the way I read the question it may be unanswerable even if I did understand it. As the reader would self-define negligible and well acceptable and answer yes/no based on the definition they used, which might be different to the definition writer intended.
It's possible we've become accustom to a slow, global BGP, due to a perception of fragility (and complexity), which favours stability over speed. I suppose the size of the current BGP and the nature of the FSM it lends itself to does some to account for those perceptions. At a per-ISP level, it is not impossible to speed up (i)BGP convergence. On a global scale, taking the least common denominator to allow for all manner of network we don't know about, allowing the ship a wide turn in BGP waters, at least on a perceptive level, seems like an unsigned social agreement amongst autonomous systems. Ultimately, I feel we aren't talking enough about this, and hopefully, this thread gets us to that point. Mark.
participants (5)
-
Adam Thompson -
Mark Tinka -
Randy Bush -
Saku Ytti -
shahrooz＠cs.umass.edu