Does the following ever happen in reality? Do you think it is strange and unlikely? The lateral (i.e., non-transit) peer of an AS is also the transit provider of the AS's transit provider. Example: AS A has AS B as a transit provider and AS C as a lateral peer, and AS C is a transit provider of AS B. Thank you. Sriram
It would likely be common with Hurricane Electric as a peer. They peer freely and their good value means they're in many BGP mixes. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Kotikalapudi Sriram (Fed) via NANOG" <nanog@lists.nanog.org> To: nanog@nanog.org Cc: "Kotikalapudi Sriram (Fed)" <kotikalapudi.sriram@nist.gov> Sent: Monday, April 7, 2025 8:14:55 AM Subject: [NANOG] question about peering relationships Does the following ever happen in reality? Do you think it is strange and unlikely? The lateral (i.e., non-transit) peer of an AS is also the transit provider of the AS's transit provider. Example: AS A has AS B as a transit provider and AS C as a lateral peer, and AS C is a transit provider of AS B. Thank you. Sriram _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/NSV3GXEZ...
On Mon, Apr 07, 2025 at 08:17:13AM -0500, Mike Hammett via NANOG wrote:
It would likely be common with Hurricane Electric as a peer. They peer freely and their good value means they're in many BGP mixes.
This also creates various issues as well as they lack a number of BGP communities that are fairly standard across networks, so the resulting behaviors can be rather inconsistent, or you end up with traffic being long-hauled, even possibly off-continent due to scope. I regularly see this behavior for prefixes behind HE, so be aware you may actually not be seeing what you might naturally expect unless you actually are looking closely. If you just have 1-2 providers it may not be noticable, but the larger the mix you deal with the more unexpected behaviors you can quickly expose. - Jared
Yes, this is quite common - as an example, AS64505 can have AS64500 as a transit provider and peer with AS64510, and AS64510 can be a transit provider of AS64500, if that’s what you mean. It's rather normal for this to happen, nothing out of the ordinary. AS64510 would only export their originating routes to an IX while sending their full table to transit customers. You could (or should) preference peering higher than transit to save on transit costs etc. but I digress... Regards, Christopher Hawker Get Outlook for iOS<https://aka.ms/o0ukef> ________________________________ From: Sriram, Kotikalapudi (Fed) via NANOG <nanog@lists.nanog.org> Sent: Monday, April 7, 2025 11:15 pm To: nanog@nanog.org <nanog@nanog.org> Cc: Sriram, Kotikalapudi (Fed) <kotikalapudi.sriram@nist.gov> Subject: [NANOG] question about peering relationships Does the following ever happen in reality? Do you think it is strange and unlikely? The lateral (i.e., non-transit) peer of an AS is also the transit provider of the AS's transit provider. Example: AS A has AS B as a transit provider and AS C as a lateral peer, and AS C is a transit provider of AS B. Thank you. Sriram _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/NSV3GXEZ...
nanog@lists.nanog.org (Sriram, Kotikalapudi (Fed) via NANOG) wrote:
Does the following ever happen in reality? Do you think it is strange and unlikely? The lateral (i.e., non-transit) peer of an AS is also the transit provider of the AS's transit provider. Example: AS A has AS B as a transit provider and AS C as a lateral peer, and AS C is a transit provider of AS B.
Yes, and it's very often a mess traffic-engineering wise...
On Mon, Apr 7, 2025 at 8:10 AM Elmar K. Bins via NANOG <nanog@lists.nanog.org> wrote:
nanog@lists.nanog.org (Sriram, Kotikalapudi (Fed) via NANOG) wrote:
Does the following ever happen in reality? Do you think it is strange and unlikely? The lateral (i.e., non-transit) peer of an AS is also the transit provider of the AS's transit provider. Example: AS A has AS B as a transit provider and AS C as a lateral peer, and AS C is a transit provider of AS B.
Yes, and it's very often a mess traffic-engineering wise...
Hi Elmar, Would you mind discussing this further and offering examples of some of the traffic engineering challenges? AS A should see AS C's origin and customer routes from both AS C directly (peer) and AS B (provider). Unless AS A is playing local pref games, the peering routes have shorter AS paths and are preferred -- a sensible outcome. AS C should see AS A's routes both from AS A directly (peer) and AS B (customer). Unless AS C is playing local pref games, the peering routes have shorter AS paths and are preferred -- a sensible outcome. AS B should see AS A's routes both from AS A directly (customer) and AS B (provider). Unless AS B is playing local pref games, the customer routes have shorter AS paths and are preferred -- a sensible outcome. AS C does not see AS B's routes via its lateral peering link with AS A because on a peering link the AS only sends its origin and customer routes. Thus AS C sends AS B's packets to its customer, AS B -- a sensible outcome. Now, if one of the networks is playing local pref games, which they shouldn't be doing, then they may misroute packets the long way around the planet. But that's their fault for playing local pref games. Regards, Bill Herrin -- William Herrin bill@herrin.us https://bill.herrin.us/
On Wed, Apr 9, 2025 at 8:31 AM William Herrin via NANOG < nanog@lists.nanog.org> wrote:
nanog@lists.nanog.org (Sriram, Kotikalapudi (Fed) via NANOG) wrote:
Does the following ever happen in reality? Do you think it is strange and unlikely? The lateral (i.e., non-transit) peer of an AS is also the transit
On Mon, Apr 7, 2025 at 8:10 AM Elmar K. Bins via NANOG <nanog@lists.nanog.org> wrote: provider of the AS's transit provider. Example: AS A has AS B as a transit provider and AS C as a lateral peer, and AS C is a transit provider of AS B.
Yes, and it's very often a mess traffic-engineering wise...
Hi Elmar,
Would you mind discussing this further and offering examples of some of the traffic engineering challenges?
[...description of default BGP behaviour elided...]
Now, if one of the networks is playing local pref games, which they shouldn't be doing, then they may misroute packets the long way around the planet. But that's their fault for playing local pref games.
Bill--can you clarify why you feel setting localpref values for peers differently from customers is something ISPs "shouldn't be doing?" If I'm an ISP (call me ASN Ishmael....er, ASN A), and I have a customer, ASN B, who has a customer, ASN C. I also have a peer, D, that provides service to their customer, ASN C. I hear ASN C's prefixes with equal AS-PATH length from ASN D and ASN B; default BGP behaviour looks for tie breakers further down the decision pathway. However, if I send the traffic through ASN D, as a peer, I make no revenue from carrying that traffic. If I send the traffic via ASN B, however, I earn revenue for passing the traffic along the customer, assume 95th percentile billed, link. Is it your assertion that as an ISP, as a provider of services to my customers, I should leave it up to tie-breaking heuristics further and further down the BGP decision tree as to whether I can earn money by carrying traffic or not? Or do you perhaps see some wisdom in preferring to send traffic through my customer's links when possible, so that revenue can be earned to keep the company afloat, rather than passing it for free through a (probably competitor's) peering link, who then hands it to the destination? If peer ASN D is also a competitor of mine, if I let BGP toss the coin for me, there's a good chance in doing so I've taken money out of my bank account, and handed it to my competitor. To paraphrase Randy Bush, "I highly encourage my competition to follow your advice." ;)
Regards, Bill Herrin
Thanks! Matt PS--yes, I know the alternative is to put filter lists on your peering sessions that actively drop any ASNs that happen to be within your downstream customer cone, rather than messing with localprefs. However, that increases the fragility of your connectivity mesh, because if ASN C's link to ASN B drops, you're now black-holing them; they can reach the rest of the internet via ASN D, but they can't reach you, because you're filtering out their announcements at your peering edge. Whups. Maybe localprefs aren't such an evil thing after all... ^_^;
On Wed, Apr 9, 2025 at 10:17 AM Matthew Petach <mpetach@netflight.com> wrote:
On Wed, Apr 9, 2025 at 8:31 AM William Herrin via NANOG <nanog@lists.nanog.org> wrote:
On Mon, Apr 7, 2025 at 8:10 AM Elmar K. Bins via NANOG <nanog@lists.nanog.org> wrote:
Yes, and it's very often a mess traffic-engineering wise...
Now, if one of the networks is playing local pref games, which they shouldn't be doing, then they may misroute packets the long way around the planet. But that's their fault for playing local pref games.
Bill--can you clarify why you feel setting localpref values for peers differently from customers is something ISPs "shouldn't be doing?"
Hi Matt, Because, as Elmar alluded to, it makes a mess traffic engineering wise. Like the one where I ended up having to announce both a covering and disaggregates to overcome a provider of a provider localprefing my routes on a grand tour of the continental United States when they had a peeing route to me five miles down the road. https://mailman.nanog.org/pipermail/nanog/2024-January/224628.html
Is it your assertion that as an ISP, as a provider of services to my customers, I should leave it up to tie-breaking heuristics further and further down the BGP decision tree as to whether I can earn money by carrying traffic or not?
Not all hamburger joints are in the business of selling quality beef. Is yours? Regards, Bill Herrin -- William Herrin bill@herrin.us https://bill.herrin.us/
Because, as Elmar alluded to, it makes a mess traffic engineering wise. Like the one where I ended up having to announce both a covering and disaggregates to overcome a provider of a provider localprefing my routes on a grand tour of the continental United States when they had a peeing route to me five miles down the road. https://mailman.nanog.org/pipermail/nanog/2024-January/224628.html
Ah yes, the 'everyone should do something different because I am inconvenienced' argument returns. On Wed, Apr 9, 2025 at 3:04 PM William Herrin via NANOG < nanog@lists.nanog.org> wrote:
On Wed, Apr 9, 2025 at 10:17 AM Matthew Petach <mpetach@netflight.com> wrote:
On Wed, Apr 9, 2025 at 8:31 AM William Herrin via NANOG < nanog@lists.nanog.org> wrote:
On Mon, Apr 7, 2025 at 8:10 AM Elmar K. Bins via NANOG <nanog@lists.nanog.org> wrote:
Yes, and it's very often a mess traffic-engineering wise...
Now, if one of the networks is playing local pref games, which they shouldn't be doing, then they may misroute packets the long way around the planet. But that's their fault for playing local pref games.
Bill--can you clarify why you feel setting localpref values for peers differently from customers is something ISPs "shouldn't be doing?"
Hi Matt,
Because, as Elmar alluded to, it makes a mess traffic engineering wise. Like the one where I ended up having to announce both a covering and disaggregates to overcome a provider of a provider localprefing my routes on a grand tour of the continental United States when they had a peeing route to me five miles down the road. https://mailman.nanog.org/pipermail/nanog/2024-January/224628.html
Is it your assertion that as an ISP, as a provider of services to my customers, I should leave it up to tie-breaking heuristics further and further down the BGP decision tree as to whether I can earn money by carrying traffic or not?
Not all hamburger joints are in the business of selling quality beef. Is yours?
Regards, Bill Herrin
-- William Herrin bill@herrin.us https://bill.herrin.us/ _______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/MKGB2PKD...
On Wed, Apr 9, 2025 at 12:03 PM William Herrin <bill@herrin.us> wrote:
On Wed, Apr 9, 2025 at 10:17 AM Matthew Petach <mpetach@netflight.com> wrote:
On Wed, Apr 9, 2025 at 8:31 AM William Herrin via NANOG < nanog@lists.nanog.org> wrote:
On Mon, Apr 7, 2025 at 8:10 AM Elmar K. Bins via NANOG <nanog@lists.nanog.org> wrote:
Yes, and it's very often a mess traffic-engineering wise...
Now, if one of the networks is playing local pref games, which they shouldn't be doing, then they may misroute packets the long way around the planet. But that's their fault for playing local pref games.
Bill--can you clarify why you feel setting localpref values for peers differently from customers is something ISPs "shouldn't be doing?"
Hi Matt,
Because, as Elmar alluded to, it makes a mess traffic engineering wise. Like the one where I ended up having to announce both a covering and disaggregates to overcome a provider of a provider localprefing my routes on a grand tour of the continental United States when they had a peeing route to me five miles down the road. https://mailman.nanog.org/pipermail/nanog/2024-January/224628.html
Is it your assertion that as an ISP, as a provider of services to my customers, I should leave it up to tie-breaking heuristics further and further down the BGP decision tree as to whether I can earn money by carrying traffic or not?
Not all hamburger joints are in the business of selling quality beef. Is yours?
I might posit that all hamburger joints should be in the business of deterministically selling hamburgers, at the very least. If I sell connectivity to a customer, the customer is likely to want some level of assurance that their traffic will indeed deterministically pass across that link, modulo any overriding traffic engineering they apply. I would be an unhappy customer if I discovered that my network provider believed that Heisenberg and Schrodinger were the patron saints of packet flows[0]; indeed, it's likely I would quickly find myself a new provider, where a bit more certainty about traffic patterns could be expected. Reiterating my question again, and this time hoping for a clearer answer than a muddy analogy to the quality of beef at a hamburger joint: Is it your assertion that ISPs should leave routing decisions purely to the default BGP path selection algorithm, with no hints, nudges, or fingers on the scale to steer traffic flows? Is "oldest path" really what we should let be the deciding factor in where traffic goes, all else being equal between paths learned from downstream customers and paths learned from peers? Meanwhile, all this talk of hamburger has made me hungry. I'm going to go make some lunch while I await your explanation of how you think ISPs should provide some determinism in their traffic flows. :) Thanks! Matt [0] it's highly uncertain where the traffic may go, and we won't know where it went until we go to look at it.
On Wed, Apr 9, 2025 at 2:01 PM Matthew Petach <mpetach@netflight.com> wrote:
If I sell connectivity to a customer, the customer is likely to want some level of assurance that their traffic will indeed deterministically pass across that link,
Well that would be your first mistake. Your BGP customer wants their packets to go where *they* tell them to go, not where you feel like sending them. As do their downstream BGP customers who don't have the luxury of calling you up and threatening to withhold payment.
I would be an unhappy customer if I discovered that my network provider believed that Heisenberg and Schrodinger were the patron saints of packet flows[0];
I don't even know what you're on about here. No aspect of the BGP protocol is remotely non-deterministic. Even when you use it badly.
Is it your assertion that ISPs should leave routing decisions purely to the default BGP path selection algorithm, with no hints, nudges, or fingers on the scale to steer traffic flows?
Absolutely not. My position is that like fabled "goto," use of local pref should be considered harmful. That doesn't exclude the use of BGP's inbuilt tools like meds and prepends, nor does it exclude the use of optimizers capable of incorporating smart additional information into the selection algorithm when multiple paths are available. Local pref is not smart. It's a blunt instrument that hurts.. Anyway, the chief issue with Sriram's scenario comes when you add AS D, the transit provider for AS C: D <- C <- B ^-> A-^ If C accepts the peering route from A instead of the A customer route from B then C does not propagate A's route to D. Customer propagates to transit. Peer does not. A has to make a choice between having C as a peer and having C as an indirect transit provider. He can't have both. But unless C plays games with localprefs, A can trivially express his preference using prepends. And choices like this are what A signed on for when he decided to peer with C instead of buying transit. Regards, Bill Herrin Regards, Bill Herrin -- William Herrin bill@herrin.us https://bill.herrin.us/
I’ve been in this exact predicament, where I’ve been A, receiving transit from B, and peering on an IX where C also appears (and who also provides transit to B). C have added a no-export to A community onto their routes advertised to the IX route servers to force traffic via B. It’s a commercial decision to have their customers buy more transit, plain and simple. While we may not like it because it means we must pay more for transit, we can either accept it or find another transit provider. Regards, Christopher Hawker Get Outlook for iOS<https://aka.ms/o0ukef> ________________________________ From: William Herrin via NANOG <nanog@lists.nanog.org> Sent: Thursday, April 10, 2025 8:15:55 AM To: Matthew Petach <mpetach@netflight.com> Cc: nanog@nanog.org <nanog@nanog.org>; William Herrin <bill@herrin.us> Subject: [NANOG] Re: question about peering relationships On Wed, Apr 9, 2025 at 2:01 PM Matthew Petach <mpetach@netflight.com> wrote:
If I sell connectivity to a customer, the customer is likely to want some level of assurance that their traffic will indeed deterministically pass across that link,
Well that would be your first mistake. Your BGP customer wants their packets to go where *they* tell them to go, not where you feel like sending them. As do their downstream BGP customers who don't have the luxury of calling you up and threatening to withhold payment.
I would be an unhappy customer if I discovered that my network provider believed that Heisenberg and Schrodinger were the patron saints of packet flows[0];
I don't even know what you're on about here. No aspect of the BGP protocol is remotely non-deterministic. Even when you use it badly.
Is it your assertion that ISPs should leave routing decisions purely to the default BGP path selection algorithm, with no hints, nudges, or fingers on the scale to steer traffic flows?
Absolutely not. My position is that like fabled "goto," use of local pref should be considered harmful. That doesn't exclude the use of BGP's inbuilt tools like meds and prepends, nor does it exclude the use of optimizers capable of incorporating smart additional information into the selection algorithm when multiple paths are available. Local pref is not smart. It's a blunt instrument that hurts.. Anyway, the chief issue with Sriram's scenario comes when you add AS D, the transit provider for AS C: D <- C <- B ^-> A-^ If C accepts the peering route from A instead of the A customer route from B then C does not propagate A's route to D. Customer propagates to transit. Peer does not. A has to make a choice between having C as a peer and having C as an indirect transit provider. He can't have both. But unless C plays games with localprefs, A can trivially express his preference using prepends. And choices like this are what A signed on for when he decided to peer with C instead of buying transit. Regards, Bill Herrin Regards, Bill Herrin -- William Herrin bill@herrin.us https://bill.herrin.us/ _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/VOD5FB7P...
On Wed, Apr 9, 2025 at 2:01 PM Matthew Petach <mpetach@netflight.com> wrote:
I don't even know what you're on about here. No aspect of the BGP protocol is remotely non-deterministic. Even when you use it badly.
Wondering if someone has a previous NANOG presentation or other white paper that covers multiple scenarios they can reference?
Is it your assertion that ISPs should leave routing decisions purely to the default BGP path selection algorithm, with no hints, nudges, or fingers on the scale to steer traffic flows?
Absolutely not. My position is that like fabled "goto," use of local pref should be considered harmful. That doesn't exclude the use of BGP's inbuilt tools like meds and prepends, nor does it exclude the use of optimizers capable of incorporating smart additional information into the selection algorithm when multiple paths are available. Local pref is not smart. It's a blunt instrument that hurts..
The routing table we are trying to manipulate is a rather blunt instrument. Packets are routed on the destination prefix only. BGP also does not tons of choices either especially when it has to use its limited tooling to affect the even more limited FIB. These are the tools we have. D <- C <- B ^-> A-^
If C accepts the peering route from A instead of the A customer route from B then C does not propagate A's route to D. Customer propagates to transit. Peer does not. A has to make a choice between having C as a peer and having C as an indirect transit provider. He can't have both. But unless C plays games with localprefs, A can trivially express his preference using prepends. And choices like this are what A signed on for when he decided to peer with C instead of buying transit.
Can you provide a white paper that covers your proposed 'BGP Improvements' in more detail? This could be a decent NANOG presentation too. I have always been under the impression that you can't have peering, transit, and customer relationships with the same upstream and expect things to work, with or without local preference. I know I am missing something in these conversations, and have not been able to glean a better understanding from this thread. Hoping someone can point me to a white paper or good NANOG presentation.
Can you provide a white paper that covers your proposed 'BGP Improvements' in more detail? This could be a decent NANOG presentation too. I have always been under the impression that you can't have peering, transit, and customer relationships with the same upstream and expect things to work, with or without local preference. I know I am missing something in these conversations, and have not been able to glean a better understanding from this thread. Hoping someone can point me to a white paper or good NANOG presentation.
Bill isn't proposing technical changes to BGP at all. He wants transit providers to ignore that they are a business , and also wants to ignore the mechanisms provided by transit providers to handle traffic engineering considerations that do arise. Basically he wants a pony to solve problems that the rest of us have been solving for decades. On Thu, Apr 10, 2025 at 8:33 AM Kevin Burke via NANOG <nanog@lists.nanog.org> wrote:
I don't even know what you're on about here. No aspect of the BGP
On Wed, Apr 9, 2025 at 2:01 PM Matthew Petach <mpetach@netflight.com> wrote: protocol is remotely non-deterministic. Even when you use it badly.
Wondering if someone has a previous NANOG presentation or other white paper that covers multiple scenarios they can reference?
Is it your assertion that ISPs should leave routing decisions purely to the default BGP path selection algorithm, with no hints, nudges, or fingers on the scale to steer traffic flows?
Absolutely not. My position is that like fabled "goto," use of local pref should be considered harmful. That doesn't exclude the use of BGP's inbuilt tools like meds and prepends, nor does it exclude the use of optimizers capable of incorporating smart additional information into the selection algorithm when multiple paths are available. Local pref is not smart. It's a blunt instrument that hurts..
The routing table we are trying to manipulate is a rather blunt instrument. Packets are routed on the destination prefix only. BGP also does not tons of choices either especially when it has to use its limited tooling to affect the even more limited FIB. These are the tools we have.
D <- C <- B ^-> A-^
If C accepts the peering route from A instead of the A customer route from B then C does not propagate A's route to D. Customer propagates to transit. Peer does not. A has to make a choice between having C as a peer and having C as an indirect transit provider. He can't have both. But unless C plays games with localprefs, A can trivially express his preference using prepends. And choices like this are what A signed on for when he decided to peer with C instead of buying transit.
Can you provide a white paper that covers your proposed 'BGP Improvements' in more detail? This could be a decent NANOG presentation too. I have always been under the impression that you can't have peering, transit, and customer relationships with the same upstream and expect things to work, with or without local preference. I know I am missing something in these conversations, and have not been able to glean a better understanding from this thread. Hoping someone can point me to a white paper or good NANOG presentation. _______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/5YVBVFKV...
Yep. Pretty common, especially with regional providers who are both at a peering exchange and who also buy transit. Is particularly frustrating when AS C has configured their network to prefer customer routes instead of peers which is also really common. If I'm AS A and AS C is configured this way, my traffic will always go through AS B even though I'm peered with AS C directly. On Mon, Apr 7, 2025, 7:15 AM Sriram, Kotikalapudi (Fed) via NANOG < nanog@lists.nanog.org> wrote:
Does the following ever happen in reality? Do you think it is strange and unlikely?
The lateral (i.e., non-transit) peer of an AS is also the transit provider of the AS's transit provider. Example: AS A has AS B as a transit provider and AS C as a lateral peer, and AS C is a transit provider of AS B.
Thank you.
Sriram
_______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/NSV3GXEZ...
participants (10)
-
Christopher Hawker -
Elmar K. Bins -
Forrest Christian (List Account) -
Jared Mauch -
Kevin Burke -
Matthew Petach -
Mike Hammett -
Sriram, Kotikalapudi (Fed) -
Tom Beecher -
William Herrin