I wish they'd add one more that turns off their "prefer routes learned from a customer" rule. I'm having to split my blocks in half and announce them that way to get them to send my traffic directly to me through our IX peering session as opposed to one of my transit providers. I'd rather they just let shortest path selection work. On Sun, Jul 24, 2022, 1:43 PM Siyuan Miao <aveline@misaka.io> wrote:

They do have BGP communities ... but for black-hole only :-(

On Sun, Jul 24, 2022 at 9:39 PM Ryan Hamel <administrator@rkhtech.org> wrote:

...

Yes.

Ryan

-----Original Message----- From: NANOG <nanog-bounces+ryan=rkhtech.org@nanog.org> On Behalf Of Rubens Kuhl Sent: Sunday, July 24, 2022 12:36 PM To: Nanog <nanog@nanog.org> Subject: HE.net and BGP Communities

The last mention I found on NANOG about HE.net and BGP communities for traffic engineering is from April 2021 and said they provided none.

Is that still the case a year later ?

Rubens

I do understand the reasoning behind preferring customer routes. However in the case where a customer of a customer also connects to you directly via peering doesn't it make sense to prefer the direct connection? or at least not prefer the customer learned routes. In our situation, we were buying transit, heavily prepended, from a provider on a tiny circuit. The purpose of the transit was related to another service we were acquiring from that provider and wasn't about the transit, but the transit was needed for the service to work reliably. Unfortunately this provider was also a HE customer and so we now had all of the HE traffic coming down this tiny link, since all of our other transit providers and ourselves only peered with HE. I don't remember why, but we couldn't have the transit provider not announce our routes toward HE, so we ended up doing the announce more specifics everywhere else thing. Which I hate doing on so many levels. Thus the desire for a community to tell HE that although they learned this route from a customer, it is not a customer route. On Mon, Jul 25, 2022, 5:21 AM Jared Mauch <jared@puck.nether.net> wrote:

You always want to prefer customer routes over non customer routes as a service provider. Of course having a robust set of communities to let adjustments happen helps.

Without proper tiering of routes you may see unstable routing.

Having a standard set of customer, peer, transit set of local preferences would go a long way. Same for geographic scope of routes, only use these on same continent. Makes using a provider if you do something like anycast hard if they haul you long distance.

- Jared

Sent via RFC1925 compliant device

On Jul 25, 2022, at 6:49 AM, Forrest Christian (List Account) < lists@packetflux.com> wrote:

 I wish they'd add one more that turns off their "prefer routes learned from a customer" rule. I'm having to split my blocks in half and announce them that way to get them to send my traffic directly to me through our IX peering session as opposed to one of my transit providers.

I'd rather they just let shortest path selection work.

On Sun, Jul 24, 2022, 1:43 PM Siyuan Miao <aveline@misaka.io> wrote:

...

They do have BGP communities ... but for black-hole only :-(

On Sun, Jul 24, 2022 at 9:39 PM Ryan Hamel <administrator@rkhtech.org> wrote:

...

Yes.

Ryan

-----Original Message----- From: NANOG <nanog-bounces+ryan=rkhtech.org@nanog.org> On Behalf Of Rubens Kuhl Sent: Sunday, July 24, 2022 12:36 PM To: Nanog <nanog@nanog.org> Subject: HE.net and BGP Communities

The last mention I found on NANOG about HE.net and BGP communities for traffic engineering is from April 2021 and said they provided none.

Is that still the case a year later ?

Rubens

On Mon, 25 Jul 2022 at 17:19, Brian Turnbow via NANOG <nanog@nanog.org> wrote:

Hi,

...

I do understand the reasoning behind preferring customer routes. However in the case where a customer of a customer also connects to you directly via peering doesn't it make sense to prefer the direct connection? or at >least not prefer the customer learned routes.

Business not technical. Fill up the bandwidth and they will buy more!!

...

In our situation, we were buying transit, heavily prepended, from a provider on a tiny circuit. The purpose of the transit was related to another service we were acquiring from that provider and wasn't about the transit, but >the transit was needed for the service to work reliably. Unfortunately this provider was also a HE customer and so we now had all of the HE traffic coming down this tiny link, since all of our other transit providers and >ourselves only peered with HE.

...

I don't remember why, but we couldn't have the transit provider not announce our routes toward HE, so we ended up doing the announce more specifics everywhere else thing. Which I hate doing on so many levels.

...

Thus the desire for a community to tell HE that although they learned this route from a customer, it is not a customer route.

You can use communities to set the local preference with HE. This should do the trick 6730:0008 set local pref to 64 (lowest they have afaik)

Google has let you down, 6730 (sunrise) is not 6939 (HE) ;) Afaik HE does not provide TE communities. Lukas

On Mon, 25 Jul 2022 at 16:23, Forrest Christian (List Account) <lists@packetflux.com> wrote:

I do understand the reasoning behind preferring customer routes. However in the case where a customer of a customer also connects to you directly via peering doesn't it make sense to prefer the direct connection? or at least not prefer the customer learned routes.

No, I don't think it makes sense at all. Not preferring customer routes over peering routes would mean: - less revenue for HE - less revenue for the HE customer (your transit provider) - risk of upsetting the customer for the previous reason (less revenue) - for your transit provider it would mean they would provide a transit service with less redundancy (see below), less ingress-paths (see below), and more complicated configuration (and therefore more complex troubleshooting) The only difference is that HE doesn't provide BGP communities for your transit provider for TE. *But even if HE would provide THE communities*, it's not in the best interest of your transit provider to actually pass those BGP communities to HE or configure them for you, because it would mean less traffic on revenue generating links, so it would not make sense from a business perspective. It would also likely require manual configuration of policies, which can be difficult, undesirable or not possible at all, depending on the amount of automation in that network.

I don't remember why, but we couldn't have the transit provider not announce our routes toward HE

And why would they send for your routes specifically or pass through BGP communities to HE, when they are already refusing to withdraw announcements to HE? Businesses don't usually go the extra mile to make everything worse for them (revenue, configuration and support complexity, redundancy, performance due to amount of ingress paths). HE is special in this regards for 2 reasons: - HE doesn't provide TE communities for customers (which is bad and a showstopper for some) - but more importantly, HE has an open peering policy so they actual peer with you in the first place; otherwise we wouldn't have this discussion There are a few technical aspects that to consider: Once HE's best path is a peer route, as opposed to a customer route (which is what you are asking), then those routes would not be announced by HE to its peers anymore (after all, they don't exchange peer routes with peers). So you'd be worse off in some aspects, because your ingress traffic wouldn't be attracted by HE's (vast) peering network anymore. Also if you are single-homed to this transit-provider, and this transit-provider also temporarily single-homes to HE (maybe for maintenance one 1 of 2 transits), you'd be disconnected from pretty much the rest of the internet (that doesn't have HE transit), until you shutdown your HE peering. So less ingress paths, less redundancy, etc. There are good business and technical reasons for preferring customer routes. Many networks don't even peer with customers of customers, also for good reasons. I don't think it makes any sense to consider a shorter as-path over peering when it also comes from a customer (with a longer as-path). - lukas

I do understand the reasoning behind preferring customer routes. However in the case where a customer of a customer also connects to you directly via peering doesn't it make sense to prefer the direct connection? or at least not prefer the customer learned routes.

So from my experience of working at transit providers over more years than I care to contemplate I can assure you what may seem to make sense as a customer does not necessarily translate to how IP routing works. IP has no concept of Customer or Peer it is simply designed to hand the packet to a valid next hop as determined by policies. As such routes are normally divided into customer, peer or further upstream transit if you are not one of the tier 1 providers. A peer provides you no income, a customer (a customer of a customer is largely the same thing as being a direct customer). Take the example of the customer buying a transit service on a 95th percentile basis. So as a transit provider I get paid based on how much traffic I hand to that port(s) and in turn I provide connectivity to all my peers, customers and upstream transits all over the world. I can't do this for free as then I can't pay for my network and I am a company not a charity. Customer X advertises his lets say a /22 to me, all good. But then customer X advertises his /22 and some disaggregated /24's to a local peering exchange that I am also connected to. If I do not both prioritise customer X's customer port routes AND drop any more specific routes learnt from customer X then I will end up handing all customer X's outgoing traffic to them over the peering instead of the revenue generating port. I have seen customers do this both through innocent and malicious intent. Sure, there are a lot of complex policies that I might apply to accept local traffic in one area and hand other traffic via the transit port but why on earth would I do that and likely cause all sorts of other potential routing issues while reducing the revenue I am entitled to?

I wish they'd add one more that turns off their "prefer routes learned from a customer" rule. I'm having to split my blocks in >half and announce them that way to get them to send my traffic directly to me through our IX peering session as opposed to >one of my transit providers. I'd rather they just let shortest path selection work.

I think this is by design so you don't end up with free inbound transit. If one of their transit customers is trying to reach your prefixes, my guess it'd make sense to offload that over IX first, although I'm not sure if that's always happens due to path selection.