holow29,

Usually decisions made about limiting route propagation outbound for 
certain prefixes tend to happen along financial and operational 
constraints.

Remember, outbound route announcements control inbound traffic 
volumes.  So, if you've got some full pipes across the ocean, for example, 
one way to limit the amount of traffic trying to go across 
them is to stop announcing some high-traffic prefixes across them.

Note that it is almost always the route *announcer* that is subtracting 
out routes to control traffic volumes; very seldom does the receiver 
completely subtract out prefixes heard.  The more usual case is for 
the receiver to de-preference the routes down, but to keep them in 
their tables as a fallback, because they can control that behaviour.

From the route-exporting network's perspective, however, they have 
no reason to believe that the receiving network would listen to and 
obey MEDs that they send, to shift traffic away from the congested 
path.  They might have tried adding some prepends along the path, 
only to discover the receiving network was using LOCALPREF knob
adjustments to control traffic, which kinda shits all over attempts 
to traffic engineer via prepending, at which point the only knob you 
have left to turn in order to move traffic off the pathway is to stop 
announcing prefixes entirely.

As a not-so-parenthetical aside, this is why I *strongly* encourage 
networks that use LOCALPREF as a control knob to limit the depth 
to which LOCALPREF adjustments are made; ie, allow overriding 
of the default LOCALPREF only to paths that are $AVG_PATH_LENGTH+2 
or shorter.  That way, if a peer wants to signal a pathway is congested 
and should not be used under normal circumstances, they can still do 
that by triple-prepending their ASN.  That way, peers can continue to 
announce prefixes, but still have some control over traffic flows via 
as-path prepending, versus being backed into the corner of having to 
completely stop announcing prefixes entirely across certain peering 
sessions.

It is very likely that in this case, CT is not announcing prefixes to AS701 
in order to control inbound traffic volumes across certain connections, 
because CT has determined that is the only control knob they have 
available that works; they have probably already experimented and 
found that AS701 squashes inbound MEDs, and uses LOCALPREF 
to force traffic flows regardless of as-path prepending. 

Your only recourse in a situation like this is likely to be getting 
a second connection that bypasses the CT-VZ choke point.

Best of luck!

Matt



On Thu, Jul 21, 2022 at 8:56 AM holow29 <holow29@gmail.com> wrote:
In this instance, I would define "correct" behavior as VZ having any route to this subnet; after all, customers don't pay VZ to access just their network or a subset of the internet. You make a good point, though I would expect that if it isn't VZ's business decision to not have this route, they would have some sort of vested interest in figuring out why CT is not advertising it to them.
From VZ engineer, I heard that no one is advertising it to AS701, so I assume it is not a case of VZ refusing to accept it (which is what I had initially assumed having been frustrated in the past with VZ on other matters + seeing all their BGP issues in the past few years).

On Thu, Jul 21, 2022 at 11:40 AM Tom Beecher <beecher@beecher.cc> wrote:
I would expect Verizon to be able to contact CT and figure out why they aren't passing the correct routes just as I might expect Baidu to do the same thing, I suppose. 

What defines 'correct'? ASN's routinely make traffic engineering or business decisions not to announce certain prefixes to certain other ASNs. This certainly does sometimes cause reachability issues for end users, but that's a choice someone made. That's part of why it's generally good practice to get more than 1 upstream if you can; it gives you more control to mitigate the impact of these choices. 

It is completely possible that Baidu or CT are intentionally not announcing prefixes to VZ. It is also completely possible that they are and VZ is not accepting it. 



On Thu, Jul 21, 2022 at 11:24 AM holow29 <holow29@gmail.com> wrote:
I would expect Verizon to be able to contact CT and figure out why they aren't passing the correct routes just as I might expect Baidu to do the same thing, I suppose. Ultimately, whose responsibility is it other than CT? That is my question. Maybe in this instance, it is common in the industry for this to be the responsibility of Baidu. That seems counterintuitive to me as it is Verizon without the proper route ultimately, but I am not an expert.
Certainly, I think it is incorrect to ask the customer to try to resolve these issues after bringing it to the attention of these services. If Verizon couldn't reach one of Google's edge servers, would it be Verizon or Google's responsibility to fix that if the issue were an intermediate peer failing to broadcase the correct route? I have the sneaking suspicion that Verizon might actually take some action in that instance...

On Thu, Jul 21, 2022 at 9:31 AM Tom Beecher <beecher@beecher.cc> wrote:
Unfortunately it seems like policy that Verizon pushes any issues that aren't internal routing issues to an external party, but surely they have a responsibility to maintain their peering and routes to external services as well.

Baidu is behind CT. If CT is not passing on routes to 701 , what exactly do you expect Verizon to do about it? 


On Wed, Jul 20, 2022 at 6:40 PM holow29 <holow29@gmail.com> wrote:
To follow up on this:
I've engaged Verizon's executive office to finally try to get to a network engineer (because I don't have a contact myself). The (proxied) response from the engineer was that they aren't receiving any announcements for these routes to AS701, and I would need to take it up with Baidu. I guess I would like to understand if that seems reasonable to people since it is my presumption that Baidu would return and say something similar (that they advertise their routes to their peers correctly and to take it up with Verizon). To me, it seems like there is clearly a failing in one of Verizon's peers where they are not advertising or accepting this route correctly, but that it would be incumbent on Verizon to do the legwork to fix it since they are the ones who know their peering agreements and have these contacts. Unfortunately it seems like policy that Verizon pushes any issues that aren't internal routing issues to an external party, but surely they have a responsibility to maintain their peering and routes to external services as well. In other words, this type of buckpassing does not seem right to me (and I've heard it from them on other routing issues before), especially given that they are the ones empowered to fix it. Any thoughts?

(As it happens, pan.baidu.com now resolves to an IP range that is routable by Verizon, but it could always revert, and it seems like Verizon should have these routes regardless.)

Thanks.

On Fri, Jun 24, 2022 at 7:41 AM Matthew Huff <mhuff@ox.com> wrote:

From my limited vantage point it appears that there is some issue between Verizon & Baidu. Baidu has 182.61.0.0/16 registered, but is only advertising pieces of it globally (or at least from what I can see). In our tables,we are receiving none from Verizon of  the subnets that are advertised directly from Baidu (origin AS of 38365). The few within that registered range that have a different origin AS are coming to us from Verizon. For example:

 

*>   182.61.0.0/19    144.121.203.141                        0 46887 3356 4134 58466 38365 i

*>   182.61.0.0/18    144.121.203.141                        0 46887 6461 4134 58466 38365 38365 i

*>   182.61.32.0/19   144.121.203.141                        0 46887 3356 4134 58466 38365 i

*>   182.61.64.0/18   204.148.121.221                        0 701 6453 55967 i

*    182.61.128.0/23  204.148.121.221                        0 701 4134 4134 4134 4134 4134 58540 ?

*>   182.61.130.0/24  144.121.203.141                        0 46887 6461 4134 23724 38365 38365 38365 i

*>   182.61.130.0/23  144.121.203.141                        0 46887 6461 4134 58466 38365 38365 i

*>   182.61.131.0/24  144.121.203.141                        0 46887 6461 4134 23724 38365 38365 38365 i

*>   182.61.132.0/23  144.121.203.141                        0 46887 3356 4134 58466 38365 i

*>   182.61.132.0/22  144.121.203.141                        0 46887 6461 4134 58466 38365 38365 i

*>   182.61.134.0/23  144.121.203.141                        0 46887 3356 4134 58466 38365 i

*>   182.61.136.0/22  144.121.203.141                        0 46887 3356 4134 58466 38365 i

*>   182.61.136.0/21  144.121.203.141                        0 46887 6461 4134 58466 38365 38365 i

*>   182.61.140.0/22  144.121.203.141                        0 46887 3356 4134 58466 38365 i

*>   182.61.144.0/21  144.121.203.141                        0 46887 3356 4134 58466 38365 i

*>   182.61.144.0/20  144.121.203.141                        0 46887 6461 4134 58466 38365 38365 i

*>   182.61.160.0/19  204.148.121.221                        0 701 6453 55967 i

*>   182.61.192.0/23  144.121.203.141                        0 46887 3356 4134 58540 i

*>   182.61.194.0/23  144.121.203.141                        0 46887 3356 4134 58540 i

*>   182.61.200.0/22  144.121.203.141                        0 46887 6461 4134 23724 38365 i

*>   182.61.200.0/21  144.121.203.141                        0 46887 6461 4134 58466 38365 38365 i

*>   182.61.216.0/21  144.121.203.141                        0 46887 6461 4134 58466 38365 38365 i

*>   182.61.223.0/24  144.121.203.141                        0 46887 6461 4134 58466 38365 38365 i

*>   182.61.224.0/19  144.121.203.141                        0 46887 6461 4134 58466 38365 38365 i

 

We are getting the 182.61.200.0/21 and 182.61.200.0/22 from all of our other peers:

 

asr-inet2#sh ip bgp 182.61.200.0/21

BGP routing table entry for 182.61.200.0/21, version 15779018

Paths: (2 available, best #2, table default)

  Advertised to update-groups:

     3        

  Refresh Epoch 1

  54004 6128 1299 4134 58466 38365 38365, (aggregated by 38365 119.75.208.225)

    148.77.99.201 from 148.77.99.201 (24.157.4.25)

      Origin IGP, localpref 100, valid, external, atomic-aggregate

      rx pathid: 0, tx pathid: 0

      Updated on Apr 29 2022 21:02:05 EDT

  Refresh Epoch 1

  46887 6461 4134 58466 38365 38365, (aggregated by 38365 119.75.208.225)

    129.77.17.254 from 129.77.17.254 (129.77.40.7)

      Origin IGP, metric 0, localpref 100, valid, internal, atomic-aggregate, best

      rx pathid: 0, tx pathid: 0x0

      Updated on May 3 2022 04:02:50 EDT

 

 

From: NANOG <nanog-bounces+mhuff=ox.com@nanog.org> On Behalf Of holow29
Sent: Thursday, June 23, 2022 5:49 PM
To: nanog@nanog.org
Subject: Verizon no BGP route to some of AS38365 (182.61.200.0/24)

 

I've been trying (to no avail) for over a month now to get Verizon to investigate their lack of BGP routing to 182.61.200.0/24, which hosts Baidu Wangpan at  pan.baidu.com (Baidu's cloud services/equivalent of Google Drive).

 

Easily verified through Verizon's Looking Glass.

 

We all know Verizon's BGP routing is a disaster, but does anyone have any ideas?