Verizon no BGP route to some of AS38365 (182.61.200.0/24)
I've been trying (to no avail) for over a month now to get Verizon to investigate their lack of BGP routing to 182.61.200.0/24, which hosts Baidu Wangpan at pan.baidu.com (Baidu's cloud services/equivalent of Google Drive). Easily verified through Verizon's Looking Glass. We all know Verizon's BGP routing is a disaster, but does anyone have any ideas?
On 6/23/2022 11:48 AM, holow29 wrote:
I've been trying (to no avail) for over a month now to get Verizon to investigate their lack of BGP routing to 182.61.200.0/24 <http://182.61.200.0/24>, which hosts Baidu Wangpan at pan.baidu.com <http://pan.baidu.com> (Baidu's cloud services/equivalent of Google Drive).
Easily verified through Verizon's Looking Glass.
We all know Verizon's BGP routing is a disaster, but does anyone have any ideas?
Looks like chinanet is the routing disaster. But over Lumen I can get to Baidu's 182.61.254.169 IP when tracerouting to 182.61.254.1. scott
From my limited vantage point it appears that there is some issue between Verizon & Baidu. Baidu has 182.61.0.0/16 registered, but is only advertising pieces of it globally (or at least from what I can see). In our tables,we are receiving none from Verizon of the subnets that are advertised directly from Baidu (origin AS of 38365). The few within that registered range that have a different origin AS are coming to us from Verizon. For example: *> 182.61.0.0/19 144.121.203.141 0 46887 3356 4134 58466 38365 i *> 182.61.0.0/18 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i *> 182.61.32.0/19 144.121.203.141 0 46887 3356 4134 58466 38365 i *> 182.61.64.0/18 204.148.121.221 0 701 6453 55967 i * 182.61.128.0/23 204.148.121.221 0 701 4134 4134 4134 4134 4134 58540 ? *> 182.61.130.0/24 144.121.203.141 0 46887 6461 4134 23724 38365 38365 38365 i *> 182.61.130.0/23 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i *> 182.61.131.0/24 144.121.203.141 0 46887 6461 4134 23724 38365 38365 38365 i *> 182.61.132.0/23 144.121.203.141 0 46887 3356 4134 58466 38365 i *> 182.61.132.0/22 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i *> 182.61.134.0/23 144.121.203.141 0 46887 3356 4134 58466 38365 i *> 182.61.136.0/22 144.121.203.141 0 46887 3356 4134 58466 38365 i *> 182.61.136.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i *> 182.61.140.0/22 144.121.203.141 0 46887 3356 4134 58466 38365 i *> 182.61.144.0/21 144.121.203.141 0 46887 3356 4134 58466 38365 i *> 182.61.144.0/20 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i *> 182.61.160.0/19 204.148.121.221 0 701 6453 55967 i *> 182.61.192.0/23 144.121.203.141 0 46887 3356 4134 58540 i *> 182.61.194.0/23 144.121.203.141 0 46887 3356 4134 58540 i *> 182.61.200.0/22 144.121.203.141 0 46887 6461 4134 23724 38365 i *> 182.61.200.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i *> 182.61.216.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i *> 182.61.223.0/24 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i *> 182.61.224.0/19 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i We are getting the 182.61.200.0/21 and 182.61.200.0/22 from all of our other peers: asr-inet2#sh ip bgp 182.61.200.0/21 BGP routing table entry for 182.61.200.0/21, version 15779018 Paths: (2 available, best #2, table default) Advertised to update-groups: 3 Refresh Epoch 1 54004 6128 1299 4134 58466 38365 38365, (aggregated by 38365 119.75.208.225) 148.77.99.201 from 148.77.99.201 (24.157.4.25) Origin IGP, localpref 100, valid, external, atomic-aggregate rx pathid: 0, tx pathid: 0 Updated on Apr 29 2022 21:02:05 EDT Refresh Epoch 1 46887 6461 4134 58466 38365 38365, (aggregated by 38365 119.75.208.225) 129.77.17.254 from 129.77.17.254 (129.77.40.7) Origin IGP, metric 0, localpref 100, valid, internal, atomic-aggregate, best rx pathid: 0, tx pathid: 0x0 Updated on May 3 2022 04:02:50 EDT From: NANOG <nanog-bounces+mhuff=ox.com@nanog.org> On Behalf Of holow29 Sent: Thursday, June 23, 2022 5:49 PM To: nanog@nanog.org Subject: Verizon no BGP route to some of AS38365 (182.61.200.0/24) I've been trying (to no avail) for over a month now to get Verizon to investigate their lack of BGP routing to 182.61.200.0/24<http://182.61.200.0/24>, which hosts Baidu Wangpan at pan.baidu.com<http://pan.baidu.com> (Baidu's cloud services/equivalent of Google Drive). Easily verified through Verizon's Looking Glass. We all know Verizon's BGP routing is a disaster, but does anyone have any ideas?
To follow up on this: I've engaged Verizon's executive office to finally try to get to a network engineer (because I don't have a contact myself). The (proxied) response from the engineer was that they aren't receiving any announcements for these routes to AS701, and I would need to take it up with Baidu. I guess I would like to understand if that seems reasonable to people since it is my presumption that Baidu would return and say something similar (that they advertise their routes to their peers correctly and to take it up with Verizon). To me, it seems like there is clearly a failing in one of Verizon's peers where they are not advertising or accepting this route correctly, but that it would be incumbent on Verizon to do the legwork to fix it since they are the ones who know their peering agreements and have these contacts. Unfortunately it seems like policy that Verizon pushes any issues that aren't internal routing issues to an external party, but surely they have a responsibility to maintain their peering and routes to external services as well. In other words, this type of buckpassing does not seem right to me (and I've heard it from them on other routing issues before), especially given that they are the ones empowered to fix it. Any thoughts? (As it happens, pan.baidu.com now resolves to an IP range that is routable by Verizon, but it could always revert, and it seems like Verizon should have these routes regardless.) Thanks. On Fri, Jun 24, 2022 at 7:41 AM Matthew Huff <mhuff@ox.com> wrote:
From my limited vantage point it appears that there is some issue between Verizon & Baidu. Baidu has 182.61.0.0/16 registered, but is only advertising pieces of it globally (or at least from what I can see). In our tables,we are receiving none from Verizon of the subnets that are advertised directly from Baidu (origin AS of 38365). The few within that registered range that have a different origin AS are coming to us from Verizon. For example:
*> 182.61.0.0/19 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.0.0/18 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.32.0/19 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.64.0/18 204.148.121.221 0 701 6453 55967 i
* 182.61.128.0/23 204.148.121.221 0 701 4134 4134 4134 4134 4134 58540 ?
*> 182.61.130.0/24 144.121.203.141 0 46887 6461 4134 23724 38365 38365 38365 i
*> 182.61.130.0/23 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.131.0/24 144.121.203.141 0 46887 6461 4134 23724 38365 38365 38365 i
*> 182.61.132.0/23 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.132.0/22 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.134.0/23 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.136.0/22 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.136.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.140.0/22 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.144.0/21 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.144.0/20 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.160.0/19 204.148.121.221 0 701 6453 55967 i
*> 182.61.192.0/23 144.121.203.141 0 46887 3356 4134 58540 i
*> 182.61.194.0/23 144.121.203.141 0 46887 3356 4134 58540 i
*> 182.61.200.0/22 144.121.203.141 0 46887 6461 4134 23724 38365 i
*> 182.61.200.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.216.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.223.0/24 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.224.0/19 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
We are getting the 182.61.200.0/21 and 182.61.200.0/22 from all of our other peers:
asr-inet2#sh ip bgp 182.61.200.0/21
BGP routing table entry for 182.61.200.0/21, version 15779018
Paths: (2 available, best #2, table default)
Advertised to update-groups:
3
Refresh Epoch 1
54004 6128 1299 4134 58466 38365 38365, (aggregated by 38365 119.75.208.225)
148.77.99.201 from 148.77.99.201 (24.157.4.25)
Origin IGP, localpref 100, valid, external, atomic-aggregate
rx pathid: 0, tx pathid: 0
Updated on Apr 29 2022 21:02:05 EDT
Refresh Epoch 1
46887 6461 4134 58466 38365 38365, (aggregated by 38365 119.75.208.225)
129.77.17.254 from 129.77.17.254 (129.77.40.7)
Origin IGP, metric 0, localpref 100, valid, internal, atomic-aggregate, best
rx pathid: 0, tx pathid: 0x0
Updated on May 3 2022 04:02:50 EDT
*From:* NANOG <nanog-bounces+mhuff=ox.com@nanog.org> *On Behalf Of * holow29 *Sent:* Thursday, June 23, 2022 5:49 PM *To:* nanog@nanog.org *Subject:* Verizon no BGP route to some of AS38365 (182.61.200.0/24)
I've been trying (to no avail) for over a month now to get Verizon to investigate their lack of BGP routing to 182.61.200.0/24, which hosts Baidu Wangpan at pan.baidu.com (Baidu's cloud services/equivalent of Google Drive).
Easily verified through Verizon's Looking Glass.
We all know Verizon's BGP routing is a disaster, but does anyone have any ideas?
Unfortunately it seems like policy that Verizon pushes any issues that aren't internal routing issues to an external party, but surely they have a responsibility to maintain their peering and routes to external services as well.
Baidu is behind CT. If CT is not passing on routes to 701 , what exactly do you expect Verizon to do about it? On Wed, Jul 20, 2022 at 6:40 PM holow29 <holow29@gmail.com> wrote:
To follow up on this: I've engaged Verizon's executive office to finally try to get to a network engineer (because I don't have a contact myself). The (proxied) response from the engineer was that they aren't receiving any announcements for these routes to AS701, and I would need to take it up with Baidu. I guess I would like to understand if that seems reasonable to people since it is my presumption that Baidu would return and say something similar (that they advertise their routes to their peers correctly and to take it up with Verizon). To me, it seems like there is clearly a failing in one of Verizon's peers where they are not advertising or accepting this route correctly, but that it would be incumbent on Verizon to do the legwork to fix it since they are the ones who know their peering agreements and have these contacts. Unfortunately it seems like policy that Verizon pushes any issues that aren't internal routing issues to an external party, but surely they have a responsibility to maintain their peering and routes to external services as well. In other words, this type of buckpassing does not seem right to me (and I've heard it from them on other routing issues before), especially given that they are the ones empowered to fix it. Any thoughts?
(As it happens, pan.baidu.com now resolves to an IP range that is routable by Verizon, but it could always revert, and it seems like Verizon should have these routes regardless.)
Thanks.
On Fri, Jun 24, 2022 at 7:41 AM Matthew Huff <mhuff@ox.com> wrote:
From my limited vantage point it appears that there is some issue between Verizon & Baidu. Baidu has 182.61.0.0/16 registered, but is only advertising pieces of it globally (or at least from what I can see). In our tables,we are receiving none from Verizon of the subnets that are advertised directly from Baidu (origin AS of 38365). The few within that registered range that have a different origin AS are coming to us from Verizon. For example:
*> 182.61.0.0/19 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.0.0/18 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.32.0/19 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.64.0/18 204.148.121.221 0 701 6453 55967 i
* 182.61.128.0/23 204.148.121.221 0 701 4134 4134 4134 4134 4134 58540 ?
*> 182.61.130.0/24 144.121.203.141 0 46887 6461 4134 23724 38365 38365 38365 i
*> 182.61.130.0/23 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.131.0/24 144.121.203.141 0 46887 6461 4134 23724 38365 38365 38365 i
*> 182.61.132.0/23 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.132.0/22 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.134.0/23 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.136.0/22 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.136.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.140.0/22 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.144.0/21 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.144.0/20 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.160.0/19 204.148.121.221 0 701 6453 55967 i
*> 182.61.192.0/23 144.121.203.141 0 46887 3356 4134 58540 i
*> 182.61.194.0/23 144.121.203.141 0 46887 3356 4134 58540 i
*> 182.61.200.0/22 144.121.203.141 0 46887 6461 4134 23724 38365 i
*> 182.61.200.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.216.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.223.0/24 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.224.0/19 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
We are getting the 182.61.200.0/21 and 182.61.200.0/22 from all of our other peers:
asr-inet2#sh ip bgp 182.61.200.0/21
BGP routing table entry for 182.61.200.0/21, version 15779018
Paths: (2 available, best #2, table default)
Advertised to update-groups:
3
Refresh Epoch 1
54004 6128 1299 4134 58466 38365 38365, (aggregated by 38365 119.75.208.225)
148.77.99.201 from 148.77.99.201 (24.157.4.25)
Origin IGP, localpref 100, valid, external, atomic-aggregate
rx pathid: 0, tx pathid: 0
Updated on Apr 29 2022 21:02:05 EDT
Refresh Epoch 1
46887 6461 4134 58466 38365 38365, (aggregated by 38365 119.75.208.225)
129.77.17.254 from 129.77.17.254 (129.77.40.7)
Origin IGP, metric 0, localpref 100, valid, internal, atomic-aggregate, best
rx pathid: 0, tx pathid: 0x0
Updated on May 3 2022 04:02:50 EDT
*From:* NANOG <nanog-bounces+mhuff=ox.com@nanog.org> *On Behalf Of * holow29 *Sent:* Thursday, June 23, 2022 5:49 PM *To:* nanog@nanog.org *Subject:* Verizon no BGP route to some of AS38365 (182.61.200.0/24)
I've been trying (to no avail) for over a month now to get Verizon to investigate their lack of BGP routing to 182.61.200.0/24, which hosts Baidu Wangpan at pan.baidu.com (Baidu's cloud services/equivalent of Google Drive).
Easily verified through Verizon's Looking Glass.
We all know Verizon's BGP routing is a disaster, but does anyone have any ideas?
I would expect Verizon to be able to contact CT and figure out why they aren't passing the correct routes just as I might expect Baidu to do the same thing, I suppose. Ultimately, whose responsibility is it other than CT? That is my question. Maybe in this instance, it is common in the industry for this to be the responsibility of Baidu. That seems counterintuitive to me as it is Verizon without the proper route ultimately, but I am not an expert. Certainly, I think it is incorrect to ask the customer to try to resolve these issues after bringing it to the attention of these services. If Verizon couldn't reach one of Google's edge servers, would it be Verizon or Google's responsibility to fix that if the issue were an intermediate peer failing to broadcase the correct route? I have the sneaking suspicion that Verizon might actually take some action in that instance... On Thu, Jul 21, 2022 at 9:31 AM Tom Beecher <beecher@beecher.cc> wrote:
Unfortunately it seems like policy that Verizon pushes any issues that
aren't internal routing issues to an external party, but surely they have a responsibility to maintain their peering and routes to external services as well.
Baidu is behind CT. If CT is not passing on routes to 701 , what exactly do you expect Verizon to do about it?
On Wed, Jul 20, 2022 at 6:40 PM holow29 <holow29@gmail.com> wrote:
To follow up on this: I've engaged Verizon's executive office to finally try to get to a network engineer (because I don't have a contact myself). The (proxied) response from the engineer was that they aren't receiving any announcements for these routes to AS701, and I would need to take it up with Baidu. I guess I would like to understand if that seems reasonable to people since it is my presumption that Baidu would return and say something similar (that they advertise their routes to their peers correctly and to take it up with Verizon). To me, it seems like there is clearly a failing in one of Verizon's peers where they are not advertising or accepting this route correctly, but that it would be incumbent on Verizon to do the legwork to fix it since they are the ones who know their peering agreements and have these contacts. Unfortunately it seems like policy that Verizon pushes any issues that aren't internal routing issues to an external party, but surely they have a responsibility to maintain their peering and routes to external services as well. In other words, this type of buckpassing does not seem right to me (and I've heard it from them on other routing issues before), especially given that they are the ones empowered to fix it. Any thoughts?
(As it happens, pan.baidu.com now resolves to an IP range that is routable by Verizon, but it could always revert, and it seems like Verizon should have these routes regardless.)
Thanks.
On Fri, Jun 24, 2022 at 7:41 AM Matthew Huff <mhuff@ox.com> wrote:
From my limited vantage point it appears that there is some issue between Verizon & Baidu. Baidu has 182.61.0.0/16 registered, but is only advertising pieces of it globally (or at least from what I can see). In our tables,we are receiving none from Verizon of the subnets that are advertised directly from Baidu (origin AS of 38365). The few within that registered range that have a different origin AS are coming to us from Verizon. For example:
*> 182.61.0.0/19 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.0.0/18 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.32.0/19 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.64.0/18 204.148.121.221 0 701 6453 55967 i
* 182.61.128.0/23 204.148.121.221 0 701 4134 4134 4134 4134 4134 58540 ?
*> 182.61.130.0/24 144.121.203.141 0 46887 6461 4134 23724 38365 38365 38365 i
*> 182.61.130.0/23 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.131.0/24 144.121.203.141 0 46887 6461 4134 23724 38365 38365 38365 i
*> 182.61.132.0/23 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.132.0/22 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.134.0/23 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.136.0/22 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.136.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.140.0/22 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.144.0/21 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.144.0/20 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.160.0/19 204.148.121.221 0 701 6453 55967 i
*> 182.61.192.0/23 144.121.203.141 0 46887 3356 4134 58540 i
*> 182.61.194.0/23 144.121.203.141 0 46887 3356 4134 58540 i
*> 182.61.200.0/22 144.121.203.141 0 46887 6461 4134 23724 38365 i
*> 182.61.200.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.216.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.223.0/24 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.224.0/19 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
We are getting the 182.61.200.0/21 and 182.61.200.0/22 from all of our other peers:
asr-inet2#sh ip bgp 182.61.200.0/21
BGP routing table entry for 182.61.200.0/21, version 15779018
Paths: (2 available, best #2, table default)
Advertised to update-groups:
3
Refresh Epoch 1
54004 6128 1299 4134 58466 38365 38365, (aggregated by 38365 119.75.208.225)
148.77.99.201 from 148.77.99.201 (24.157.4.25)
Origin IGP, localpref 100, valid, external, atomic-aggregate
rx pathid: 0, tx pathid: 0
Updated on Apr 29 2022 21:02:05 EDT
Refresh Epoch 1
46887 6461 4134 58466 38365 38365, (aggregated by 38365 119.75.208.225)
129.77.17.254 from 129.77.17.254 (129.77.40.7)
Origin IGP, metric 0, localpref 100, valid, internal, atomic-aggregate, best
rx pathid: 0, tx pathid: 0x0
Updated on May 3 2022 04:02:50 EDT
*From:* NANOG <nanog-bounces+mhuff=ox.com@nanog.org> *On Behalf Of * holow29 *Sent:* Thursday, June 23, 2022 5:49 PM *To:* nanog@nanog.org *Subject:* Verizon no BGP route to some of AS38365 (182.61.200.0/24)
I've been trying (to no avail) for over a month now to get Verizon to investigate their lack of BGP routing to 182.61.200.0/24, which hosts Baidu Wangpan at pan.baidu.com (Baidu's cloud services/equivalent of Google Drive).
Easily verified through Verizon's Looking Glass.
We all know Verizon's BGP routing is a disaster, but does anyone have any ideas?
I would expect Verizon to be able to contact CT and figure out why they aren't passing the *correct* routes just as I might expect Baidu to do the same thing, I suppose.
What defines 'correct'? ASN's routinely make traffic engineering or business decisions not to announce certain prefixes to certain other ASNs. This certainly does sometimes cause reachability issues for end users, but that's a choice someone made. That's part of why it's generally good practice to get more than 1 upstream if you can; it gives you more control to mitigate the impact of these choices. It is completely possible that Baidu or CT are intentionally not announcing prefixes to VZ. It is also completely possible that they are and VZ is not accepting it. On Thu, Jul 21, 2022 at 11:24 AM holow29 <holow29@gmail.com> wrote:
I would expect Verizon to be able to contact CT and figure out why they aren't passing the correct routes just as I might expect Baidu to do the same thing, I suppose. Ultimately, whose responsibility is it other than CT? That is my question. Maybe in this instance, it is common in the industry for this to be the responsibility of Baidu. That seems counterintuitive to me as it is Verizon without the proper route ultimately, but I am not an expert. Certainly, I think it is incorrect to ask the customer to try to resolve these issues after bringing it to the attention of these services. If Verizon couldn't reach one of Google's edge servers, would it be Verizon or Google's responsibility to fix that if the issue were an intermediate peer failing to broadcase the correct route? I have the sneaking suspicion that Verizon might actually take some action in that instance...
On Thu, Jul 21, 2022 at 9:31 AM Tom Beecher <beecher@beecher.cc> wrote:
Unfortunately it seems like policy that Verizon pushes any issues that
aren't internal routing issues to an external party, but surely they have a responsibility to maintain their peering and routes to external services as well.
Baidu is behind CT. If CT is not passing on routes to 701 , what exactly do you expect Verizon to do about it?
On Wed, Jul 20, 2022 at 6:40 PM holow29 <holow29@gmail.com> wrote:
To follow up on this: I've engaged Verizon's executive office to finally try to get to a network engineer (because I don't have a contact myself). The (proxied) response from the engineer was that they aren't receiving any announcements for these routes to AS701, and I would need to take it up with Baidu. I guess I would like to understand if that seems reasonable to people since it is my presumption that Baidu would return and say something similar (that they advertise their routes to their peers correctly and to take it up with Verizon). To me, it seems like there is clearly a failing in one of Verizon's peers where they are not advertising or accepting this route correctly, but that it would be incumbent on Verizon to do the legwork to fix it since they are the ones who know their peering agreements and have these contacts. Unfortunately it seems like policy that Verizon pushes any issues that aren't internal routing issues to an external party, but surely they have a responsibility to maintain their peering and routes to external services as well. In other words, this type of buckpassing does not seem right to me (and I've heard it from them on other routing issues before), especially given that they are the ones empowered to fix it. Any thoughts?
(As it happens, pan.baidu.com now resolves to an IP range that is routable by Verizon, but it could always revert, and it seems like Verizon should have these routes regardless.)
Thanks.
On Fri, Jun 24, 2022 at 7:41 AM Matthew Huff <mhuff@ox.com> wrote:
From my limited vantage point it appears that there is some issue between Verizon & Baidu. Baidu has 182.61.0.0/16 registered, but is only advertising pieces of it globally (or at least from what I can see). In our tables,we are receiving none from Verizon of the subnets that are advertised directly from Baidu (origin AS of 38365). The few within that registered range that have a different origin AS are coming to us from Verizon. For example:
*> 182.61.0.0/19 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.0.0/18 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.32.0/19 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.64.0/18 204.148.121.221 0 701 6453 55967 i
* 182.61.128.0/23 204.148.121.221 0 701 4134 4134 4134 4134 4134 58540 ?
*> 182.61.130.0/24 144.121.203.141 0 46887 6461 4134 23724 38365 38365 38365 i
*> 182.61.130.0/23 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.131.0/24 144.121.203.141 0 46887 6461 4134 23724 38365 38365 38365 i
*> 182.61.132.0/23 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.132.0/22 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.134.0/23 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.136.0/22 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.136.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.140.0/22 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.144.0/21 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.144.0/20 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.160.0/19 204.148.121.221 0 701 6453 55967 i
*> 182.61.192.0/23 144.121.203.141 0 46887 3356 4134 58540 i
*> 182.61.194.0/23 144.121.203.141 0 46887 3356 4134 58540 i
*> 182.61.200.0/22 144.121.203.141 0 46887 6461 4134 23724 38365 i
*> 182.61.200.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.216.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.223.0/24 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.224.0/19 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
We are getting the 182.61.200.0/21 and 182.61.200.0/22 from all of our other peers:
asr-inet2#sh ip bgp 182.61.200.0/21
BGP routing table entry for 182.61.200.0/21, version 15779018
Paths: (2 available, best #2, table default)
Advertised to update-groups:
3
Refresh Epoch 1
54004 6128 1299 4134 58466 38365 38365, (aggregated by 38365 119.75.208.225)
148.77.99.201 from 148.77.99.201 (24.157.4.25)
Origin IGP, localpref 100, valid, external, atomic-aggregate
rx pathid: 0, tx pathid: 0
Updated on Apr 29 2022 21:02:05 EDT
Refresh Epoch 1
46887 6461 4134 58466 38365 38365, (aggregated by 38365 119.75.208.225)
129.77.17.254 from 129.77.17.254 (129.77.40.7)
Origin IGP, metric 0, localpref 100, valid, internal, atomic-aggregate, best
rx pathid: 0, tx pathid: 0x0
Updated on May 3 2022 04:02:50 EDT
*From:* NANOG <nanog-bounces+mhuff=ox.com@nanog.org> *On Behalf Of * holow29 *Sent:* Thursday, June 23, 2022 5:49 PM *To:* nanog@nanog.org *Subject:* Verizon no BGP route to some of AS38365 (182.61.200.0/24)
I've been trying (to no avail) for over a month now to get Verizon to investigate their lack of BGP routing to 182.61.200.0/24, which hosts Baidu Wangpan at pan.baidu.com (Baidu's cloud services/equivalent of Google Drive).
Easily verified through Verizon's Looking Glass.
We all know Verizon's BGP routing is a disaster, but does anyone have any ideas?
In this instance, I would define "correct" behavior as VZ having any route to this subnet; after all, customers don't pay VZ to access just their network or a subset of the internet. You make a good point, though I would expect that if it isn't VZ's business decision to not have this route, they would have some sort of vested interest in figuring out why CT is not advertising it to them.
From VZ engineer, I heard that no one is advertising it to AS701, so I assume it is not a case of VZ refusing to accept it (which is what I had initially assumed having been frustrated in the past with VZ on other matters + seeing all their BGP issues in the past few years).
On Thu, Jul 21, 2022 at 11:40 AM Tom Beecher <beecher@beecher.cc> wrote:
I would expect Verizon to be able to contact CT and figure out why they
aren't passing the *correct* routes just as I might expect Baidu to do the same thing, I suppose.
What defines 'correct'? ASN's routinely make traffic engineering or business decisions not to announce certain prefixes to certain other ASNs. This certainly does sometimes cause reachability issues for end users, but that's a choice someone made. That's part of why it's generally good practice to get more than 1 upstream if you can; it gives you more control to mitigate the impact of these choices.
It is completely possible that Baidu or CT are intentionally not announcing prefixes to VZ. It is also completely possible that they are and VZ is not accepting it.
On Thu, Jul 21, 2022 at 11:24 AM holow29 <holow29@gmail.com> wrote:
I would expect Verizon to be able to contact CT and figure out why they aren't passing the correct routes just as I might expect Baidu to do the same thing, I suppose. Ultimately, whose responsibility is it other than CT? That is my question. Maybe in this instance, it is common in the industry for this to be the responsibility of Baidu. That seems counterintuitive to me as it is Verizon without the proper route ultimately, but I am not an expert. Certainly, I think it is incorrect to ask the customer to try to resolve these issues after bringing it to the attention of these services. If Verizon couldn't reach one of Google's edge servers, would it be Verizon or Google's responsibility to fix that if the issue were an intermediate peer failing to broadcase the correct route? I have the sneaking suspicion that Verizon might actually take some action in that instance...
On Thu, Jul 21, 2022 at 9:31 AM Tom Beecher <beecher@beecher.cc> wrote:
Unfortunately it seems like policy that Verizon pushes any issues that
aren't internal routing issues to an external party, but surely they have a responsibility to maintain their peering and routes to external services as well.
Baidu is behind CT. If CT is not passing on routes to 701 , what exactly do you expect Verizon to do about it?
On Wed, Jul 20, 2022 at 6:40 PM holow29 <holow29@gmail.com> wrote:
To follow up on this: I've engaged Verizon's executive office to finally try to get to a network engineer (because I don't have a contact myself). The (proxied) response from the engineer was that they aren't receiving any announcements for these routes to AS701, and I would need to take it up with Baidu. I guess I would like to understand if that seems reasonable to people since it is my presumption that Baidu would return and say something similar (that they advertise their routes to their peers correctly and to take it up with Verizon). To me, it seems like there is clearly a failing in one of Verizon's peers where they are not advertising or accepting this route correctly, but that it would be incumbent on Verizon to do the legwork to fix it since they are the ones who know their peering agreements and have these contacts. Unfortunately it seems like policy that Verizon pushes any issues that aren't internal routing issues to an external party, but surely they have a responsibility to maintain their peering and routes to external services as well. In other words, this type of buckpassing does not seem right to me (and I've heard it from them on other routing issues before), especially given that they are the ones empowered to fix it. Any thoughts?
(As it happens, pan.baidu.com now resolves to an IP range that is routable by Verizon, but it could always revert, and it seems like Verizon should have these routes regardless.)
Thanks.
On Fri, Jun 24, 2022 at 7:41 AM Matthew Huff <mhuff@ox.com> wrote:
From my limited vantage point it appears that there is some issue between Verizon & Baidu. Baidu has 182.61.0.0/16 registered, but is only advertising pieces of it globally (or at least from what I can see). In our tables,we are receiving none from Verizon of the subnets that are advertised directly from Baidu (origin AS of 38365). The few within that registered range that have a different origin AS are coming to us from Verizon. For example:
*> 182.61.0.0/19 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.0.0/18 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.32.0/19 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.64.0/18 204.148.121.221 0 701 6453 55967 i
* 182.61.128.0/23 204.148.121.221 0 701 4134 4134 4134 4134 4134 58540 ?
*> 182.61.130.0/24 144.121.203.141 0 46887 6461 4134 23724 38365 38365 38365 i
*> 182.61.130.0/23 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.131.0/24 144.121.203.141 0 46887 6461 4134 23724 38365 38365 38365 i
*> 182.61.132.0/23 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.132.0/22 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.134.0/23 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.136.0/22 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.136.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.140.0/22 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.144.0/21 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.144.0/20 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.160.0/19 204.148.121.221 0 701 6453 55967 i
*> 182.61.192.0/23 144.121.203.141 0 46887 3356 4134 58540 i
*> 182.61.194.0/23 144.121.203.141 0 46887 3356 4134 58540 i
*> 182.61.200.0/22 144.121.203.141 0 46887 6461 4134 23724 38365 i
*> 182.61.200.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.216.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.223.0/24 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.224.0/19 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
We are getting the 182.61.200.0/21 and 182.61.200.0/22 from all of our other peers:
asr-inet2#sh ip bgp 182.61.200.0/21
BGP routing table entry for 182.61.200.0/21, version 15779018
Paths: (2 available, best #2, table default)
Advertised to update-groups:
3
Refresh Epoch 1
54004 6128 1299 4134 58466 38365 38365, (aggregated by 38365 119.75.208.225)
148.77.99.201 from 148.77.99.201 (24.157.4.25)
Origin IGP, localpref 100, valid, external, atomic-aggregate
rx pathid: 0, tx pathid: 0
Updated on Apr 29 2022 21:02:05 EDT
Refresh Epoch 1
46887 6461 4134 58466 38365 38365, (aggregated by 38365 119.75.208.225)
129.77.17.254 from 129.77.17.254 (129.77.40.7)
Origin IGP, metric 0, localpref 100, valid, internal, atomic-aggregate, best
rx pathid: 0, tx pathid: 0x0
Updated on May 3 2022 04:02:50 EDT
*From:* NANOG <nanog-bounces+mhuff=ox.com@nanog.org> *On Behalf Of * holow29 *Sent:* Thursday, June 23, 2022 5:49 PM *To:* nanog@nanog.org *Subject:* Verizon no BGP route to some of AS38365 (182.61.200.0/24)
I've been trying (to no avail) for over a month now to get Verizon to investigate their lack of BGP routing to 182.61.200.0/24, which hosts Baidu Wangpan at pan.baidu.com (Baidu's cloud services/equivalent of Google Drive).
Easily verified through Verizon's Looking Glass.
We all know Verizon's BGP routing is a disaster, but does anyone have any ideas?
In this instance, I would define "correct" behavior as VZ having any route to this subnet; after all, customers don't pay VZ to access just their network or a subset of the internet.
https://en.wikipedia.org/wiki/Default-free_zone 701 is still (AFAIK) in the DFZ , meaning if they don't get an advertisement for a prefix, they ain't getting there. On Thu, Jul 21, 2022 at 11:54 AM holow29 <holow29@gmail.com> wrote:
In this instance, I would define "correct" behavior as VZ having any route to this subnet; after all, customers don't pay VZ to access just their network or a subset of the internet. You make a good point, though I would expect that if it isn't VZ's business decision to not have this route, they would have some sort of vested interest in figuring out why CT is not advertising it to them. From VZ engineer, I heard that no one is advertising it to AS701, so I assume it is not a case of VZ refusing to accept it (which is what I had initially assumed having been frustrated in the past with VZ on other matters + seeing all their BGP issues in the past few years).
On Thu, Jul 21, 2022 at 11:40 AM Tom Beecher <beecher@beecher.cc> wrote:
I would expect Verizon to be able to contact CT and figure out why they
aren't passing the *correct* routes just as I might expect Baidu to do the same thing, I suppose.
What defines 'correct'? ASN's routinely make traffic engineering or business decisions not to announce certain prefixes to certain other ASNs. This certainly does sometimes cause reachability issues for end users, but that's a choice someone made. That's part of why it's generally good practice to get more than 1 upstream if you can; it gives you more control to mitigate the impact of these choices.
It is completely possible that Baidu or CT are intentionally not announcing prefixes to VZ. It is also completely possible that they are and VZ is not accepting it.
On Thu, Jul 21, 2022 at 11:24 AM holow29 <holow29@gmail.com> wrote:
I would expect Verizon to be able to contact CT and figure out why they aren't passing the correct routes just as I might expect Baidu to do the same thing, I suppose. Ultimately, whose responsibility is it other than CT? That is my question. Maybe in this instance, it is common in the industry for this to be the responsibility of Baidu. That seems counterintuitive to me as it is Verizon without the proper route ultimately, but I am not an expert. Certainly, I think it is incorrect to ask the customer to try to resolve these issues after bringing it to the attention of these services. If Verizon couldn't reach one of Google's edge servers, would it be Verizon or Google's responsibility to fix that if the issue were an intermediate peer failing to broadcase the correct route? I have the sneaking suspicion that Verizon might actually take some action in that instance...
On Thu, Jul 21, 2022 at 9:31 AM Tom Beecher <beecher@beecher.cc> wrote:
Unfortunately it seems like policy that Verizon pushes any issues that
aren't internal routing issues to an external party, but surely they have a responsibility to maintain their peering and routes to external services as well.
Baidu is behind CT. If CT is not passing on routes to 701 , what exactly do you expect Verizon to do about it?
On Wed, Jul 20, 2022 at 6:40 PM holow29 <holow29@gmail.com> wrote:
To follow up on this: I've engaged Verizon's executive office to finally try to get to a network engineer (because I don't have a contact myself). The (proxied) response from the engineer was that they aren't receiving any announcements for these routes to AS701, and I would need to take it up with Baidu. I guess I would like to understand if that seems reasonable to people since it is my presumption that Baidu would return and say something similar (that they advertise their routes to their peers correctly and to take it up with Verizon). To me, it seems like there is clearly a failing in one of Verizon's peers where they are not advertising or accepting this route correctly, but that it would be incumbent on Verizon to do the legwork to fix it since they are the ones who know their peering agreements and have these contacts. Unfortunately it seems like policy that Verizon pushes any issues that aren't internal routing issues to an external party, but surely they have a responsibility to maintain their peering and routes to external services as well. In other words, this type of buckpassing does not seem right to me (and I've heard it from them on other routing issues before), especially given that they are the ones empowered to fix it. Any thoughts?
(As it happens, pan.baidu.com now resolves to an IP range that is routable by Verizon, but it could always revert, and it seems like Verizon should have these routes regardless.)
Thanks.
On Fri, Jun 24, 2022 at 7:41 AM Matthew Huff <mhuff@ox.com> wrote:
From my limited vantage point it appears that there is some issue between Verizon & Baidu. Baidu has 182.61.0.0/16 registered, but is only advertising pieces of it globally (or at least from what I can see). In our tables,we are receiving none from Verizon of the subnets that are advertised directly from Baidu (origin AS of 38365). The few within that registered range that have a different origin AS are coming to us from Verizon. For example:
*> 182.61.0.0/19 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.0.0/18 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.32.0/19 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.64.0/18 204.148.121.221 0 701 6453 55967 i
* 182.61.128.0/23 204.148.121.221 0 701 4134 4134 4134 4134 4134 58540 ?
*> 182.61.130.0/24 144.121.203.141 0 46887 6461 4134 23724 38365 38365 38365 i
*> 182.61.130.0/23 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.131.0/24 144.121.203.141 0 46887 6461 4134 23724 38365 38365 38365 i
*> 182.61.132.0/23 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.132.0/22 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.134.0/23 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.136.0/22 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.136.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.140.0/22 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.144.0/21 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.144.0/20 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.160.0/19 204.148.121.221 0 701 6453 55967 i
*> 182.61.192.0/23 144.121.203.141 0 46887 3356 4134 58540 i
*> 182.61.194.0/23 144.121.203.141 0 46887 3356 4134 58540 i
*> 182.61.200.0/22 144.121.203.141 0 46887 6461 4134 23724 38365 i
*> 182.61.200.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.216.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.223.0/24 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.224.0/19 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
We are getting the 182.61.200.0/21 and 182.61.200.0/22 from all of our other peers:
asr-inet2#sh ip bgp 182.61.200.0/21
BGP routing table entry for 182.61.200.0/21, version 15779018
Paths: (2 available, best #2, table default)
Advertised to update-groups:
3
Refresh Epoch 1
54004 6128 1299 4134 58466 38365 38365, (aggregated by 38365 119.75.208.225)
148.77.99.201 from 148.77.99.201 (24.157.4.25)
Origin IGP, localpref 100, valid, external, atomic-aggregate
rx pathid: 0, tx pathid: 0
Updated on Apr 29 2022 21:02:05 EDT
Refresh Epoch 1
46887 6461 4134 58466 38365 38365, (aggregated by 38365 119.75.208.225)
129.77.17.254 from 129.77.17.254 (129.77.40.7)
Origin IGP, metric 0, localpref 100, valid, internal, atomic-aggregate, best
rx pathid: 0, tx pathid: 0x0
Updated on May 3 2022 04:02:50 EDT
*From:* NANOG <nanog-bounces+mhuff=ox.com@nanog.org> *On Behalf Of * holow29 *Sent:* Thursday, June 23, 2022 5:49 PM *To:* nanog@nanog.org *Subject:* Verizon no BGP route to some of AS38365 (182.61.200.0/24)
I've been trying (to no avail) for over a month now to get Verizon to investigate their lack of BGP routing to 182.61.200.0/24, which hosts Baidu Wangpan at pan.baidu.com (Baidu's cloud services/equivalent of Google Drive).
Easily verified through Verizon's Looking Glass.
We all know Verizon's BGP routing is a disaster, but does anyone have any ideas?
From the route-exporting network's perspective, however, they have no reason to believe that the receiving network would listen to and obey MEDs that they send, to shift traffic away from the congested
holow29, Usually decisions made about limiting route propagation outbound for certain prefixes tend to happen along financial and operational constraints. Remember, outbound route announcements control inbound traffic volumes. So, if you've got some full pipes across the ocean, for example, one way to limit the amount of traffic trying to go across them is to stop announcing some high-traffic prefixes across them. Note that it is almost always the route *announcer* that is subtracting out routes to control traffic volumes; very seldom does the receiver completely subtract out prefixes heard. The more usual case is for the receiver to de-preference the routes down, but to keep them in their tables as a fallback, because they can control that behaviour. path. They might have tried adding some prepends along the path, only to discover the receiving network was using LOCALPREF knob adjustments to control traffic, which kinda shits all over attempts to traffic engineer via prepending, at which point the only knob you have left to turn in order to move traffic off the pathway is to stop announcing prefixes entirely. As a not-so-parenthetical aside, this is why I *strongly* encourage networks that use LOCALPREF as a control knob to limit the depth to which LOCALPREF adjustments are made; ie, allow overriding of the default LOCALPREF only to paths that are $AVG_PATH_LENGTH+2 or shorter. That way, if a peer wants to signal a pathway is congested and should not be used under normal circumstances, they can still do that by triple-prepending their ASN. That way, peers can continue to announce prefixes, but still have some control over traffic flows via as-path prepending, versus being backed into the corner of having to completely stop announcing prefixes entirely across certain peering sessions. It is very likely that in this case, CT is not announcing prefixes to AS701 in order to control inbound traffic volumes across certain connections, because CT has determined that is the only control knob they have available that works; they have probably already experimented and found that AS701 squashes inbound MEDs, and uses LOCALPREF to force traffic flows regardless of as-path prepending. Your only recourse in a situation like this is likely to be getting a second connection that bypasses the CT-VZ choke point. Best of luck! Matt On Thu, Jul 21, 2022 at 8:56 AM holow29 <holow29@gmail.com> wrote:
In this instance, I would define "correct" behavior as VZ having any route to this subnet; after all, customers don't pay VZ to access just their network or a subset of the internet. You make a good point, though I would expect that if it isn't VZ's business decision to not have this route, they would have some sort of vested interest in figuring out why CT is not advertising it to them. From VZ engineer, I heard that no one is advertising it to AS701, so I assume it is not a case of VZ refusing to accept it (which is what I had initially assumed having been frustrated in the past with VZ on other matters + seeing all their BGP issues in the past few years).
On Thu, Jul 21, 2022 at 11:40 AM Tom Beecher <beecher@beecher.cc> wrote:
I would expect Verizon to be able to contact CT and figure out why they
aren't passing the *correct* routes just as I might expect Baidu to do the same thing, I suppose.
What defines 'correct'? ASN's routinely make traffic engineering or business decisions not to announce certain prefixes to certain other ASNs. This certainly does sometimes cause reachability issues for end users, but that's a choice someone made. That's part of why it's generally good practice to get more than 1 upstream if you can; it gives you more control to mitigate the impact of these choices.
It is completely possible that Baidu or CT are intentionally not announcing prefixes to VZ. It is also completely possible that they are and VZ is not accepting it.
On Thu, Jul 21, 2022 at 11:24 AM holow29 <holow29@gmail.com> wrote:
I would expect Verizon to be able to contact CT and figure out why they aren't passing the correct routes just as I might expect Baidu to do the same thing, I suppose. Ultimately, whose responsibility is it other than CT? That is my question. Maybe in this instance, it is common in the industry for this to be the responsibility of Baidu. That seems counterintuitive to me as it is Verizon without the proper route ultimately, but I am not an expert. Certainly, I think it is incorrect to ask the customer to try to resolve these issues after bringing it to the attention of these services. If Verizon couldn't reach one of Google's edge servers, would it be Verizon or Google's responsibility to fix that if the issue were an intermediate peer failing to broadcase the correct route? I have the sneaking suspicion that Verizon might actually take some action in that instance...
On Thu, Jul 21, 2022 at 9:31 AM Tom Beecher <beecher@beecher.cc> wrote:
Unfortunately it seems like policy that Verizon pushes any issues that
aren't internal routing issues to an external party, but surely they have a responsibility to maintain their peering and routes to external services as well.
Baidu is behind CT. If CT is not passing on routes to 701 , what exactly do you expect Verizon to do about it?
On Wed, Jul 20, 2022 at 6:40 PM holow29 <holow29@gmail.com> wrote:
To follow up on this: I've engaged Verizon's executive office to finally try to get to a network engineer (because I don't have a contact myself). The (proxied) response from the engineer was that they aren't receiving any announcements for these routes to AS701, and I would need to take it up with Baidu. I guess I would like to understand if that seems reasonable to people since it is my presumption that Baidu would return and say something similar (that they advertise their routes to their peers correctly and to take it up with Verizon). To me, it seems like there is clearly a failing in one of Verizon's peers where they are not advertising or accepting this route correctly, but that it would be incumbent on Verizon to do the legwork to fix it since they are the ones who know their peering agreements and have these contacts. Unfortunately it seems like policy that Verizon pushes any issues that aren't internal routing issues to an external party, but surely they have a responsibility to maintain their peering and routes to external services as well. In other words, this type of buckpassing does not seem right to me (and I've heard it from them on other routing issues before), especially given that they are the ones empowered to fix it. Any thoughts?
(As it happens, pan.baidu.com now resolves to an IP range that is routable by Verizon, but it could always revert, and it seems like Verizon should have these routes regardless.)
Thanks.
On Fri, Jun 24, 2022 at 7:41 AM Matthew Huff <mhuff@ox.com> wrote:
From my limited vantage point it appears that there is some issue between Verizon & Baidu. Baidu has 182.61.0.0/16 registered, but is only advertising pieces of it globally (or at least from what I can see). In our tables,we are receiving none from Verizon of the subnets that are advertised directly from Baidu (origin AS of 38365). The few within that registered range that have a different origin AS are coming to us from Verizon. For example:
*> 182.61.0.0/19 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.0.0/18 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.32.0/19 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.64.0/18 204.148.121.221 0 701 6453 55967 i
* 182.61.128.0/23 204.148.121.221 0 701 4134 4134 4134 4134 4134 58540 ?
*> 182.61.130.0/24 144.121.203.141 0 46887 6461 4134 23724 38365 38365 38365 i
*> 182.61.130.0/23 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.131.0/24 144.121.203.141 0 46887 6461 4134 23724 38365 38365 38365 i
*> 182.61.132.0/23 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.132.0/22 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.134.0/23 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.136.0/22 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.136.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.140.0/22 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.144.0/21 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.144.0/20 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.160.0/19 204.148.121.221 0 701 6453 55967 i
*> 182.61.192.0/23 144.121.203.141 0 46887 3356 4134 58540 i
*> 182.61.194.0/23 144.121.203.141 0 46887 3356 4134 58540 i
*> 182.61.200.0/22 144.121.203.141 0 46887 6461 4134 23724 38365 i
*> 182.61.200.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.216.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.223.0/24 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.224.0/19 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
We are getting the 182.61.200.0/21 and 182.61.200.0/22 from all of our other peers:
asr-inet2#sh ip bgp 182.61.200.0/21
BGP routing table entry for 182.61.200.0/21, version 15779018
Paths: (2 available, best #2, table default)
Advertised to update-groups:
3
Refresh Epoch 1
54004 6128 1299 4134 58466 38365 38365, (aggregated by 38365 119.75.208.225)
148.77.99.201 from 148.77.99.201 (24.157.4.25)
Origin IGP, localpref 100, valid, external, atomic-aggregate
rx pathid: 0, tx pathid: 0
Updated on Apr 29 2022 21:02:05 EDT
Refresh Epoch 1
46887 6461 4134 58466 38365 38365, (aggregated by 38365 119.75.208.225)
129.77.17.254 from 129.77.17.254 (129.77.40.7)
Origin IGP, metric 0, localpref 100, valid, internal, atomic-aggregate, best
rx pathid: 0, tx pathid: 0x0
Updated on May 3 2022 04:02:50 EDT
*From:* NANOG <nanog-bounces+mhuff=ox.com@nanog.org> *On Behalf Of * holow29 *Sent:* Thursday, June 23, 2022 5:49 PM *To:* nanog@nanog.org *Subject:* Verizon no BGP route to some of AS38365 (182.61.200.0/24)
I've been trying (to no avail) for over a month now to get Verizon to investigate their lack of BGP routing to 182.61.200.0/24, which hosts Baidu Wangpan at pan.baidu.com (Baidu's cloud services/equivalent of Google Drive).
Easily verified through Verizon's Looking Glass.
We all know Verizon's BGP routing is a disaster, but does anyone have any ideas?
On Thu, Jul 21, 2022 at 11:44 AM holow29 <holow29@gmail.com> wrote:
I would expect Verizon to be able to contact CT and figure out why they aren't passing the correct routes just as I might expect Baidu to do the same thing, I suppose. Ultimately,
You seem to be misunderstanding the relationship here... Baidu is a CUSTOMER of VZ/701 VZ's only responsibility here is to provide reachabilty to the routes which Baidu announces to VZ. There's no reason VZ should believe anything other than the fact that Baidu is not sending the particular prefix (or covering routes) to VZ 'today'.. for reasons which are entirely Baidu's. whose responsibility is it other than CT? That is my question. Maybe in
this instance, it is common in the industry for this to be the responsibility of Baidu. That seems counterintuitive to me as it is Verizon without the proper route ultimately, but I am not an expert.
The person announcing the routes (or not) is the one who makes the decision to do so... There's, of course, a possibility that VZ (or any isp instead of VZ in this particular case) has decided to filter certain prefixes from Baidu's peering, but that's not generally the case for ISPs... Note: there are no ROA for the /24 in question, but Baidu does claim they may announce: 182.61.200.0/22 according to IRR data...
Certainly, I think it is incorrect to ask the customer to try to resolve these issues after bringing it to the attention of these services. If Verizon couldn't reach one of Google's edge servers, would it be Verizon or Google's responsibility to fix that if the issue were an intermediate peer failing to broadcase the correct route? I have the sneaking suspicion that Verizon might actually take some action in that instance...
you have misunderstood the customer/provider relationship I suspect.
On Thu, Jul 21, 2022 at 9:31 AM Tom Beecher <beecher@beecher.cc> wrote:
Unfortunately it seems like policy that Verizon pushes any issues that
aren't internal routing issues to an external party, but surely they have a responsibility to maintain their peering and routes to external services as well.
Baidu is behind CT. If CT is not passing on routes to 701 , what exactly do you expect Verizon to do about it?
On Wed, Jul 20, 2022 at 6:40 PM holow29 <holow29@gmail.com> wrote:
To follow up on this: I've engaged Verizon's executive office to finally try to get to a network engineer (because I don't have a contact myself). The (proxied) response from the engineer was that they aren't receiving any announcements for these routes to AS701, and I would need to take it up with Baidu. I guess I would like to understand if that seems reasonable to people since it is my presumption that Baidu would return and say something similar (that they advertise their routes to their peers correctly and to take it up with Verizon). To me, it seems like there is clearly a failing in one of Verizon's peers where they are not advertising or accepting this route correctly, but that it would be incumbent on Verizon to do the legwork to fix it since they are the ones who know their peering agreements and have these contacts. Unfortunately it seems like policy that Verizon pushes any issues that aren't internal routing issues to an external party, but surely they have a responsibility to maintain their peering and routes to external services as well. In other words, this type of buckpassing does not seem right to me (and I've heard it from them on other routing issues before), especially given that they are the ones empowered to fix it. Any thoughts?
(As it happens, pan.baidu.com now resolves to an IP range that is routable by Verizon, but it could always revert, and it seems like Verizon should have these routes regardless.)
Thanks.
On Fri, Jun 24, 2022 at 7:41 AM Matthew Huff <mhuff@ox.com> wrote:
From my limited vantage point it appears that there is some issue between Verizon & Baidu. Baidu has 182.61.0.0/16 registered, but is only advertising pieces of it globally (or at least from what I can see). In our tables,we are receiving none from Verizon of the subnets that are advertised directly from Baidu (origin AS of 38365). The few within that registered range that have a different origin AS are coming to us from Verizon. For example:
*> 182.61.0.0/19 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.0.0/18 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.32.0/19 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.64.0/18 204.148.121.221 0 701 6453 55967 i
* 182.61.128.0/23 204.148.121.221 0 701 4134 4134 4134 4134 4134 58540 ?
*> 182.61.130.0/24 144.121.203.141 0 46887 6461 4134 23724 38365 38365 38365 i
*> 182.61.130.0/23 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.131.0/24 144.121.203.141 0 46887 6461 4134 23724 38365 38365 38365 i
*> 182.61.132.0/23 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.132.0/22 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.134.0/23 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.136.0/22 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.136.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.140.0/22 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.144.0/21 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.144.0/20 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.160.0/19 204.148.121.221 0 701 6453 55967 i
*> 182.61.192.0/23 144.121.203.141 0 46887 3356 4134 58540 i
*> 182.61.194.0/23 144.121.203.141 0 46887 3356 4134 58540 i
*> 182.61.200.0/22 144.121.203.141 0 46887 6461 4134 23724 38365 i
*> 182.61.200.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.216.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.223.0/24 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.224.0/19 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
We are getting the 182.61.200.0/21 and 182.61.200.0/22 from all of our other peers:
asr-inet2#sh ip bgp 182.61.200.0/21
BGP routing table entry for 182.61.200.0/21, version 15779018
Paths: (2 available, best #2, table default)
Advertised to update-groups:
3
Refresh Epoch 1
54004 6128 1299 4134 58466 38365 38365, (aggregated by 38365 119.75.208.225)
148.77.99.201 from 148.77.99.201 (24.157.4.25)
Origin IGP, localpref 100, valid, external, atomic-aggregate
rx pathid: 0, tx pathid: 0
Updated on Apr 29 2022 21:02:05 EDT
Refresh Epoch 1
46887 6461 4134 58466 38365 38365, (aggregated by 38365 119.75.208.225)
129.77.17.254 from 129.77.17.254 (129.77.40.7)
Origin IGP, metric 0, localpref 100, valid, internal, atomic-aggregate, best
rx pathid: 0, tx pathid: 0x0
Updated on May 3 2022 04:02:50 EDT
*From:* NANOG <nanog-bounces+mhuff=ox.com@nanog.org> *On Behalf Of * holow29 *Sent:* Thursday, June 23, 2022 5:49 PM *To:* nanog@nanog.org *Subject:* Verizon no BGP route to some of AS38365 (182.61.200.0/24)
I've been trying (to no avail) for over a month now to get Verizon to investigate their lack of BGP routing to 182.61.200.0/24, which hosts Baidu Wangpan at pan.baidu.com (Baidu's cloud services/equivalent of Google Drive).
Easily verified through Verizon's Looking Glass.
We all know Verizon's BGP routing is a disaster, but does anyone have any ideas?
Baido is NOT a customer of Verizon. They are a customer of China Telecom who has a peering relationship with Verizon.
On Jul 21, 2022, at 1:41 PM, Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Thu, Jul 21, 2022 at 11:44 AM holow29 <holow29@gmail.com> wrote: I would expect Verizon to be able to contact CT and figure out why they aren't passing the correct routes just as I might expect Baidu to do the same thing, I suppose. Ultimately,
You seem to be misunderstanding the relationship here... Baidu is a CUSTOMER of VZ/701
VZ's only responsibility here is to provide reachabilty to the routes which Baidu announces to VZ. There's no reason VZ should believe anything other than the fact that Baidu is not sending the particular prefix (or covering routes) to VZ 'today'.. for reasons which are entirely Baidu's.
whose responsibility is it other than CT? That is my question. Maybe in this instance, it is common in the industry for this to be the responsibility of Baidu. That seems counterintuitive to me as it is Verizon without the proper route ultimately, but I am not an expert.
The person announcing the routes (or not) is the one who makes the decision to do so... There's, of course, a possibility that VZ (or any isp instead of VZ in this particular case) has decided to filter certain prefixes from Baidu's peering, but that's not generally the case for ISPs...
Note: there are no ROA for the /24 in question, but Baidu does claim they may announce: 182.61.200.0/22 according to IRR data...
Certainly, I think it is incorrect to ask the customer to try to resolve these issues after bringing it to the attention of these services. If Verizon couldn't reach one of Google's edge servers, would it be Verizon or Google's responsibility to fix that if the issue were an intermediate peer failing to broadcase the correct route? I have the sneaking suspicion that Verizon might actually take some action in that instance...
you have misunderstood the customer/provider relationship I suspect.
On Thu, Jul 21, 2022 at 9:31 AM Tom Beecher <beecher@beecher.cc> wrote:
Unfortunately it seems like policy that Verizon pushes any issues that aren't internal routing issues to an external party, but surely they have a responsibility to maintain their peering and routes to external services as well.
Baidu is behind CT. If CT is not passing on routes to 701 , what exactly do you expect Verizon to do about it?
On Wed, Jul 20, 2022 at 6:40 PM holow29 <holow29@gmail.com> wrote: To follow up on this: I've engaged Verizon's executive office to finally try to get to a network engineer (because I don't have a contact myself). The (proxied) response from the engineer was that they aren't receiving any announcements for these routes to AS701, and I would need to take it up with Baidu. I guess I would like to understand if that seems reasonable to people since it is my presumption that Baidu would return and say something similar (that they advertise their routes to their peers correctly and to take it up with Verizon). To me, it seems like there is clearly a failing in one of Verizon's peers where they are not advertising or accepting this route correctly, but that it would be incumbent on Verizon to do the legwork to fix it since they are the ones who know their peering agreements and have these contacts. Unfortunately it seems like policy that Verizon pushes any issues that aren't internal routing issues to an external party, but surely they have a responsibility to maintain their peering and routes to external services as well. In other words, this type of buckpassing does not seem right to me (and I've heard it from them on other routing issues before), especially given that they are the ones empowered to fix it. Any thoughts?
(As it happens, pan.baidu.com now resolves to an IP range that is routable by Verizon, but it could always revert, and it seems like Verizon should have these routes regardless.)
Thanks.
On Fri, Jun 24, 2022 at 7:41 AM Matthew Huff <mhuff@ox.com> wrote: From my limited vantage point it appears that there is some issue between Verizon & Baidu. Baidu has 182.61.0.0/16 registered, but is only advertising pieces of it globally (or at least from what I can see). In our tables,we are receiving none from Verizon of the subnets that are advertised directly from Baidu (origin AS of 38365). The few within that registered range that have a different origin AS are coming to us from Verizon. For example:
*> 182.61.0.0/19 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.0.0/18 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.32.0/19 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.64.0/18 204.148.121.221 0 701 6453 55967 i
* 182.61.128.0/23 204.148.121.221 0 701 4134 4134 4134 4134 4134 58540 ?
*> 182.61.130.0/24 144.121.203.141 0 46887 6461 4134 23724 38365 38365 38365 i
*> 182.61.130.0/23 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.131.0/24 144.121.203.141 0 46887 6461 4134 23724 38365 38365 38365 i
*> 182.61.132.0/23 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.132.0/22 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.134.0/23 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.136.0/22 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.136.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.140.0/22 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.144.0/21 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.144.0/20 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.160.0/19 204.148.121.221 0 701 6453 55967 i
*> 182.61.192.0/23 144.121.203.141 0 46887 3356 4134 58540 i
*> 182.61.194.0/23 144.121.203.141 0 46887 3356 4134 58540 i
*> 182.61.200.0/22 144.121.203.141 0 46887 6461 4134 23724 38365 i
*> 182.61.200.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.216.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.223.0/24 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.224.0/19 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
We are getting the 182.61.200.0/21 and 182.61.200.0/22 from all of our other peers:
asr-inet2#sh ip bgp 182.61.200.0/21
BGP routing table entry for 182.61.200.0/21, version 15779018
Paths: (2 available, best #2, table default)
Advertised to update-groups:
3
Refresh Epoch 1
54004 6128 1299 4134 58466 38365 38365, (aggregated by 38365 119.75.208.225)
148.77.99.201 from 148.77.99.201 (24.157.4.25)
Origin IGP, localpref 100, valid, external, atomic-aggregate
rx pathid: 0, tx pathid: 0
Updated on Apr 29 2022 21:02:05 EDT
Refresh Epoch 1
46887 6461 4134 58466 38365 38365, (aggregated by 38365 119.75.208.225)
129.77.17.254 from 129.77.17.254 (129.77.40.7)
Origin IGP, metric 0, localpref 100, valid, internal, atomic-aggregate, best
rx pathid: 0, tx pathid: 0x0
Updated on May 3 2022 04:02:50 EDT
From: NANOG <nanog-bounces+mhuff=ox.com@nanog.org> On Behalf Of holow29 Sent: Thursday, June 23, 2022 5:49 PM To: nanog@nanog.org Subject: Verizon no BGP route to some of AS38365 (182.61.200.0/24)
I've been trying (to no avail) for over a month now to get Verizon to investigate their lack of BGP routing to 182.61.200.0/24, which hosts Baidu Wangpan at pan.baidu.com (Baidu's cloud services/equivalent of Google Drive).
Easily verified through Verizon's Looking Glass.
We all know Verizon's BGP routing is a disaster, but does anyone have any ideas?
but that it would be incumbent on Verizon to do the legwork to fix it since they are the ones who know their peering >agreements and have these contacts. Unfortunately it seems like policy that Verizon pushes any issues that aren't internal >routing issues to an external party, but surely they have a responsibility to maintain their peering and routes to external >services as well. Any thoughts?
You're probably right they have a responsibility to maintain their peering and routes, but rather than move mountains to get a large network to do "the right thing" (either vzw or baidu), I'd think most of the time it's much easier to pick a different provider to work with instead.
I looked at this a little last night, but didn't have time to write an email about it. Verizon has a lookingglass: https://www.verizon.com/business/why-verizon/looking-glass/ which you can use to see that Verizon has no route covering 182.61.200.0. Looking at routeviews, I see routes for 182.61.200.0/22, and 182.61.200.0/21, but no path via Verizon. Also looking at routeviews, there's ample evidence that Verizon and China Telecom peer, so the question is, does China Telecom not advertise these routes to Verizon, or is Verizon rejecting them for some reason? I suspect only engineers at CT and VZ can answer that. On Wed, 20 Jul 2022, holow29 wrote:
To follow up on this:I've engaged Verizon's executive office to finally try to get to a network engineer (because I don't have a contact myself). The (proxied) response from the engineer was that they aren't receiving any announcements for these routes to AS701, and I would need to take it up with Baidu. I guess I would like to understand if that seems reasonable to people since it is my presumption that Baidu would return and say something similar (that they advertise their routes to their peers correctly and to take it up with Verizon). To me, it seems like there is clearly a failing in one of Verizon's peers where they are not advertising or accepting this route correctly, but that it would be incumbent on Verizon to do the legwork to fix it since they are the ones who know their peering agreements and have these contacts. Unfortunately it seems like policy that Verizon pushes any issues that aren't internal routing issues to an external party, but surely they have a responsibility to maintain their peering and routes to external services as well. In other words, this type of buckpassing does not seem right to me (and I've heard it from them on other routing issues before), especially given that they are the ones empowered to fix it. Any thoughts?
(As it happens, pan.baidu.com now resolves to an IP range that is routable by Verizon, but it could always revert, and it seems like Verizon should have these routes regardless.)
Thanks.
On Fri, Jun 24, 2022 at 7:41 AM Matthew Huff <mhuff@ox.com> wrote:
From my limited vantage point it appears that there is some issue between Verizon & Baidu. Baidu has 182.61.0.0/16 registered, but is only advertising pieces of it globally (or at least from what I can see). In our tables,we are receiving none from Verizon of the subnets that are advertised directly from Baidu (origin AS of 38365). The few within that registered range that have a different origin AS are coming to us from Verizon. For example:
*> 182.61.0.0/19 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.0.0/18 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.32.0/19 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.64.0/18 204.148.121.221 0 701 6453 55967 i
* 182.61.128.0/23 204.148.121.221 0 701 4134 4134 4134 4134 4134 58540 ?
*> 182.61.130.0/24 144.121.203.141 0 46887 6461 4134 23724 38365 38365 38365 i
*> 182.61.130.0/23 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.131.0/24 144.121.203.141 0 46887 6461 4134 23724 38365 38365 38365 i
*> 182.61.132.0/23 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.132.0/22 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.134.0/23 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.136.0/22 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.136.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.140.0/22 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.144.0/21 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.144.0/20 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.160.0/19 204.148.121.221 0 701 6453 55967 i
*> 182.61.192.0/23 144.121.203.141 0 46887 3356 4134 58540 i
*> 182.61.194.0/23 144.121.203.141 0 46887 3356 4134 58540 i
*> 182.61.200.0/22 144.121.203.141 0 46887 6461 4134 23724 38365 i
*> 182.61.200.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.216.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.223.0/24 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.224.0/19 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
We are getting the 182.61.200.0/21 and 182.61.200.0/22 from all of our other peers:
asr-inet2#sh ip bgp 182.61.200.0/21
BGP routing table entry for 182.61.200.0/21, version 15779018
Paths: (2 available, best #2, table default)
Advertised to update-groups:
3
Refresh Epoch 1
54004 6128 1299 4134 58466 38365 38365, (aggregated by 38365 119.75.208.225)
148.77.99.201 from 148.77.99.201 (24.157.4.25)
Origin IGP, localpref 100, valid, external, atomic-aggregate
rx pathid: 0, tx pathid: 0
Updated on Apr 29 2022 21:02:05 EDT
Refresh Epoch 1
46887 6461 4134 58466 38365 38365, (aggregated by 38365 119.75.208.225)
129.77.17.254 from 129.77.17.254 (129.77.40.7)
Origin IGP, metric 0, localpref 100, valid, internal, atomic-aggregate, best
rx pathid: 0, tx pathid: 0x0
Updated on May 3 2022 04:02:50 EDT
From: NANOG <nanog-bounces+mhuff=ox.com@nanog.org> On Behalf Of holow29 Sent: Thursday, June 23, 2022 5:49 PM To: nanog@nanog.org Subject: Verizon no BGP route to some of AS38365 (182.61.200.0/24)
I've been trying (to no avail) for over a month now to get Verizon to investigate their lack of BGP routing to 182.61.200.0/24, which hosts Baidu Wangpan at pan.baidu.com (Baidu's cloud services/equivalent of Google Drive).
Easily verified through Verizon's Looking Glass.
We all know Verizon's BGP routing is a disaster, but does anyone have any ideas?
---------------------------------------------------------------------- Jon Lewis, MCP :) | I route StackPath, Sr. Neteng | therefore you are _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
Hello, On Thu, 21 Jul 2022 12:20:37 -0400 (EDT) Jon Lewis <jlewis@lewis.org> wrote:
I looked at this a little last night, but didn't have time to write an email about it. Verizon has a lookingglass:
https://www.verizon.com/business/why-verizon/looking-glass/
which you can use to see that Verizon has no route covering 182.61.200.0. Looking at routeviews, I see routes for 182.61.200.0/22, and 182.61.200.0/21, but no path via Verizon.
Just tested using: Region: Asia Pacific Location: HKG Show BGP route Net: 182.61.200.0 and the answer is: 182.61.200.0/22 (2 entries, 1 announced) *BGP Preference: 170/-101 Age: 4w0d 22:32:07 Metric: 10 Metric2: 500502 Announcement bits (4): 0-KRT 3-RT 8-BGP_RT_Background 9-Resolve tree 4 AS path: (65336) 4134 23724 38365 I (Atomic Originator) Communities: Localpref: 100 Paul
Well that shows my assertion was probably wrong. Given the geopolitical situation between the US and China, along with certain government orders, could likely infer this is intentional. On Thu, Jul 21, 2022 at 12:33 PM Paul Rolland <rol@witbe.net> wrote:
Hello,
On Thu, 21 Jul 2022 12:20:37 -0400 (EDT) Jon Lewis <jlewis@lewis.org> wrote:
I looked at this a little last night, but didn't have time to write an email about it. Verizon has a lookingglass:
https://www.verizon.com/business/why-verizon/looking-glass/
which you can use to see that Verizon has no route covering 182.61.200.0. Looking at routeviews, I see routes for 182.61.200.0/22, and 182.61.200.0/21, but no path via Verizon.
Just tested using: Region: Asia Pacific Location: HKG Show BGP route Net: 182.61.200.0
and the answer is:
182.61.200.0/22 (2 entries, 1 announced) *BGP Preference: 170/-101 Age: 4w0d 22:32:07 Metric: 10 Metric2: 500502 Announcement bits (4): 0-KRT 3-RT 8-BGP_RT_Background 9-Resolve tree 4 AS path: (65336) 4134 23724 38365 I (Atomic Originator) Communities: Localpref: 100
Paul
Hello, On Thu, 21 Jul 2022 12:39:55 -0400 Tom Beecher <beecher@beecher.cc> wrote:
Well that shows my assertion was probably wrong.
Given the geopolitical situation between the US and China, along with certain government orders, could likely infer this is intentional.
Well, I remember VZN when it was UUNet... and then, they had 3 main ASNs: - 701 (US) - 702 (EU) - 703 (APAC) Playing with the LG, it may seem that the route is visible in the "703 region", so that may be traffic engineering, geo-whatever reason, config mistake, ... Paul -- Paul Rolland E-Mail : rol(at)witbe.net CTO - Witbe.net SA Tel. +33 (0)1 47 67 77 77 18 Rue d'Arras, Bat. A11 Fax. +33 (0)1 47 67 77 99 F-92000 Nanterre RIPE : PR12-RIPE Please no HTML, I'm not a browser - Pas d'HTML, je ne suis pas un navigateur "Some people dream of success... while others wake up and work hard at it" "I worry about my child and the Internet all the time, even though she's too young to have logged on yet. Here's what I worry about. I worry that 10 or 15 years from now, she will come to me and say 'Daddy, where were you when they took freedom of the press away from the Internet?'" --Mike Godwin, Electronic Frontier Foundation
On Thu, 21 Jul 2022, Paul Rolland wrote:
Hello,
On Thu, 21 Jul 2022 12:20:37 -0400 (EDT) Jon Lewis <jlewis@lewis.org> wrote:
I looked at this a little last night, but didn't have time to write an email about it. Verizon has a lookingglass:
https://www.verizon.com/business/why-verizon/looking-glass/
which you can use to see that Verizon has no route covering 182.61.200.0. Looking at routeviews, I see routes for 182.61.200.0/22, and 182.61.200.0/21, but no path via Verizon.
Just tested using: Region: Asia Pacific Location: HKG Show BGP route Net: 182.61.200.0
and the answer is:
182.61.200.0/22 (2 entries, 1 announced) *BGP Preference: 170/-101 Age: 4w0d 22:32:07 Metric: 10 Metric2: 500502 Announcement bits (4): 0-KRT 3-RT 8-BGP_RT_Background 9-Resolve tree 4 AS path: (65336) 4134 23724 38365 I (Atomic Originator) Communities: Localpref: 100
I'd only looked using their Ashburn location. So, Verizon does receive the route, but doesn't appear to propagate it beyond the APAC region. That could still be either network's choice. The fact that their looking glass doesn't show anything for "Communities:" is suspicious to me, because I seriously doubt VZ isn't community tagging all routes...so I'm assuming there's a filter that's removing those for display...but then why even include the Communities: line in the output?...or maybe they've migrated to large communities and their looking glass hasn't been updated to display those? ---------------------------------------------------------------------- Jon Lewis, MCP :) | I route StackPath, Sr. Neteng | therefore you are _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
Also looking at routeviews, there's ample evidence that Verizon and China Telecom peer, so the question is, does China Telecom not advertise these routes to Verizon, or is Verizon rejecting them for some reason? I suspect only engineers at CT and VZ can answer that.
I took a quick look this morning from our view at all prefixes we see with an origin of AS38365. - Some prefixes I see via multiple upstreams, and for those, CT (AS4134) adds a +4 prepend via 701, but does not prepend anything else. - Other prefixes I see via multiple upstreams , with no 701 path at all. This would appear, at least externally, to be an intentional CT or Baidu decision. On Thu, Jul 21, 2022 at 12:21 PM Jon Lewis <jlewis@lewis.org> wrote:
I looked at this a little last night, but didn't have time to write an email about it. Verizon has a lookingglass:
https://www.verizon.com/business/why-verizon/looking-glass/
which you can use to see that Verizon has no route covering 182.61.200.0. Looking at routeviews, I see routes for 182.61.200.0/22, and 182.61.200.0/21, but no path via Verizon.
Also looking at routeviews, there's ample evidence that Verizon and China Telecom peer, so the question is, does China Telecom not advertise these routes to Verizon, or is Verizon rejecting them for some reason? I suspect only engineers at CT and VZ can answer that.
On Wed, 20 Jul 2022, holow29 wrote:
To follow up on this:I've engaged Verizon's executive office to finally try to get to a network engineer (because I don't have a contact myself). The (proxied) response from the engineer was that they aren't receiving any announcements for these routes to AS701, and I would need to take it up with Baidu. I guess I would like to understand if that seems reasonable to people since it is my presumption that Baidu would return and say something similar (that they advertise their routes to their peers correctly and to take it up with Verizon). To me, it seems like there is clearly a failing in one of Verizon's peers where they are not advertising or accepting this route correctly, but that it would be incumbent on Verizon to do the legwork to fix it since they are the ones who know their peering agreements and have these contacts. Unfortunately it seems like policy that Verizon pushes any issues that aren't internal routing issues to an external party, but surely they have a responsibility to maintain their peering and routes to external services as well. In other words, this type of buckpassing does not seem right to me (and I've heard it from them on other routing issues before), especially given that they are the ones empowered to fix it. Any thoughts?
(As it happens, pan.baidu.com now resolves to an IP range that is routable by Verizon, but it could always revert, and it seems like Verizon should have these routes regardless.)
Thanks.
On Fri, Jun 24, 2022 at 7:41 AM Matthew Huff <mhuff@ox.com> wrote:
From my limited vantage point it appears that there is some issue between Verizon & Baidu. Baidu has 182.61.0.0/16 registered, but is only advertising pieces of it globally (or at least from what I can see). In our tables,we are receiving none from Verizon of the subnets that are advertised directly from Baidu (origin AS of 38365). The few within that registered range that have a different origin AS are coming to us from Verizon. For example:
*> 182.61.0.0/19 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.0.0/18 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.32.0/19 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.64.0/18 204.148.121.221 0 701 6453 55967 i
* 182.61.128.0/23 204.148.121.221 0 701 4134 4134 4134 4134 4134 58540 ?
*> 182.61.130.0/24 144.121.203.141 0 46887 6461 4134 23724 38365 38365 38365 i
*> 182.61.130.0/23 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.131.0/24 144.121.203.141 0 46887 6461 4134 23724 38365 38365 38365 i
*> 182.61.132.0/23 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.132.0/22 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.134.0/23 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.136.0/22 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.136.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.140.0/22 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.144.0/21 144.121.203.141 0 46887 3356 4134 58466 38365 i
*> 182.61.144.0/20 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.160.0/19 204.148.121.221 0 701 6453 55967 i
*> 182.61.192.0/23 144.121.203.141 0 46887 3356 4134 58540 i
*> 182.61.194.0/23 144.121.203.141 0 46887 3356 4134 58540 i
*> 182.61.200.0/22 144.121.203.141 0 46887 6461 4134 23724 38365 i
*> 182.61.200.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.216.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.223.0/24 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
*> 182.61.224.0/19 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i
We are getting the 182.61.200.0/21 and 182.61.200.0/22 from all of our other peers:
asr-inet2#sh ip bgp 182.61.200.0/21
BGP routing table entry for 182.61.200.0/21, version 15779018
Paths: (2 available, best #2, table default)
Advertised to update-groups:
3
Refresh Epoch 1
54004 6128 1299 4134 58466 38365 38365, (aggregated by 38365 119.75.208.225)
148.77.99.201 from 148.77.99.201 (24.157.4.25)
Origin IGP, localpref 100, valid, external, atomic-aggregate
rx pathid: 0, tx pathid: 0
Updated on Apr 29 2022 21:02:05 EDT
Refresh Epoch 1
46887 6461 4134 58466 38365 38365, (aggregated by 38365 119.75.208.225)
129.77.17.254 from 129.77.17.254 (129.77.40.7)
Origin IGP, metric 0, localpref 100, valid, internal, atomic-aggregate, best
rx pathid: 0, tx pathid: 0x0
Updated on May 3 2022 04:02:50 EDT
From: NANOG <nanog-bounces+mhuff=ox.com@nanog.org> On Behalf Of holow29 Sent: Thursday, June 23, 2022 5:49 PM To: nanog@nanog.org Subject: Verizon no BGP route to some of AS38365 (182.61.200.0/24)
I've been trying (to no avail) for over a month now to get Verizon to investigate their lack of BGP routing to 182.61.200.0/24, which hosts Baidu Wangpan at pan.baidu.com (Baidu's cloud services/equivalent of Google Drive).
Easily verified through Verizon's Looking Glass.
We all know Verizon's BGP routing is a disaster, but does anyone have any ideas?
---------------------------------------------------------------------- Jon Lewis, MCP :) | I route StackPath, Sr. Neteng | therefore you are _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
Oddly enough I *do* see this via Verizon-but-XO: 182.61.200.0/22 *[BGP/170] 3d 09:25:39, MED 100, localpref 100 AS path: 2828 4134 23724 38365 I, validation-state: unverified On Wed, Jul 20, 2022, at 3:18 PM, holow29 wrote:
To follow up on this: I've engaged Verizon's executive office to finally try to get to a network engineer (because I don't have a contact myself). The (proxied) response from the engineer was that they aren't receiving any announcements for these routes to AS701, and I would need to take it up with Baidu. I guess I would like to understand if that seems reasonable to people since it is my presumption that Baidu would return and say something similar (that they advertise their routes to their peers correctly and to take it up with Verizon). To me, it seems like there is clearly a failing in one of Verizon's peers where they are not advertising or accepting this route correctly, but that it would be incumbent on Verizon to do the legwork to fix it since they are the ones who know their peering agreements and have these contacts. Unfortunately it seems like policy that Verizon pushes any issues that aren't internal routing issues to an external party, but surely they have a responsibility to maintain their peering and routes to external services as well. In other words, this type of buckpassing does not seem right to me (and I've heard it from them on other routing issues before), especially given that they are the ones empowered to fix it. Any thoughts?
(As it happens, pan.baidu.com now resolves to an IP range that is routable by Verizon, but it could always revert, and it seems like Verizon should have these routes regardless.)
Thanks.
On Fri, Jun 24, 2022 at 7:41 AM Matthew Huff <mhuff@ox.com> wrote:
From my limited vantage point it appears that there is some issue between Verizon & Baidu. Baidu has 182.61.0.0/16 registered, but is only advertising pieces of it globally (or at least from what I can see). In our tables,we are receiving none from Verizon of the subnets that are advertised directly from Baidu (origin AS of 38365). The few within that registered range that have a different origin AS are coming to us from Verizon. For example:__ __ __ *> 182.61.0.0/19 144.121.203.141 0 46887 3356 4134 58466 38365 i__ *> 182.61.0.0/18 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i__ *> 182.61.32.0/19 144.121.203.141 0 46887 3356 4134 58466 38365 i__ *> 182.61.64.0/18 204.148.121.221 0 701 6453 55967 i__ * 182.61.128.0/23 204.148.121.221 0 701 4134 4134 4134 4134 4134 58540 ?__ *> 182.61.130.0/24 144.121.203.141 0 46887 6461 4134 23724 38365 38365 38365 i__ *> 182.61.130.0/23 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i__ *> 182.61.131.0/24 144.121.203.141 0 46887 6461 4134 23724 38365 38365 38365 i__ *> 182.61.132.0/23 144.121.203.141 0 46887 3356 4134 58466 38365 i__ *> 182.61.132.0/22 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i__ *> 182.61.134.0/23 144.121.203.141 0 46887 3356 4134 58466 38365 i__ *> 182.61.136.0/22 144.121.203.141 0 46887 3356 4134 58466 38365 i__ *> 182.61.136.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i__ *> 182.61.140.0/22 144.121.203.141 0 46887 3356 4134 58466 38365 i__ *> 182.61.144.0/21 144.121.203.141 0 46887 3356 4134 58466 38365 i__ *> 182.61.144.0/20 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i__ *> 182.61.160.0/19 204.148.121.221 0 701 6453 55967 i__ *> 182.61.192.0/23 144.121.203.141 0 46887 3356 4134 58540 i__ *> 182.61.194.0/23 144.121.203.141 0 46887 3356 4134 58540 i__ *> 182.61.200.0/22 144.121.203.141 0 46887 6461 4134 23724 38365 i__ *> 182.61.200.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i__ *> 182.61.216.0/21 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i__ *> 182.61.223.0/24 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i__ *> 182.61.224.0/19 144.121.203.141 0 46887 6461 4134 58466 38365 38365 i__ __ __ We are getting the 182.61.200.0/21 and 182.61.200.0/22 from all of our other peers:__ __ __ asr-inet2#sh ip bgp 182.61.200.0/21__ BGP routing table entry for 182.61.200.0/21, version 15779018__ Paths: (2 available, best #2, table default)__ Advertised to update-groups:__ 3 __ __ Refresh Epoch 1__ 54004 6128 1299 4134 58466 38365 38365, (aggregated by 38365 119.75.208.225)__ 148.77.99.201 from 148.77.99.201 (24.157.4.25)__ Origin IGP, localpref 100, valid, external, atomic-aggregate__ rx pathid: 0, tx pathid: 0__ Updated on Apr 29 2022 21:02:05 EDT__ Refresh Epoch 1__ 46887 6461 4134 58466 38365 38365, (aggregated by 38365 119.75.208.225)__ 129.77.17.254 from 129.77.17.254 (129.77.40.7)__ Origin IGP, metric 0, localpref 100, valid, internal, atomic-aggregate, best__ rx pathid: 0, tx pathid: 0x0__ Updated on May 3 2022 04:02:50 EDT__ __ __ __ __
*From:* NANOG <nanog-bounces+mhuff=ox.com@nanog.org> *On Behalf Of *holow29 *Sent:* Thursday, June 23, 2022 5:49 PM *To:* nanog@nanog.org *Subject:* Verizon no BGP route to some of AS38365 (182.61.200.0/24)__
__ __ I've been trying (to no avail) for over a month now to get Verizon to investigate their lack of BGP routing to 182.61.200.0/24, which hosts Baidu Wangpan at pan.baidu.com (Baidu's cloud services/equivalent of Google Drive).__ __ __ Easily verified through Verizon's Looking Glass.__ __ __ We all know Verizon's BGP routing is a disaster, but does anyone have any ideas?__
participants (11)
-
Christopher Morrow
-
holow29
-
Jon Lewis
-
Matthew Huff
-
Matthew Petach
-
Nick Suan
-
Paul Rolland
-
Rafael Possamai
-
scott
-
sronan@ronan-online.com
-
Tom Beecher