Comcast route server not reflecting their reality
I'm seeing odditiies when trying to traffic-engineering my way around an AS 3356-to-7922 performance issue. I buy transit from 3356 and announce my /19 to 3356. I set traffic engineering community 65000:7922 to supress announcement of it from 3356 to 7922. When I do that, at route-server.newyork.ny.ibone.comcast.net I see the AS-path for my /19 change from: 3356 <me> to: 2914 <me> which makes sense as I also have transit from 2914. So that's great, 7922 should be routing to me via 2914 or, any path not via 3356. But then if I go to a customer location on 7922's network, and traceroute to my /19, it still goes via 7922 3356 <me> ! Is 7922's looking glass no longer reflecting actual route selection in their network? Does Comcast do traffic engineering that is not otherwise reflected at their looking glass? If I deaggregate my /19 and announce a /24 from it to 2914, I can draw the inbound traffic to me away from the 7922 3356 path. Which is not a real nice long-term solution... -Will
Hi Will, Unlike AS2914 which purely interconnects with AS7922, it seems that AS3356 also has direct interconnections to the various Comcast subsidiary networks which are hidden from the DFZ through no-export communities.. I figured this out due to a routing leak which happened a few years ago: https://dyn.com/blog/widespread-impact-caused-by-level-3-bgp-route-leak/ Give it a shot with the following list of communities - that should swing your traffic away from the AS3356 links: 65000:7922 65000:7015 65000:7016 65000:7725 65000:13367 65000:20214 65000:22909 65000:33287 65000:33490 65000:33491 65000:33650 65000:33651 65000:33652 65000:33657 65000:33659 65000:33660 65000:33662 65000:33667 65000:33668 Best regards, Martijn Schmidt ________________________________ From: NANOG <nanog-bounces@nanog.org> on behalf of Will Orton <will@loopfree.net> Sent: 10 September 2019 01:06 To: nanog@nanog.org <nanog@nanog.org> Subject: Comcast route server not reflecting their reality I'm seeing odditiies when trying to traffic-engineering my way around an AS 3356-to-7922 performance issue. I buy transit from 3356 and announce my /19 to 3356. I set traffic engineering community 65000:7922 to supress announcement of it from 3356 to 7922. When I do that, at route-server.newyork.ny.ibone.comcast.net I see the AS-path for my /19 change from: 3356 <me> to: 2914 <me> which makes sense as I also have transit from 2914. So that's great, 7922 should be routing to me via 2914 or, any path not via 3356. But then if I go to a customer location on 7922's network, and traceroute to my /19, it still goes via 7922 3356 <me> ! Is 7922's looking glass no longer reflecting actual route selection in their network? Does Comcast do traffic engineering that is not otherwise reflected at their looking glass? If I deaggregate my /19 and announce a /24 from it to 2914, I can draw the inbound traffic to me away from the 7922 3356 path. Which is not a real nice long-term solution... -Will
On Tue, Sep 10, 2019 at 04:42:10PM +0000, Martijn Schmidt wrote:
Hi Will,
Unlike AS2914 which purely interconnects with AS7922, it seems that AS3356 also has direct interconnections to the various Comcast subsidiary networks which are hidden from the DFZ through no-export communities.. I figured this out due to a routing leak which happened a few years ago: https://dyn.com/blog/widespread-impact-caused-by-level-3-bgp-route-leak/
Give it a shot with the following list of communities - that should swing your traffic away from the AS3356 links: 65000:7922 65000:7015 65000:7016 65000:7725 65000:13367 65000:20214 65000:22909 65000:33287 65000:33490 65000:33491 65000:33650 65000:33651 65000:33652 65000:33657 65000:33659 65000:33660 65000:33662 65000:33667 65000:33668
Best regards, Martijn Schmidt
Thanks Martijn, That's the info I was missing. I have found I can now 65000:XXX or even 65001:XXX to 3356 with the Comcast regional network AS numbers and see the traffic jump to the 7922 network which has a different set of links to 3356, then I can further do 65001:7922 to get it off 3356 entirely if I need to. And I can find the specific regional AS# by checking the aforementioned Comcast route views server so it can be applied to only problem areas. I wish none of this was necessary but when you can't get >100mbit from a Comcast gigabit subscriber into 3356 then customers complain. -Will
Unlike AS2914 which purely interconnects with AS7922, it seems that AS3356 also has direct interconnections to the various Comcast subsidiary networks which are hidden from the DFZ through no-export communities.
I'm curious how this works and why it's done this way. I have a friend who has transit from AS7922 (the remote AS in his BGP sessions are all 7922), but CenturyLink's looking glass shows an AS path of "33657 <his ASN>" and includes the no-export community. Why is the AS path rewritten? What is the design here? -Ross
On Wed, Sep 11, 2019 at 12:29:58PM -0400, Ross Tajvar wrote:
I'm curious how this works and why it's done this way. I have a friend who has transit from AS7922 (the remote AS in his BGP sessions are all 7922),
Are you sure your friend has transit from as7922? It sounds like your friend is buying from Comcast Enterprise (Ethernet Dedicated Internet), which places him on the regional CRAN network, not the 7922 national backbone (ibone).
but CenturyLink's looking glass shows an AS path of "33657 <his ASN>" and includes the no-export community. Why is the AS path rewritten? What is the design here?
IIRC, Comcast uses "local-as 7922" path rewriting for all enterprise EDI customers running BGP off of regional CRANs, unless I'm mistaken. So if your regional CRAN is as7015 (for New England), even if your physical circuit and BGP session land on a local SUR in your area, you have to use remote-as 7922 to setup the session with Comcast ENS, regardless of the fact that actual peer's router is using as7015. What Level 3/CTL looking glass is showing ("33657 <his ASN>") is most likely the true path reflecting of actual network topology behind the scenes. Your friend is connected to 33567/CRAN, not 7922/ibone. James
IIRC, Comcast uses "local-as 7922" path rewriting for all enterprise EDI customers running BGP off of regional CRANs, unless I'm mistaken. So if your regional CRAN is as7015 (for New England), even if your physical circuit and BGP session land on a local SUR in your area, you have to use remote-as 7922 to setup the session with Comcast ENS, regardless of the fact that actual peer's router is using as7015.
This totally makes sense, thanks for the explanation!
participants (4)
-
James Jun -
Martijn Schmidt -
Ross Tajvar -
Will Orton