Issues with 4-octet BGP AS and Akamai?
Hello BGP and/or Akamai experts, Has anyone come across issues with using the new 4-octet BGP AS number format and reaching websites hosted by Akamai? One of my customers currently uses the AS number of one of their partner companies, which is in the standard 2-octed AS format. They were recently assigned their own AS which is in the 4-octet AS format. When they advertise their networks using the 2-octet AS number everything works fine, they can browse to all websites and all their outbound and inbound internet traffic works perfectly. However when they stop advertising their networks using the 2-octet AS and advertise their networks using the new 4-octet AS number, they lose reachability to a number of websites, all of which seem to be hosted by Akamai, including www.cisco.com<http://www.cisco.com>, www.dell.com<http://www.dell.com>, and a number of others. Websites that are not hosted by Akamai work just fine. We checked their route advertisements from several looking glass servers and the routes seem to be propagating properly, albeit there may be a number of older routers still out there that don't understand the new 4 octet BGP AS format. For example, their AS shows up as AS_TRANS 23456 on some looking glass routers. My understanding is the 4-octet ASN's have been out for quite some time now, so any interoperability issues should have been resolved by now. But could there be some old BGP speakers who don't understand the new format or the translated AS 23456 format and drop the routes? Has anyone experienced similar issues when using the new format 4 byte BGP ASN's? Any thoughts on who at Akamai we can speak to in order to troubleshoot this further? Thanks, Greg [http://www.cisco.com/web/europe/images/email/signature/logo05.jpg] Gregory Gombas CCIE# 19649 - R&S Network Consulting Engineer Advanced Services grgombas@cisco.com<mailto:grgombas@cisco.com> Office: +1-212-714-4497 Mobile: +1-201-675-9457 Cisco Systems Limited One Penn Plaza 6th & 9th Floors New York, NY 10119 United States Cisco.com [Think before you print.]Think before you print. This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message. For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/index.html
Hi, What prefix and ASN is this about? Are you sure you are advertising from an AS4 capable router? Do you see the expected 4-byte ASN as origin in a aggregator looking glass like http://lg.ring.nlnog.net/prefix_detail/lg01/ipv4?q=www.nlnog.net ? Kind regards, Job
Thank you all for your assistance thus far. I wanted to confirm with my customer that it was okay to share more details and they said it was okay. We did just send an email to Akamai net support and awaiting their reply. The customer is NYULH (AS 394666). They currently use NYU (AS 12) for internet connectivity. They advertise the following prefixes to NYU: 216.165.124.0/24 216.165.125.0/24 216.165.126.0/24 216.165.127.0/24 NYU aggregates the above prefixes, strips NYULH's AS number, and replaces it with their own AS number (AS 12). The aggregates are as follows: 216.165.124.0/23 216.165.126.0/23 Below is a sample /23 route seen from one of the looking glass servers with origin AS 12: 216.165.124.0/23 [DIGITALOCEAN3 2017-11-11 from 162.243.188.2] * (100/-) [AS12i] Type: BGP unicast univ BGP.origin: IGP BGP.as_path: 393406 3630 12 BGP.next_hop: 162.243.188.2 BGP.local_pref: 100 BGP.atomic_aggr: BGP.aggregator: 192.168.255.3 AS12 BGP.community: (14061,2000) (14061,2002) (14061,3000) (14061,3001) (65363,714) (65363,2906) (65363,13335) (65363,13414) (65363,14061) (65363,20940) (65363,32934) (65363,41690) (65363,46489) (65363,65340) BGP.ext_community: (RPKI Origin Validation State: not-found) With their routes originating from AS 12, all their internet connectivity works fine. However when they failover to their secondary path which is F5 Silverline DDOS protection over Optimimum Lightpath, they are unable to connect to any Akamai hosted websites. The difference between their primary path and secondary path is that the secondary path does not strip their origin AS 394666. To answer Job's question, yes the originating router is AS4 capable. I checked the looking glass link you provided and see the correct origin AS 394666. See below: 216.165.124.0/24 [DIGITALOCEAN5 14:14:44 from 5.101.111.2] * (100/-) [AS394666i] Type: BGP unicast univ BGP.origin: IGP BGP.as_path: 202109 2914 55002 394666 BGP.next_hop: 5.101.111.2 BGP.local_pref: 100 BGP.community: (2914,410) (2914,1203) (2914,2201) (2914,3200) (14061,2100) (14061,2101) (14061,4000) (14061,4001) BGP.ext_community: (RPKI Origin Validation State: not-found) However we noticed some of Level 3's looking glass routers only see the AS_Trans 23456 as shown in the output below. I'm assuming that means some of Level3's routers are not AS4 capable, but does that mean they will drop the routes? Report generated from: car1.jan1 Route results for 216.165.124.0/24 from Jackson, MS BGP routing table entry for 216.165.124.0/24 Paths: (2 available, best #2) 1299 55002 23456 AS-path translation: { TELIANET DEFENSENET-1 AS23456 } ear3.Dallas1 (metric 43807) Origin IGP, metric 100000, localpref 86, valid, internal Community: North_America Lclprf_86 United_States Level3_Peer Dallas Level3:10497 Originator: ear3.Dallas1 1299 55002 23456 AS-path translation: { TELIANET DEFENSENET-1 AS23456 } ear3.Dallas1 (metric 43807) Origin IGP, metric 100000, localpref 86, valid, internal, best Community: North_America Lclprf_86 United_States Level3_Peer Dallas Level3:10497 Originator: ear3.Dallas1 Thanks, Greg Gregory Gombas CCIE# 19649 - R&S Network Consulting Engineer Advanced Services grgombas@cisco.com Office: +1-212-714-4497 Mobile: +1-201-675-9457 Cisco Systems Limited One Penn Plaza 6th & 9th Floors New York, NY 10119 United States Cisco.com Think before you print. This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message. For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/index.html -----Original Message----- From: Job Snijders [mailto:job@ntt.net] Sent: Tuesday, November 14, 2017 11:36 AM To: Greg Gombas -X (grgombas) <grgombas@cisco.com> Cc: nanog@nanog.org Subject: Re: Issues with 4-octet BGP AS and Akamai? Hi, What prefix and ASN is this about? Are you sure you are advertising from an AS4 capable router? Do you see the expected 4-byte ASN as origin in a aggregator looking glass like http://lg.ring.nlnog.net/prefix_detail/lg01/ipv4?q=www.nlnog.net ? Kind regards, Job
It should be noted that AS_TRANS aka 23456 shouldn’t be visible on the global internet and many people may filter that on AS4_PATH cable devices. The fact that you’re seeing an AS_TRANS path from the Telia LG is likely an indication that route may be not fully internet visible. It’s fairly suspicious that someone in 2017 is still having that issue. There’s a chance that is being filtered if someone is putting AS_TRANS in the AS4_PATH, but it does appear the route is fully visible: https://stat.ripe.net/widget/routing-history#w.resource=216.165.0.0/17&w.normalise_visibility=true Note that the routes in red have low visibility, which includes some of the CIDR blocks you specified. https://stat.ripe.net/216.165.124.0%2F23#tabId=routing - Jared
On Nov 14, 2017, at 12:36 PM, Greg Gombas -X (grgombas) <grgombas@cisco.com> wrote:
Thank you all for your assistance thus far. I wanted to confirm with my customer that it was okay to share more details and they said it was okay. We did just send an email to Akamai net support and awaiting their reply.
The customer is NYULH (AS 394666). They currently use NYU (AS 12) for internet connectivity. They advertise the following prefixes to NYU:
216.165.124.0/24 216.165.125.0/24 216.165.126.0/24 216.165.127.0/24
NYU aggregates the above prefixes, strips NYULH's AS number, and replaces it with their own AS number (AS 12). The aggregates are as follows:
216.165.124.0/23 216.165.126.0/23
Below is a sample /23 route seen from one of the looking glass servers with origin AS 12:
216.165.124.0/23 [DIGITALOCEAN3 2017-11-11 from 162.243.188.2] * (100/-) [AS12i] Type: BGP unicast univ BGP.origin: IGP BGP.as_path: 393406 3630 12 BGP.next_hop: 162.243.188.2 BGP.local_pref: 100 BGP.atomic_aggr: BGP.aggregator: 192.168.255.3 AS12 BGP.community: (14061,2000) (14061,2002) (14061,3000) (14061,3001) (65363,714) (65363,2906) (65363,13335) (65363,13414) (65363,14061) (65363,20940) (65363,32934) (65363,41690) (65363,46489) (65363,65340) BGP.ext_community: (RPKI Origin Validation State: not-found)
With their routes originating from AS 12, all their internet connectivity works fine.
However when they failover to their secondary path which is F5 Silverline DDOS protection over Optimimum Lightpath, they are unable to connect to any Akamai hosted websites. The difference between their primary path and secondary path is that the secondary path does not strip their origin AS 394666.
To answer Job's question, yes the originating router is AS4 capable. I checked the looking glass link you provided and see the correct origin AS 394666. See below:
216.165.124.0/24 [DIGITALOCEAN5 14:14:44 from 5.101.111.2] * (100/-) [AS394666i] Type: BGP unicast univ BGP.origin: IGP BGP.as_path: 202109 2914 55002 394666 BGP.next_hop: 5.101.111.2 BGP.local_pref: 100 BGP.community: (2914,410) (2914,1203) (2914,2201) (2914,3200) (14061,2100) (14061,2101) (14061,4000) (14061,4001) BGP.ext_community: (RPKI Origin Validation State: not-found)
However we noticed some of Level 3's looking glass routers only see the AS_Trans 23456 as shown in the output below. I'm assuming that means some of Level3's routers are not AS4 capable, but does that mean they will drop the routes?
Report generated from: car1.jan1
Route results for 216.165.124.0/24 from Jackson, MS
BGP routing table entry for 216.165.124.0/24 Paths: (2 available, best #2) 1299 55002 23456 AS-path translation: { TELIANET DEFENSENET-1 AS23456 } ear3.Dallas1 (metric 43807) Origin IGP, metric 100000, localpref 86, valid, internal Community: North_America Lclprf_86 United_States Level3_Peer Dallas Level3:10497 Originator: ear3.Dallas1 1299 55002 23456 AS-path translation: { TELIANET DEFENSENET-1 AS23456 } ear3.Dallas1 (metric 43807) Origin IGP, metric 100000, localpref 86, valid, internal, best Community: North_America Lclprf_86 United_States Level3_Peer Dallas Level3:10497 Originator: ear3.Dallas1
Thanks, Greg
Gregory Gombas CCIE# 19649 - R&S Network Consulting Engineer Advanced Services grgombas@cisco.com Office: +1-212-714-4497 Mobile: +1-201-675-9457 Cisco Systems Limited One Penn Plaza 6th & 9th Floors New York, NY 10119 United States Cisco.com
Think before you print. This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message. For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/index.html
-----Original Message----- From: Job Snijders [mailto:job@ntt.net] Sent: Tuesday, November 14, 2017 11:36 AM To: Greg Gombas -X (grgombas) <grgombas@cisco.com> Cc: nanog@nanog.org Subject: Re: Issues with 4-octet BGP AS and Akamai?
Hi,
What prefix and ASN is this about?
Are you sure you are advertising from an AS4 capable router?
Do you see the expected 4-byte ASN as origin in a aggregator looking glass like http://lg.ring.nlnog.net/prefix_detail/lg01/ipv4?q=www.nlnog.net ?
Kind regards,
Job
Can you share details? Did you contact akamai? Feel free to ping me offline. - Jared
On Nov 13, 2017, at 9:36 PM, Greg Gombas -X (grgombas) <grgombas@cisco.com> wrote:
Hello BGP and/or Akamai experts,
Has anyone come across issues with using the new 4-octet BGP AS number format and reaching websites hosted by Akamai?
One of my customers currently uses the AS number of one of their partner companies, which is in the standard 2-octed AS format. They were recently assigned their own AS which is in the 4-octet AS format.
When they advertise their networks using the 2-octet AS number everything works fine, they can browse to all websites and all their outbound and inbound internet traffic works perfectly.
However when they stop advertising their networks using the 2-octet AS and advertise their networks using the new 4-octet AS number, they lose reachability to a number of websites, all of which seem to be hosted by Akamai, including www.cisco.com<http://www.cisco.com>, www.dell.com<http://www.dell.com>, and a number of others. Websites that are not hosted by Akamai work just fine.
We checked their route advertisements from several looking glass servers and the routes seem to be propagating properly, albeit there may be a number of older routers still out there that don't understand the new 4 octet BGP AS format. For example, their AS shows up as AS_TRANS 23456 on some looking glass routers.
My understanding is the 4-octet ASN's have been out for quite some time now, so any interoperability issues should have been resolved by now. But could there be some old BGP speakers who don't understand the new format or the translated AS 23456 format and drop the routes?
Has anyone experienced similar issues when using the new format 4 byte BGP ASN's?
Any thoughts on who at Akamai we can speak to in order to troubleshoot this further?
Thanks, Greg
[http://www.cisco.com/web/europe/images/email/signature/logo05.jpg]
Gregory Gombas CCIE# 19649 - R&S Network Consulting Engineer Advanced Services grgombas@cisco.com<mailto:grgombas@cisco.com> Office: +1-212-714-4497 Mobile: +1-201-675-9457
Cisco Systems Limited One Penn Plaza 6th & 9th Floors New York, NY 10119 United States Cisco.com
[Think before you print.]Think before you print.
This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message. For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/index.html
Greg, I have a 4 byte ASN and have not had any issues with reach ability, including the 2 websites you have linked. James
Are you advertising out multiple circuits? Check the pathing both directions if you can. A lot of CDNs enforce uRPF strict. On Tuesday, November 14, 2017, james machado <hvgeekwtrvl@gmail.com> wrote:
Greg,
I have a 4 byte ASN and have not had any issues with reach ability, including the 2 websites you have linked.
James
Hi Tyler, Unfortunately we had a limited window to test so could not check the reverse path. During our failover testing we stopped advertising out the primary path and only advertised out the secondary path. Routes are advertised out the secondary path through a DDOS prevention company called F5 Silverline which is reached via a GRE tunnel running over the Optimum Lightpath network. So outgoing traffic would go from NYULH going out the Optimum Lightpath circuit and return traffic coming in on F5 Silverline’s network then tunneled over Optimum Lightpath back to NYULH. So traffic was definitely routing asymmetrically. However F5 Silverline assured us they have many customers using a similar setup but have no issues with Akamai. I would think that many customers using similar DDOS prevention services such as F5 Silverline and Prolexic are routing asymmetrically as well, wouldn’t uRPF be affecting them all? Thanks, Greg [http://www.cisco.com/web/europe/images/email/signature/logo05.jpg] Gregory Gombas CCIE# 19649 – R&S Network Consulting Engineer Advanced Services grgombas@cisco.com<mailto:grgombas@cisco.com> Office: +1-212-714-4497 Mobile: +1-201-675-9457 Cisco Systems Limited One Penn Plaza 6th & 9th Floors New York, NY 10119 United States Cisco.com [Think before you print.]Think before you print. This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message. For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/index.html From: Tyler Conrad [mailto:tyler@tgconrad.com] Sent: Tuesday, November 14, 2017 1:30 PM To: james machado <hvgeekwtrvl@gmail.com> Cc: Greg Gombas -X (grgombas) <grgombas@cisco.com>; nanog@nanog.org Subject: Re: Issues with 4-octet BGP AS and Akamai? Are you advertising out multiple circuits? Check the pathing both directions if you can. A lot of CDNs enforce uRPF strict. On Tuesday, November 14, 2017, james machado <hvgeekwtrvl@gmail.com<mailto:hvgeekwtrvl@gmail.com>> wrote: Greg, I have a 4 byte ASN and have not had any issues with reach ability, including the 2 websites you have linked. James
Greg, I don't see a routing database object for your routes pointing too your AS394666 /24's, I only see one for AS12 for the /23 and /24's. It is possible (and probable) you are being filtered due to that. james route: 216.165.124.0/23 descr: NEW YORK UNIVERSITY (added by MAINT-AS6517) origin: AS12 remarks: This route object was registered by Global Cloud Xchange MAINT-AS6517 on behalf of their customer: NEW YORK UNIVERSITY notify: support@relianceglobalcom.com mnt-by: MAINT-AS6517 changed: support@globalcloudxchange.com 20160506 #00:49:14Z source: RADB (125-127) route: 216.165.127.0/24 descr: New York University Medical Center (maintained by NYU NOC) origin: AS12 mnt-by: MAINT-AS12 changed: noc@nyu.edu 20121121 #16:23:31Z source: RADB
Hi James, On Wed, Nov 15, 2017 at 1:40 AM, james machado <hvgeekwtrvl@gmail.com> wrote:
I don't see a routing database object for your routes pointing too your AS394666 /24's, I only see one for AS12 for the /23 and /24's. It is possible (and probable) you are being filtered due to that.
This is a really good observation, and a likely explanation! @ OP - during IP space migrations from AS A to AS B you should ensure that route objects exist for both ASNs. You may also want to double check with your upstream providers what their AS path filters look like for your circuits. Kind regards, Job
About who to speak with at Akamai... please forgive me if any of this contact info is out-of-date, as I'm pulling from my notes from an old network diagram... Akamai Customer Care - 877-425-2832 Akamai NOCC - 877-625-2624 - 877-6-akamai (same as above) - 617-444-3007 - nocc-shift@akamai.com - (if you do anything that would your cluster, give them at least 3 hours notice and give them IP of cluster - hardware issues and 24x7 contact: nocc-tix@akamai.com +1-877-6AKAMAI Akamai Network Support - traffic issues: netsupport-tix@akamai.com +1-888-421-1003 -Aaron
participants (7)
-
Aaron Gould -
Greg Gombas -X (grgombas) -
james machado -
Jared Mauch -
Job Snijders -
Job Snijders -
Tyler Conrad