Traffic engineering and peering for CDNs
Lately I have been putting in some effort to maximize our IX connections by trying to work with the top 5-ish list of ASNs that still send us traffic via a paid transit connection despite the fact that we are both present on the same IX(s). In one case I missed the fact that one ASN wasn't using the IXs route-servers, that's on me for not spotting that one. Even with proper IX peering in place though it seems like some CDNs are better at using the IX connections than others. ASN 15169 for instance does an excellent job sending more than 99.99% of traffic via the IX connection; thank you. While others only seem to manage to send 60 - 80% of traffic via the IX. What I am not understanding about the respective CDN's network wherein they don't send traffic to me through a consistent path? Is the content coming from widely different places and rather than transport it across their own network from a remote site they would rather hot-potato it out a local transit connection? Are their transit costs so low that they don't care about using an IX connection over transit unlike a small operator like me? Is this just a non-obvious issue wherein they maybe just can't originate enough of the traffic near the IX and therefore don't make use of the IX connection, again a hot-potato phenomenon? Secondly can someone explain to me why some CDNs want a gigabit or two of traffic to be exchanged between our respective networks before they would peer with me via a public IX? I totally get those kinds of thresholds before engaging in a private interconnect but I don't understand the reluctance with regard to a public IX, that they are already established at. Is it again just a simple case of bandwidth economics that operate at a different scale than I can comprehend? I'm hoping the community can shed some light on this for me as I'm trying to avoid grilling the operators that are working with me as I don't expect those front line individuals to necessarily have a full view of the factors at play. Thanks, Graham Johnston Network Planner Westman Communications Group 204.717.2829 johnstong@westmancom.com<mailto:johnstong@westmancom.com> P think green; don't print this email.
On Mon, 6 Jun 2016, Graham Johnston wrote:
What I am not understanding about the respective CDN's network wherein they don't send traffic to me through a consistent path? Is the content coming from widely different places and rather than transport it across their own network from a remote site they would rather hot-potato it out a local transit connection?
Depends on the CDN, but its possible the traffic is coming from different locations and not all CDNs even have a network, so if you don't have peering with their location serving the traffic (and they don't have a network), the traffic will have to come to you via other paths.
Are their transit costs so low that they don't care about using an IX connection over transit unlike a small operator like me? Is this just a
Maybe. Or maybe the traffic to you is small enough (to them) that you're not on their radar as a desirable peer. Or maybe they just haven't gotten around to sending you a peering request yet.
Secondly can someone explain to me why some CDNs want a gigabit or two of traffic to be exchanged between our respective networks before they would peer with me via a public IX?
Which ones want that much? We like to see some traffic before moving from "IX route-server peering" to direct peering via the IX, just because there are so many possible peers and only so much router resources. It's really not worth the resources (router or management) to direct peer with a network with which there's virtually no traffic being exchanged, just because we're on the same IX(s). 1-2G to peer seems kind of high. Some might insist that you move peering to PNI if you're doing >1-2G across an IX. ---------------------------------------------------------------------- Jon Lewis, MCP :) | I route | therefore you are _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
Hello,
On Jun 6, 2016, at 7:36 AM, Graham Johnston <johnstong@westmancom.com> wrote:
Lately I have been putting in some effort to maximize our IX connections by trying to work with the top 5-ish list of ASNs that still send us traffic via a paid transit connection despite the fact that we are both present on the same IX(s). In one case I missed the fact that one ASN wasn't using the IXs route-servers, that's on me for not spotting that one.
Even with proper IX peering in place though it seems like some CDNs are better at using the IX connections than others. ASN 15169 for instance does an excellent job sending more than 99.99% of traffic via the IX connection; thank you. While others only seem to manage to send 60 - 80% of traffic via the IX. What I am not understanding about the respective CDN's network wherein they don't send traffic to me through a consistent path? Is the content coming from widely different places and rather than transport it across their own network from a remote site they would rather hot-potato it out a local transit connection? Are their transit costs so low that they don't care about using an IX connection over transit unlike a small operator like me? Is this just a non-obvious issue wherein they maybe just can't originate enough of the traffic near the IX and therefore don't make use of the IX connection, again a hot-potato phenomenon?
Most CDN’s do not have a backbone. Transit costs are not free, but as most traffic is served by local nodes from cache, the costs of transport between locations in many cases is higher than just sending via transit. In some cases, the CDN may not have good mapping and may not be certain which node is best to serve your customers. In other cases, not all content exists on all nodes, and they may redirect to serve from the nodes which have the content. Finally, there may be an outage or capacity limits from the closest location, and another location may be serving to make up the shortfall.
Secondly can someone explain to me why some CDNs want a gigabit or two of traffic to be exchanged between our respective networks before they would peer with me via a public IX? I totally get those kinds of thresholds before engaging in a private interconnect but I don't understand the reluctance with regard to a public IX, that they are already established at. Is it again just a simple case of bandwidth economics that operate at a different scale than I can comprehend?
This sounds like a surprisingly high threshold, but to some extent it boils down like this — setting up sessions requires some time. In the ideal world, the peer is intelligent and has everything set up properly, but even in this case, it still requires some time for making sure things go up properly. Some (but not all) CDN’s have it automated to reduce this time. Some potential peering networks are poorly run, and will leak routes, not announce all of their routes, will not configure the sessions properly, etc — this adds up to significantly more time. Before the CDN starts setting up peering with another network, it is not necessarily obvious if the potential peer is run by competent people or not. Many CDN’s are members of the route servers. If you are exchanging a small amount of traffic, and both you and the CDN are on the Route Server for the IX, there maybe no reason to set up direct sessions which will require both more coordination time for configuration, and more router cpu time/ram on an ongoing basis. From the perspective of the CDN, most likely, 1Gbps or less is a perfectly reasonable amount of traffic to exchange to peers who are learned only via the route server, and not directly.
I'm hoping the community can shed some light on this for me as I'm trying to avoid grilling the operators that are working with me as I don't expect those front line individuals to necessarily have a full view of the factors at play.
Thanks, Graham Johnston Network Planner Westman Communications Group 204.717.2829 johnstong@westmancom.com<mailto:johnstong@westmancom.com> P think green; don't print this email.
Best Regards, -Phil Rosenthal ISPrime
Some rely on performance testing to the client's DNS resolver and if they're not using on-net ones, they'll be directed to use a different CDN node. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "Graham Johnston" <johnstong@westmancom.com> To: "nanog@nanog.org" <nanog@nanog.org> Sent: Monday, June 6, 2016 8:36:43 AM Subject: Traffic engineering and peering for CDNs Lately I have been putting in some effort to maximize our IX connections by trying to work with the top 5-ish list of ASNs that still send us traffic via a paid transit connection despite the fact that we are both present on the same IX(s). In one case I missed the fact that one ASN wasn't using the IXs route-servers, that's on me for not spotting that one. Even with proper IX peering in place though it seems like some CDNs are better at using the IX connections than others. ASN 15169 for instance does an excellent job sending more than 99.99% of traffic via the IX connection; thank you. While others only seem to manage to send 60 - 80% of traffic via the IX. What I am not understanding about the respective CDN's network wherein they don't send traffic to me through a consistent path? Is the content coming from widely different places and rather than transport it across their own network from a remote site they would rather hot-potato it out a local transit connection? Are their transit costs so low that they don't care about using an IX connection over transit unlike a small operator like me? Is this just a non-obvious issue wherein they maybe just can't originate enough of the traffic near the IX and therefore don't make use of the IX connection, again a hot-potato phenomenon? Secondly can someone explain to me why some CDNs want a gigabit or two of traffic to be exchanged between our respective networks before they would peer with me via a public IX? I totally get those kinds of thresholds before engaging in a private interconnect but I don't understand the reluctance with regard to a public IX, that they are already established at. Is it again just a simple case of bandwidth economics that operate at a different scale than I can comprehend? I'm hoping the community can shed some light on this for me as I'm trying to avoid grilling the operators that are working with me as I don't expect those front line individuals to necessarily have a full view of the factors at play. Thanks, Graham Johnston Network Planner Westman Communications Group 204.717.2829 johnstong@westmancom.com<mailto:johnstong@westmancom.com> P think green; don't print this email.
as far as im aware ... a friend of mine on INEX in Ireland said most cdns use source ip of the DNS requests to determine which network to direct them to ... so if you use you have your own resolver on an ip address in your network range cdns can accurately determine what network the request is comming from and determine what ip address / what network that the cdn has nearest to your network... ff you use 3rd party dns servers for your clients... you may not get an optimal ip answer for your dns queries from the CDNS involved I hope this helps Tom Smyth On Mon, Jun 6, 2016 at 6:53 PM, Mike Hammett <nanog@ics-il.net> wrote:
Some rely on performance testing to the client's DNS resolver and if they're not using on-net ones, they'll be directed to use a different CDN node.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest Internet Exchange http://www.midwest-ix.com
----- Original Message -----
From: "Graham Johnston" <johnstong@westmancom.com> To: "nanog@nanog.org" <nanog@nanog.org> Sent: Monday, June 6, 2016 8:36:43 AM Subject: Traffic engineering and peering for CDNs
Lately I have been putting in some effort to maximize our IX connections by trying to work with the top 5-ish list of ASNs that still send us traffic via a paid transit connection despite the fact that we are both present on the same IX(s). In one case I missed the fact that one ASN wasn't using the IXs route-servers, that's on me for not spotting that one.
Even with proper IX peering in place though it seems like some CDNs are better at using the IX connections than others. ASN 15169 for instance does an excellent job sending more than 99.99% of traffic via the IX connection; thank you. While others only seem to manage to send 60 - 80% of traffic via the IX. What I am not understanding about the respective CDN's network wherein they don't send traffic to me through a consistent path? Is the content coming from widely different places and rather than transport it across their own network from a remote site they would rather hot-potato it out a local transit connection? Are their transit costs so low that they don't care about using an IX connection over transit unlike a small operator like me? Is this just a non-obvious issue wherein they maybe just can't originate enough of the traffic near the IX and therefore don't make use of the IX connection, again a hot-potato phenomenon?
Secondly can someone explain to me why some CDNs want a gigabit or two of traffic to be exchanged between our respective networks before they would peer with me via a public IX? I totally get those kinds of thresholds before engaging in a private interconnect but I don't understand the reluctance with regard to a public IX, that they are already established at. Is it again just a simple case of bandwidth economics that operate at a different scale than I can comprehend?
I'm hoping the community can shed some light on this for me as I'm trying to avoid grilling the operators that are working with me as I don't expect those front line individuals to necessarily have a full view of the factors at play.
Thanks, Graham Johnston Network Planner Westman Communications Group 204.717.2829 johnstong@westmancom.com<mailto:johnstong@westmancom.com> P think green; don't print this email.
-- Kindest regards, Tom Smyth Mobile: +353 87 6193172 --------------------------------- PLEASE CONSIDER THE ENVIRONMENT BEFORE YOU PRINT THIS E-MAIL This email contains information which may be confidential or privileged. The information is intended solely for the use of the individual or entity named above. If you are not the intended recipient, be aware that any disclosure, copying, distribution or use of the contents of this information is prohibited. If you have received this electronic transmission in error, please notify me by telephone or by electronic mail immediately. Any opinions expressed are those of the author, not the company's .This email does not constitute either offer or acceptance of any contractually binding agreement. Such offer or acceptance must be communicated in writing. You are requested to carry out your own virus check before opening any attachment. Thomas Smyth accepts no liability for any loss or damage which may be caused by malicious software or attachments.
On 6/Jun/16 20:03, Tom Smyth wrote:
as far as im aware ... a friend of mine on INEX in Ireland said most cdns use source ip of the DNS requests to determine which network to direct them to ... so if you use you have your own resolver on an ip address in your network range cdns can accurately determine what network the request is comming from and determine what ip address / what network that the cdn has nearest to your network...
ff you use 3rd party dns servers for your clients... you may not get an optimal ip answer for your dns queries from the CDNS involved
Some CDN's use DNS (in addition to latency, congestion levels, busy state, e.t.c.). Others use Anycast routing, which I tend to prefer. The problem is the latter run a network while the former may typically not. Mark.
On 6/7/16, 2:46 AM, "NANOG on behalf of Mark Tinka" <nanog-bounces@nanog.org on behalf of mark.tinka@seacom.mu> wrote:
On 6/Jun/16 20:03, Tom Smyth wrote:
as far as im aware ... a friend of mine on INEX in Ireland said most cdns use source ip of the DNS requests to determine which network to direct them to ... so if you use you have your own resolver on an ip address in your network range cdns can accurately determine what network the request is comming from and determine what ip address / what network that the cdn has nearest to your network...
ff you use 3rd party dns servers for your clients... you may not get an optimal ip answer for your dns queries from the CDNS involved
Some CDN's use DNS (in addition to latency, congestion levels, busy state, e.t.c.).
Others use Anycast routing, which I tend to prefer. The problem is the latter run a network while the former may typically not.
Mark.
Also some companies make layer 7 decisions for their CDN¹s in conjunction with these other methods. Their applications makes a decision on what host to send you to based on routing information, your source address, and other accumulated data.
Hi Graham! In addition to the other two comments, I´d like to add some topics:
Lately I have been putting in some effort to maximize our IX connections by trying to work with the top 5-ish list of ASNs that still send us traffic via a paid transit connection despite the fact that we are both present on the same IX(s). In one case I missed the fact that one ASN wasn't using the IXs route-servers, that's on me for not spotting that one.
This brings up some ideas ... see here: 1) Check if the CDN is on the routeserver 2) Check if the CDN has maybe tagged his prefixes with a "do-not-announce" tag (verify at looking glass) (relevant of course only for outbound traffic) 3) Try to establish a direct peering session with the CDN over the IXP so that you are known to the CDN ... some CDN´s could maybe also prefer or have higher priority on direct sessions than via only routeserver... ... quite some networks give you a larger set of prefixes on a direct session... 4) Talk with the CDN and check his geolocation tagging's for your prefixes and maybe let correct them after you have found out what they are doing 5) Think on the fact, that CDN´s could take their routing decision on the geo-location of the used dns resolver server and not on the users ip address 6) Check, that your network or your customers are not referring to public dns resolvers 7) Think on the fact, that ipv6 could maybe be in place and active too ... so don´t forget your ipv6 path and ipv6 dns resolvers... 8) Check your RADB/RIPE/AFRINIC/... routing db entries - if you did something wrong, maybe you are filtered... (check: http://irrexplorer.nlnog.net/ and looking glasses) 9) Check that you have your set of IP prefixes in good order and do not implement strange magic with asymmetric more specifics ... reduce everything to supernetworks on all edges! 10) Check if you are not handing over bgp tagging's which could cause some prevention automatics and check if you do not send high metric values which are maybe causing negative routing decisions on the other side 11) Think on the fact, that CDN´s have the geographic distance as one of their parameters - if the CDN node is too far away from your network - maybe a closer located content box via transit is rated higher than maybe the long distance via peering (if there is no node or no node with the right content next to you) ... typically CDN are very close to larger IXP´s... 12) Think on the fact, that your ip-transit upstream could be a peering for the CDN ... This neutralizes the peering or transit rating from the point of view of the CDN, but of course not for you ... so: talk with the CDN ... therefore you must be known ... therefore, you should have done a peering session ... 13) Talk with your IXP ... maybe he can help ... Anything missing here? Anything wrong with my ideas? (disclaimer: yes, I do work for an IXP ... but this is my personal opinion) happy peering... Bernd Bernd Spiess (Manager Peering Service / DE-CIX Management GmbH)
participants (8)
-
Bernd Spiess -
Graham Johnston -
Jon Lewis -
Mark Tinka -
Mike Hammett -
Phil Rosenthal -
Tom Smyth -
Wolff, Nick