Hello, I'm trying to find out the best way to consolidate connectivity on an island. The current issues are : - Low redundancy of old cables (2) - Low system capacity of said cables (<=20Gbps) - Total service loss when both cables are down because of congestion on satelite backups - Sheer price of bandwidth On the plus side, there are over 5 AS on the island, an IXP and small-ish collocation capacity (approx 10kW available, could be upgraded, second site available later this year). We'd like to host cache servers and/or VMs on the IXP, with an option to anycast many services - without hijacking them, that goes without saying - such as quad-whatever DNS resolvers, NTP servers and whatever else could be useful for weather-induced disaster-recovery and/or offload cables. Do you think most CDNs, stream services and CSPs could accommodate a scenario where we'd host their gear or provide VMs for them to announce on the local route-servers ? If not, what could be a reasonable technical arrangement ? Thanks ! -- Jérôme Nicolle +33 6 19 31 27 14
Hi Jérôme VMs are no go for big content companies except Microsoft. You can run Microsoft CDN on VM but rest of the content will need to be cached. You can actually install this yourself Depending on how much traffic do you have , you may be able to get facebook, youtube, netflix caches i think minimum bw requirement changes per region Good luck On Thu, Jan 18, 2024 at 06:53 Jérôme Nicolle <jerome@ceriz.fr> wrote:
Hello,
I'm trying to find out the best way to consolidate connectivity on an island.
The current issues are : - Low redundancy of old cables (2) - Low system capacity of said cables (<=20Gbps) - Total service loss when both cables are down because of congestion on satelite backups - Sheer price of bandwidth
On the plus side, there are over 5 AS on the island, an IXP and small-ish collocation capacity (approx 10kW available, could be upgraded, second site available later this year).
We'd like to host cache servers and/or VMs on the IXP, with an option to anycast many services - without hijacking them, that goes without saying - such as quad-whatever DNS resolvers, NTP servers and whatever else could be useful for weather-induced disaster-recovery and/or offload cables.
Do you think most CDNs, stream services and CSPs could accommodate a scenario where we'd host their gear or provide VMs for them to announce on the local route-servers ? If not, what could be a reasonable technical arrangement ?
Thanks !
-- Jérôme Nicolle +33 6 19 31 27 14
Hello Mehmet, Le 18/01/2024 à 12:58, Mehmet a écrit :
VMs are no go for big content companies except Microsoft. You can run Microsoft CDN on VM but rest of the content will need to be cached. You can actually install this yourself
I've already read most docs for caching servers provided by major actors. What I'm mostly concerned about is their ability to peer with multiple AS on the local IXP, as to not over-replicate them. Should I establish a dedicated network peering on the IXP ? Or will they come with their own ASNs ? The peering case is quite not documented on publicly available specs, if even possible.
Depending on how much traffic do you have , you may be able to get facebook, youtube, netflix caches i think minimum bw requirement changes per region
Those I'm nearly sure I could get, if I can pool caches amongst ISPs. The current constraints are issues to any content provider, not just for local ISPs. Best regards, -- Jérôme Nicolle +33 6 19 31 27 14
Some will work directly on the IX via BGP. Others have to go behind a member of the IX. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Jérôme Nicolle" <jerome@ceriz.fr> To: "Mehmet" <mehmet@akcin.net> Cc: nanog@nanog.org Sent: Thursday, January 18, 2024 8:38:31 AM Subject: Re: Shared cache servers on an island's IXP Hello Mehmet, Le 18/01/2024 à 12:58, Mehmet a écrit :
VMs are no go for big content companies except Microsoft. You can run Microsoft CDN on VM but rest of the content will need to be cached. You can actually install this yourself
I've already read most docs for caching servers provided by major actors. What I'm mostly concerned about is their ability to peer with multiple AS on the local IXP, as to not over-replicate them. Should I establish a dedicated network peering on the IXP ? Or will they come with their own ASNs ? The peering case is quite not documented on publicly available specs, if even possible.
Depending on how much traffic do you have , you may be able to get facebook, youtube, netflix caches i think minimum bw requirement changes per region
Those I'm nearly sure I could get, if I can pool caches amongst ISPs. The current constraints are issues to any content provider, not just for local ISPs. Best regards, -- Jérôme Nicolle +33 6 19 31 27 14
Jérôme Nicolle wrote on 18/01/2024 14:38:
Those I'm nearly sure I could get, if I can pool caches amongst ISPs. The current constraints are issues to any content provider, not just for local ISPs.
two issues here: the smaller issue is that CDNs sometimes want their own routable IP address blocks, especially if they're connecting directly to the IXP, which usually means /24 in practice. It doesn't always happen, and sometimes the CDN is happy to use provider address space (i.e. IXP), or smaller address blocks. But it's something to note. The bigger issue is: who pays the transit costs for the CDN's cache-fill requirements? CDNs typically won't pay for cache-fill for installations like this, and if one local ISP is pulling disproportionate quantities of data compared to other ISPs at the IXP, then this can cause problems unless there's an shared billing mechanism built in. Nick
Hi Nick, Thanks for your remarks. It's actually an ongoing discussion. Le 18/01/2024 à 18:24, Nick Hilliard a écrit :
two issues here: the smaller issue is that CDNs sometimes want their own routable IP address blocks, especially if they're connecting directly to the IXP, which usually means /24 in practice. It doesn't always happen, and sometimes the CDN is happy to use provider address space (i.e. IXP), or smaller address blocks. But it's something to note.
I'd rather have CDN use some of their anycast /24 to peer with the IX, with a back-end connectivity for their control-plane and back-feeding.
The bigger issue is: who pays the transit costs for the CDN's cache-fill requirements? CDNs typically won't pay for cache-fill for installations like this, and if one local ISP is pulling disproportionate quantities of data compared to other ISPs at the IXP, then this can cause problems unless there's an shared billing mechanism built in.
We're willing to provide a dedicated LAN, with routed access, to fill caches and administer the machines. It would be fully dissociated from the IXP though, unless we could find a way to make it work and as to meet extra requirements upon redundancy. Best regards, -- Jérôme Nicolle +33 6 19 31 27 14
On Thu, Jan 18, 2024 at 12:53:19PM +0100, Jérôme Nicolle <jerome@ceriz.fr> wrote a message of 36 lines which said:
- Low redundancy of old cables (2) - Total service loss when both cables are down because of congestion on satelite backups
A problem which is not often mentioned is that most (all?) "local caches" (CDN, DNS resolvers, etc) do not have an "offline mode" (or "disconnected-from-master mode"). During an outage, they continue to work for some time then break suddenly, in a not-friendly way, serving various error messages instead of old data and/or useful messages. (For instance, the DNS resolver may not be able to serve stale answers.) The time during which they can continue to work when they are disconnected from their master is typically undocumented (except for the DNS), and discovered only when there is a long outage. Making the Internet work better with sometimes-broken connectivity is still an area of research.
Many CDNs have hardware options for self hosted caches. I think there would likely be concerns about <20G of connectivity to those caches though. With an incorrect setup, you could end up maxing out those links just with cache fill traffic itself. On Thu, Jan 18, 2024 at 6:54 AM Jérôme Nicolle <jerome@ceriz.fr> wrote:
Hello,
I'm trying to find out the best way to consolidate connectivity on an island.
The current issues are : - Low redundancy of old cables (2) - Low system capacity of said cables (<=20Gbps) - Total service loss when both cables are down because of congestion on satelite backups - Sheer price of bandwidth
On the plus side, there are over 5 AS on the island, an IXP and small-ish collocation capacity (approx 10kW available, could be upgraded, second site available later this year).
We'd like to host cache servers and/or VMs on the IXP, with an option to anycast many services - without hijacking them, that goes without saying - such as quad-whatever DNS resolvers, NTP servers and whatever else could be useful for weather-induced disaster-recovery and/or offload cables.
Do you think most CDNs, stream services and CSPs could accommodate a scenario where we'd host their gear or provide VMs for them to announce on the local route-servers ? If not, what could be a reasonable technical arrangement ?
Thanks !
-- Jérôme Nicolle +33 6 19 31 27 14
Hi Tom, Le 18/01/2024 à 15:20, Tom Beecher a écrit :
Many CDNs have hardware options for self hosted caches. I think there would likely be concerns about <20G of connectivity to those caches though. With an incorrect setup, you could end up maxing out those links just with cache fill traffic itself.
In a case where these servers are on a dedicated network peering with the ISPs, I think it would be safe to throttle the sync feeds to not saturate actual uplinks. At least, that we can do, but throttling uncached content to customers is forbidden (net neutrality). Though Netflix is supposedly sending pre-loaded servers, and I think that - in this location - it's gonna mean a lot already. The quastion is : how would the servers peer with local ISPs. Best regards, -- Jérôme Nicolle +33 6 19 31 27 14
participants (6)
-
Jérôme Nicolle -
Mehmet -
Mike Hammett -
Nick Hilliard -
Stephane Bortzmeyer -
Tom Beecher