That was a lot of traffic coming out of akamai aanp clusters the last couple nights! What was it? Aaron aaron1@gvtc.com
Peace, On Thu, Apr 1, 2021, 6:09 PM <aaron1@gvtc.com> wrote:
That was a lot of traffic coming out of akamai aanp clusters the last couple nights! What was it?
"Call of Duty" update again, obviously. https://www.eurogamer.net/articles/2021-03-29-this-weeks-call-of-duty-warzon... -- Töma
On Thu, Apr 1, 2021 at 11:16 AM Töma Gavrichenkov <ximaera@gmail.com> wrote:
Peace,
On Thu, Apr 1, 2021, 6:09 PM <aaron1@gvtc.com> wrote:
That was a lot of traffic coming out of akamai aanp clusters the last couple nights! What was it?
"Call of Duty" update again, obviously.
https://www.eurogamer.net/articles/2021-03-29-this-weeks-call-of-duty-warzon...
-- Töma
HAHA
On Apr 1, 2021, at 11:15 AM, Töma Gavrichenkov <ximaera@gmail.com> wrote:
Peace,
On Thu, Apr 1, 2021, 6:09 PM <aaron1@gvtc.com> wrote: That was a lot of traffic coming out of akamai aanp clusters the last couple nights! What was it?
"Call of Duty" update again, obviously.
https://www.eurogamer.net/articles/2021-03-29-this-weeks-call-of-duty-warzon...
Yes, we have had a number of big customer traffic in recent days. Hopefully traffic is flowing well for many networks. If this is negatively impacting you, please reach out. - Jared
Gaming update... I had a feeling. Thanks for the feedback folks. Thanks Jared, it's running well, before, during and after. We have a lot of capacity there. -Aaron
I remembered working for a big ISP in Europe offering cable tv + internet with +20M subscribers Every time there was a huge power outage in major cities, all tv`s would go off at the same time. I don`t have stats on power grid stability in Europe Vs N/A. The problem, was when the power was coming back in big cities, all the tv subscribers would come back online at the exact same second or minute. More or less the same 2 or 3 minutes. What happened is that it would create a kind of internal DDoS and they would all timed out and give a weird error message. Something very useful like Error Code 0x8098808 Please call our support line at this phone number. The server sysadmins would go on a panic because all systems were overloaded. They often needed to do overtime because DB crashed, key servers there crashed, DB here crashed, whatever... there was always something crashing. This was before the cloud when you could just push a slider and have tons of VMs or containers to absorb the load in real time. (in my dream) This would every time create frustration from the clients, the help desk, the support teams and also the upper management. Every time the teams were really tired after that. It was draining juice. Anyway, after some years of talking internally (red tape), we finally managed to install a random artificial penalty in the setup boxes when they boot after a power outage. Nothing like 20 minutes, but just enough to spread the load over a longer period of time. For the end user, it went transparent for them because, if the setup box would boot in 206 seconds instead of the super aggressive 34 seconds, well it booted and they could watch tv. Vs my system is totally frozen and it`s been like that for 20 minutes with weird messages because all your systems are down and the error msg said to call the help desk. This simple change to add 3 lines of code to add a random artificial boot penalty of few seconds, completely solve the problem. This way, when a city would black out, we wouldn't be self DDoS, because the systems would slowly rampup. The setup boxes would all reboot but, wait randomly before asking for the DRM package to unlock the cable TV service and validate whether billing is right. I`m no Call of Duty expert nor Akamai, but it's been many times that I observe the same question here: What's happening? Call of Duty! Okay. Would a kind of throttle help here? An artificial roll out penalty somehow? Probably not at the ISP level, but more at the game level. Well, ISP could also have some mechanisms to reduce the impact or even Akamai could force a progressive roll out. I`m not sure that the proposed solutions could work, but it seems to impact NANOG frequently and/or at least generate a call overnight/weekend. It seems to also happens just before long holidays when operations are sometimes on reduce personnel. Are big games roll out really impacting NANOG? or it's more a: Hey I was curious what happened and I thought to ask here on NANOG? #JustCurious Jean -----Original Message----- From: NANOG <nanog-bounces+jean=ddostest.me@nanog.org> On Behalf Of aaron1@gvtc.com Sent: April 1, 2021 12:12 PM To: 'Jared Mauch' <jared@puck.nether.net>; 'Töma Gavrichenkov' <ximaera@gmail.com> Cc: 'NANOG' <nanog@nanog.org> Subject: RE: wow, lots of akamai Gaming update... I had a feeling. Thanks for the feedback folks. Thanks Jared, it's running well, before, during and after. We have a lot of capacity there. -Aaron
IOS 7 seemed to be sent to everyone at once causing large spikes along with saturating many links for smaller ISPs. I believe after that it went more to a distribution type of sorts though I could be wrong. Maybe it was that 7 was so vastly different everyone was itching to try it. Sent from my iPhone On Apr 1, 2021, at 2:02 PM, Jean St-Laurent via NANOG <nanog@nanog.org> wrote: *External Email: Use Caution* I remembered working for a big ISP in Europe offering cable tv + internet with +20M subscribers Every time there was a huge power outage in major cities, all tv`s would go off at the same time. I don`t have stats on power grid stability in Europe Vs N/A. The problem, was when the power was coming back in big cities, all the tv subscribers would come back online at the exact same second or minute. More or less the same 2 or 3 minutes. What happened is that it would create a kind of internal DDoS and they would all timed out and give a weird error message. Something very useful like Error Code 0x8098808 Please call our support line at this phone number. The server sysadmins would go on a panic because all systems were overloaded. They often needed to do overtime because DB crashed, key servers there crashed, DB here crashed, whatever... there was always something crashing. This was before the cloud when you could just push a slider and have tons of VMs or containers to absorb the load in real time. (in my dream) This would every time create frustration from the clients, the help desk, the support teams and also the upper management. Every time the teams were really tired after that. It was draining juice. Anyway, after some years of talking internally (red tape), we finally managed to install a random artificial penalty in the setup boxes when they boot after a power outage. Nothing like 20 minutes, but just enough to spread the load over a longer period of time. For the end user, it went transparent for them because, if the setup box would boot in 206 seconds instead of the super aggressive 34 seconds, well it booted and they could watch tv. Vs my system is totally frozen and it`s been like that for 20 minutes with weird messages because all your systems are down and the error msg said to call the help desk. This simple change to add 3 lines of code to add a random artificial boot penalty of few seconds, completely solve the problem. This way, when a city would black out, we wouldn't be self DDoS, because the systems would slowly rampup. The setup boxes would all reboot but, wait randomly before asking for the DRM package to unlock the cable TV service and validate whether billing is right. I`m no Call of Duty expert nor Akamai, but it's been many times that I observe the same question here: What's happening? Call of Duty! Okay. Would a kind of throttle help here? An artificial roll out penalty somehow? Probably not at the ISP level, but more at the game level. Well, ISP could also have some mechanisms to reduce the impact or even Akamai could force a progressive roll out. I`m not sure that the proposed solutions could work, but it seems to impact NANOG frequently and/or at least generate a call overnight/weekend. It seems to also happens just before long holidays when operations are sometimes on reduce personnel. Are big games roll out really impacting NANOG? or it's more a: Hey I was curious what happened and I thought to ask here on NANOG? #JustCurious Jean -----Original Message----- From: NANOG <nanog-bounces+jean=ddostest.me@nanog.org> On Behalf Of aaron1@gvtc.com Sent: April 1, 2021 12:12 PM To: 'Jared Mauch' <jared@puck.nether.net>; 'Töma Gavrichenkov' <ximaera@gmail.com> Cc: 'NANOG' <nanog@nanog.org> Subject: RE: wow, lots of akamai Gaming update... I had a feeling. Thanks for the feedback folks. Thanks Jared, it's running well, before, during and after. We have a lot of capacity there. -Aaron
* nanog@nanog.org (Jean St-Laurent via NANOG) [Thu 01 Apr 2021, 21:03 CEST]:
An artificial roll out penalty somehow? Probably not at the ISP level, but more at the game level. Well, ISP could also have some mechanisms to reduce the impact or even Akamai could force a progressive roll out.
It's an online game. You can't play the game with outdated assets. You'd not see walls where other players would, for example. What you're suggesting is the ability of ISPs to market Internet access at a certain speed but not have to deliver it based on conditions they create. -- Niels.
On Thu, Apr 1, 2021 at 12:23 Niels Bakker <niels=nanog@bakker.net> wrote:
* nanog@nanog.org (Jean St-Laurent via NANOG) [Thu 01 Apr 2021, 21:03 CEST]:
An artificial roll out penalty somehow? Probably not at the ISP level, but more at the game level. Well, ISP could also have some mechanisms to reduce the impact or even Akamai could force a progressive roll out.
It's an online game. You can't play the game with outdated assets. You'd not see walls where other players would, for example.
What you're suggesting is the ability of ISPs to market Internet access at a certain speed but not have to deliver it based on conditions they create.
It’s actually worst. You can’t even login without having latest version to play multiplayer.
-- Niels.
-- Mehmet +1-424-298-1903
It’s in fact better, because the one with the new asset can actually play and not be totally frozen at loading and/or suffering lag and being kill in action. Progressive rolls out is the key to happiness From: NANOG <nanog-bounces+jean=ddostest.me@nanog.org> On Behalf Of Mehmet Akcin Sent: April 1, 2021 3:25 PM To: nanog@nanog.org Subject: Re: wow, lots of akamai On Thu, Apr 1, 2021 at 12:23 Niels Bakker <niels=nanog@bakker.net <mailto:nanog@bakker.net> > wrote: * nanog@nanog.org <mailto:nanog@nanog.org> (Jean St-Laurent via NANOG) [Thu 01 Apr 2021, 21:03 CEST]:
An artificial roll out penalty somehow? Probably not at the ISP level, but more at the game level. Well, ISP could also have some mechanisms to reduce the impact or even Akamai could force a progressive roll out.
It's an online game. You can't play the game with outdated assets. You'd not see walls where other players would, for example. What you're suggesting is the ability of ISPs to market Internet access at a certain speed but not have to deliver it based on conditions they create. It’s actually worst. You can’t even login without having latest version to play multiplayer. -- Niels. -- Mehmet +1-424-298-1903
No I didn't suggest that. -----Original Message----- From: NANOG <nanog-bounces+jean=ddostest.me@nanog.org> On Behalf Of Niels Bakker Sent: April 1, 2021 3:21 PM To: nanog@nanog.org Subject: Re: wow, lots of akamai * nanog@nanog.org (Jean St-Laurent via NANOG) [Thu 01 Apr 2021, 21:03 CEST]:
An artificial roll out penalty somehow? Probably not at the ISP level, but more at the game level. Well, ISP could also have some mechanisms to reduce the impact or even Akamai could force a progressive roll out.
It's an online game. You can't play the game with outdated assets. You'd not see walls where other players would, for example. What you're suggesting is the ability of ISPs to market Internet access at a certain speed but not have to deliver it based on conditions they create. -- Niels.
There likely is some amount of time between the product being "done" and the activation date. That time could be used (and may very well be for some platforms) to distribute the content ahead of when people need it. If too many points of congestion arise, the above mentioned time would need to be longer. Of course as an IX operator, I encourage everyone (CDNs and eyeballs) to join IXes and push them bits at maximum speed! ;-) As an eyeball ISP, sometimes the congestion is in the home, creating a poor experience, yet no one above them is to blame. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Niels Bakker" <niels=nanog@bakker.net> To: nanog@nanog.org Sent: Thursday, April 1, 2021 2:21:24 PM Subject: Re: wow, lots of akamai * nanog@nanog.org (Jean St-Laurent via NANOG) [Thu 01 Apr 2021, 21:03 CEST]:
An artificial roll out penalty somehow? Probably not at the ISP level, but more at the game level. Well, ISP could also have some mechanisms to reduce the impact or even Akamai could force a progressive roll out.
It's an online game. You can't play the game with outdated assets. You'd not see walls where other players would, for example. What you're suggesting is the ability of ISPs to market Internet access at a certain speed but not have to deliver it based on conditions they create. -- Niels.
This would be a good compromises for all. Slowly deliver the assets few days/weeks ahead. Then, on April 1st at this exact same second, you open the gate. @Mike: bull’s eye! Jean From: NANOG <nanog-bounces+jean=ddostest.me@nanog.org> On Behalf Of Mike Hammett Sent: April 1, 2021 3:31 PM To: Niels Bakker <niels=nanog@bakker.net> Cc: nanog@nanog.org Subject: Re: wow, lots of akamai There likely is some amount of time between the product being "done" and the activation date. That time could be used (and may very well be for some platforms) to distribute the content ahead of when people need it. If too many points of congestion arise, the above mentioned time would need to be longer. Of course as an IX operator, I encourage everyone (CDNs and eyeballs) to join IXes and push them bits at maximum speed! ;-) As an eyeball ISP, sometimes the congestion is in the home, creating a poor experience, yet no one above them is to blame. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com _____ From: "Niels Bakker" <niels=nanog@bakker.net <mailto:niels=nanog@bakker.net> > To: nanog@nanog.org <mailto:nanog@nanog.org> Sent: Thursday, April 1, 2021 2:21:24 PM Subject: Re: wow, lots of akamai * nanog@nanog.org <mailto:nanog@nanog.org> (Jean St-Laurent via NANOG) [Thu 01 Apr 2021, 21:03 CEST]:
An artificial roll out penalty somehow? Probably not at the ISP level, but more at the game level. Well, ISP could also have some mechanisms to reduce the impact or even Akamai could force a progressive roll out.
It's an online game. You can't play the game with outdated assets. You'd not see walls where other players would, for example. What you're suggesting is the ability of ISPs to market Internet access at a certain speed but not have to deliver it based on conditions they create. -- Niels.
* jean@ddostest.me (Jean St-Laurent) [Thu 01 Apr 2021, 21:41 CEST]:
This would be a good compromises for all. Slowly deliver the assets few days/weeks ahead.
Excellent compromise except for the people who paid for the game. Why do they need to spend storage to solve your bandwidth problem? CoD is being played on lots of devices with limited storage space, like PlayStation 4. Needing to have two versions of the game would be a heavy burden on owners. And not everybody has infinite disk space in their gaming PCs either. -- Niels.
Well yes, but I am aware of the commercial pressure to release it ASAP. How much of that is real, remains to be seen. I'm also aware it's a lot more tricky when you set release dates before you have a firm grasp on your ability to produce a stable, desired product on time. I'm not sure what kind of time lines are expected or engineered for now, but it *seems* like its a 12 - 36 hour sprint to push the content out. If so, push it out to 36 - 72 hours? Adjust accordingly for however much off I am on the first time frame. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Jean St-Laurent" <jean@ddostest.me> To: "Mike Hammett" <nanog@ics-il.net>, "Niels Bakker" <niels=nanog@bakker.net> Cc: nanog@nanog.org Sent: Thursday, April 1, 2021 2:41:38 PM Subject: RE: wow, lots of akamai This would be a good compromises for all. Slowly deliver the assets few days/weeks ahead. Then, on April 1 st at this exact same second, you open the gate. @Mike: bull’s eye! Jean From: NANOG <nanog-bounces+jean=ddostest.me@nanog.org> On Behalf Of Mike Hammett Sent: April 1, 2021 3:31 PM To: Niels Bakker <niels=nanog@bakker.net> Cc: nanog@nanog.org Subject: Re: wow, lots of akamai There likely is some amount of time between the product being "done" and the activation date. That time could be used (and may very well be for some platforms) to distribute the content ahead of when people need it. If too many points of congestion arise, the above mentioned time would need to be longer. Of course as an IX operator, I encourage everyone (CDNs and eyeballs) to join IXes and push them bits at maximum speed! ;-) As an eyeball ISP, sometimes the congestion is in the home, creating a poor experience, yet no one above them is to blame. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Niels Bakker" < niels=nanog@bakker.net > To: nanog@nanog.org Sent: Thursday, April 1, 2021 2:21:24 PM Subject: Re: wow, lots of akamai * nanog@nanog.org (Jean St-Laurent via NANOG) [Thu 01 Apr 2021, 21:03 CEST]:
An artificial roll out penalty somehow? Probably not at the ISP level, but more at the game level. Well, ISP could also have some mechanisms to reduce the impact or even Akamai could force a progressive roll out.
It's an online game. You can't play the game with outdated assets. You'd not see walls where other players would, for example. What you're suggesting is the ability of ISPs to market Internet access at a certain speed but not have to deliver it based on conditions they create. -- Niels.
* nanog@ics-il.net (Mike Hammett) [Thu 01 Apr 2021, 21:51 CEST]:
I'm not sure what kind of time lines are expected or engineered for now, but it *seems* like its a 12 - 36 hour sprint to push the content out. If so, push it out to 36 - 72 hours? Adjust accordingly for however much off I am on the first time frame.
Doesn't that mostly depend on when people turn on their gaming consoles to play? It's not on the publisher to dictate how often people want to play their game. -- Niels.
Lots of publishers will allow for new stuff to be pre-downloaded before a specified release time. There was a time that it was probably helpful in spreading the load out over time, but today it doesn't help much because either everyone starts the preload at the same time, or people don't have enough space on their console hard drives, so they have to delete the old thing and download the entire new thing on release day. which also doesn't help matters any. On Thu, Apr 1, 2021 at 4:50 PM Niels Bakker <niels=nanog@bakker.net> wrote:
* nanog@ics-il.net (Mike Hammett) [Thu 01 Apr 2021, 21:51 CEST]:
I'm not sure what kind of time lines are expected or engineered for now, but it *seems* like its a 12 - 36 hour sprint to push the content out. If so, push it out to 36 - 72 hours? Adjust accordingly for however much off I am on the first time frame.
Doesn't that mostly depend on when people turn on their gaming consoles to play? It's not on the publisher to dictate how often people want to play their game.
-- Niels.
Yes, but if they're an avid gamer, the console or PC is on all of the time anyway. If they're not an avid gamer, they probably don't care when it downloads. Also, most people I know leave their PCs on 24/7, save for whatever power saving modes they have. I know many (most?) consoles auto shut off after an amount of time and in some rare (underutilized) instances, the console will wake up for housekeeping such as this. I used to want everything instantly. Now as long as whatever game (that I don't really play anymore) or TV show or whatever downloads in the next week, I'm happy. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Niels Bakker" <niels=nanog@bakker.net> To: nanog@nanog.org Sent: Thursday, April 1, 2021 3:49:17 PM Subject: Re: wow, lots of akamai * nanog@ics-il.net (Mike Hammett) [Thu 01 Apr 2021, 21:51 CEST]:
I'm not sure what kind of time lines are expected or engineered for now, but it *seems* like its a 12 - 36 hour sprint to push the content out. If so, push it out to 36 - 72 hours? Adjust accordingly for however much off I am on the first time frame.
Doesn't that mostly depend on when people turn on their gaming consoles to play? It's not on the publisher to dictate how often people want to play their game. -- Niels.
IX’s don’t really help the source doesn’t use them. Akamai traffic. 17G via Local Cache 17G via Transit 8G via IXs. Plenty of room on IXs for more on our side. From: NANOG <nanog-bounces+lguillory=reservetele.com@nanog.org> On Behalf Of Mike Hammett Sent: Thursday, April 1, 2021 2:31 PM To: Niels Bakker <niels=nanog@bakker.net> Cc: nanog@nanog.org Subject: Re: wow, lots of akamai *External Email: Use Caution* There likely is some amount of time between the product being "done" and the activation date. That time could be used (and may very well be for some platforms) to distribute the content ahead of when people need it. If too many points of congestion arise, the above mentioned time would need to be longer. Of course as an IX operator, I encourage everyone (CDNs and eyeballs) to join IXes and push them bits at maximum speed! ;-) As an eyeball ISP, sometimes the congestion is in the home, creating a poor experience, yet no one above them is to blame. ----- Mike Hammett Intelligent Computing Solutions https://link.edgepilot.com/s/ac964af3/CIKwQEO-ZkiYpW6Z8sKObQ?u=http://www.ic... Midwest-IX https://link.edgepilot.com/s/aac6d8b8/o8NzA_6ZJESKpzRLQOS0Pw?u=http://www.mi... ________________________________ From: "Niels Bakker" <niels=nanog@bakker.net<mailto:niels=nanog@bakker.net>> To: nanog@nanog.org<mailto:nanog@nanog.org> Sent: Thursday, April 1, 2021 2:21:24 PM Subject: Re: wow, lots of akamai * nanog@nanog.org<mailto:nanog@nanog.org> (Jean St-Laurent via NANOG) [Thu 01 Apr 2021, 21:03 CEST]:
An artificial roll out penalty somehow? Probably not at the ISP level, but more at the game level. Well, ISP could also have some mechanisms to reduce the impact or even Akamai could force a progressive roll out.
It's an online game. You can't play the game with outdated assets. You'd not see walls where other players would, for example. What you're suggesting is the ability of ISPs to market Internet access at a certain speed but not have to deliver it based on conditions they create. -- Niels. Links contained in this email have been replaced. If you click on a link in the email above, the link will be analyzed for known threats. If a known threat is found, you will not be able to proceed to the destination. If suspicious content is detected, you will see a warning.
On Thu, Apr 01, 2021 at 08:09:24PM +0000, Luke Guillory wrote:
IX’s don’t really help the source doesn’t use them.
Akamai traffic. 17G via Local Cache 17G via Transit 8G via IXs.
Plenty of room on IXs for more on our side.
Often we can see the ports at the IX flatline in these cases. I've been working to improve some of them, for exampe at the DetroitIX we have 300G of capacity and you can see the impact at the IX here: https://www.detroitix.com/#stats If you look at the monthly graph you can see other game download events. It can be incredibly hard to right-size this stuff, please do reach out to myself or Niels if things didn't work right for your network. Also you should reach out to your carriers who may have had congestion to ensure they are upgrading their networks as well. I've been told by some teams they won't upgraded for a day a month of thsi big traffic, but when one of their customers has issues they immediately escalate to us to ask us to move (and we try hard to do that). It's not all throw bandwidth at the problem, but with 100g optics being so cheap these days, it may be the lower bar - jared -- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.
On 4/1/2021 3:21 PM, Niels Bakker wrote:
* nanog@nanog.org (Jean St-Laurent via NANOG) [Thu 01 Apr 2021, 21:03 CEST]:
An artificial roll out penalty somehow? Probably not at the ISP level, but more at the game level. Well, ISP could also have some mechanisms to reduce the impact or even Akamai could force a progressive roll out.
It's an online game. You can't play the game with outdated assets. You'd not see walls where other players would, for example.
What you're suggesting is the ability of ISPs to market Internet access at a certain speed but not have to deliver it based on conditions they create.
-- Niels.
There was at least one online gaming platform that used to stage their game updates, where clients would download the update over the previous week or so, and then on go-live day, the game updated itself. There were a few that had to download on go-live day, but the clients I ran, and friends ran, had downloaded prior. I don't know if it is still that way, but it seems like reasonable solution to me. Cheers, -dtb
On Thu, Apr 01, 2021 at 03:31:39PM -0400, Dave Brockman - DVS wrote:
On 4/1/2021 3:21 PM, Niels Bakker wrote:
* nanog@nanog.org (Jean St-Laurent via NANOG) [Thu 01 Apr 2021, 21:03 CEST]:
An artificial roll out penalty somehow? Probably not at the ISP level, but more at the game level. Well, ISP could also have some mechanisms to reduce the impact or even Akamai could force a progressive roll out.
It's an online game. You can't play the game with outdated assets. You'd not see walls where other players would, for example.
What you're suggesting is the ability of ISPs to market Internet access at a certain speed but not have to deliver it based on conditions they create.
-- Niels.
There was at least one online gaming platform that used to stage their game updates, where clients would download the update over the previous week or so, and then on go-live day, the game updated itself. There were a few that had to download on go-live day, but the clients I ran, and friends ran, had downloaded prior. I don't know if it is still that way, but it seems like reasonable solution to me.
Actually in the past year many of the gaming studios have started to do this for the release process. It will start during an off-peak hour and then flow through the day. The end-user induced demand and stress it places on networks doesn't fit into the historical 95/5 30-day peak planning model. I know some carriers are struggling to adjust. There's a few of us here on-list, please reach out so we can help. - Jared -- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.
As an IX, we've noticed that a lot of networks don't avail themselves of all opportunities to connect to sources of content. That lack of diversity can cause issues when there are failures, congestion, etc. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Jared Mauch" <jared@puck.nether.net> To: "Dave Brockman - DVS" <dave@dvstn.com> Cc: nanog@nanog.org Sent: Friday, April 2, 2021 7:08:09 AM Subject: Re: wow, lots of akamai On Thu, Apr 01, 2021 at 03:31:39PM -0400, Dave Brockman - DVS wrote:
On 4/1/2021 3:21 PM, Niels Bakker wrote:
* nanog@nanog.org (Jean St-Laurent via NANOG) [Thu 01 Apr 2021, 21:03 CEST]:
An artificial roll out penalty somehow? Probably not at the ISP level, but more at the game level. Well, ISP could also have some mechanisms to reduce the impact or even Akamai could force a progressive roll out.
It's an online game. You can't play the game with outdated assets. You'd not see walls where other players would, for example.
What you're suggesting is the ability of ISPs to market Internet access at a certain speed but not have to deliver it based on conditions they create.
-- Niels.
There was at least one online gaming platform that used to stage their game updates, where clients would download the update over the previous week or so, and then on go-live day, the game updated itself. There were a few that had to download on go-live day, but the clients I ran, and friends ran, had downloaded prior. I don't know if it is still that way, but it seems like reasonable solution to me.
Actually in the past year many of the gaming studios have started to do this for the release process. It will start during an off-peak hour and then flow through the day. The end-user induced demand and stress it places on networks doesn't fit into the historical 95/5 30-day peak planning model. I know some carriers are struggling to adjust. There's a few of us here on-list, please reach out so we can help. - Jared -- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.
Niels, I think to clarify Jean's point, when you buy a 300mbps circuit, you're paying for 300mbps of *internet *access. That does not mean that a network should (and in this case small-medium ones simply can't) build all of their capacity to service a large number of customer circuits at line rate at the same time for an extended period, ESPECIALLY to the exact same endpoint. It's just not economically reasonable to expect that. Remember we're talking about residential service here, not enterprise circuits. Therefore, how do you prevent this spike of [insert large number here] gigabits traversing the network at the same time from causing issues? Build more network? That sounds easy, but there are plenty of legitimate reasons why ISPs can't or don't want to do that, particularly for an event that only occurs once per quarter or so. Does Akamai bear some burden here to make these rollouts less troublesome for the ISPs they traverse through the last mile(s)? IMO yes, yes they do. When you're doing something new and unprecedented, as Akamai frequently brags about on Twitter, like having rapid, bursty growth of traffic, you need to consider that just because you can generate it, doesn't mean it can be delivered. They've gotta be more sophisticated than a bunch of servers with SSD arrays, ramdisks, and 100 gig interfaces, so there's no excuse for them here to just blindly fill every link they have after sitting idle for weeks/months at a time and expect everything to come out alright and nobody to complain about it. On Thu, Apr 1, 2021 at 1:21 PM Niels Bakker <niels=nanog@bakker.net> wrote:
* nanog@nanog.org (Jean St-Laurent via NANOG) [Thu 01 Apr 2021, 21:03 CEST]:
An artificial roll out penalty somehow? Probably not at the ISP level, but more at the game level. Well, ISP could also have some mechanisms to reduce the impact or even Akamai could force a progressive roll out.
It's an online game. You can't play the game with outdated assets. You'd not see walls where other players would, for example.
What you're suggesting is the ability of ISPs to market Internet access at a certain speed but not have to deliver it based on conditions they create.
-- Niels.
-- Matt Erculiani ERCUL-ARIN
Matt: I am going to disagree with your characterization of how Akamai - and many other CDNs - manage things. First, to be blunt, if you really think Akamai nodes are “sitting idle for weeks” before CoD comes out with a new game, you are clearly confused. More importantly, I know for a fact Akamai has spent ungodly amounts of money & resources putting content precisely where the ISPs ask them to put it, deliver it over the pipes the ISPs ask them to deliver it, at precisely the capacity the ISPs tell them. On the other hand, I agree with your characterization of residential broadband. It is ridiculous to expect a neighborhood with 1,000 homes each with 1 Gbps links to have a terabit of uplink capacity. But it also should have a lot more than 10 Gbps, IMHO. Unfortunately, most neighborhoods I have seen are closer to the latter than the former. Finally, this could quickly devolve into finger pointing. You say the CDNs bear some responsibility? They may well respond that the large broadband providers ask for cash to interconnect - but still require the CDNs to do all the work. The CDNs did not create the content, or tell the users which content to pull. When I pay $NATIONAL_PROVIDER, I expect them to provide me with access to the Internet. Not just to the content that pays that provider. Personally, I have zero problems with the ISPs saying “give me a cache to put here with this sized uplink” or “please deliver to these users over this xconn / IX / whatever”. I have a huge problem with the ISPs blaming the ISPs for delivering what the ISP’s users request. Of course, this could all be solved if there were more competition in broadband in the US (and many other countries). But that is a totally different 10,000 post thread (that we have had many dozens of times). -- TTFN, patrick
On Apr 1, 2021, at 3:53 PM, Matt Erculiani <merculiani@gmail.com> wrote:
Niels,
I think to clarify Jean's point, when you buy a 300mbps circuit, you're paying for 300mbps of internet access.
That does not mean that a network should (and in this case small-medium ones simply can't) build all of their capacity to service a large number of customer circuits at line rate at the same time for an extended period, ESPECIALLY to the exact same endpoint. It's just not economically reasonable to expect that. Remember we're talking about residential service here, not enterprise circuits.
Therefore, how do you prevent this spike of [insert large number here] gigabits traversing the network at the same time from causing issues? Build more network? That sounds easy, but there are plenty of legitimate reasons why ISPs can't or don't want to do that, particularly for an event that only occurs once per quarter or so.
Does Akamai bear some burden here to make these rollouts less troublesome for the ISPs they traverse through the last mile(s)? IMO yes, yes they do. When you're doing something new and unprecedented, as Akamai frequently brags about on Twitter, like having rapid, bursty growth of traffic, you need to consider that just because you can generate it, doesn't mean it can be delivered. They've gotta be more sophisticated than a bunch of servers with SSD arrays, ramdisks, and 100 gig interfaces, so there's no excuse for them here to just blindly fill every link they have after sitting idle for weeks/months at a time and expect everything to come out alright and nobody to complain about it.
On Thu, Apr 1, 2021 at 1:21 PM Niels Bakker <niels=nanog@bakker.net <mailto:nanog@bakker.net>> wrote: * nanog@nanog.org <mailto:nanog@nanog.org> (Jean St-Laurent via NANOG) [Thu 01 Apr 2021, 21:03 CEST]:
An artificial roll out penalty somehow? Probably not at the ISP level, but more at the game level. Well, ISP could also have some mechanisms to reduce the impact or even Akamai could force a progressive roll out.
It's an online game. You can't play the game with outdated assets. You'd not see walls where other players would, for example.
What you're suggesting is the ability of ISPs to market Internet access at a certain speed but not have to deliver it based on conditions they create.
-- Niels.
-- Matt Erculiani ERCUL-ARIN
I find all of these comments interesting and useful even if we don’t all agree on what would be the best solution. Maybe it’s just a misunderstanding from the perspective of each parties. Maybe there is no solution, but it’s still interesting. So far, everybody seems to have a bit of the solution. In the end, if I search on this mailing list: Wow, spike, huge, akamai, yesterday or a combination of these words… the answer is always CoD. 😃 Jean From: NANOG <nanog-bounces+jean=ddostest.me@nanog.org> On Behalf Of Patrick W. Gilmore Sent: April 1, 2021 4:09 PM To: North American Operators' Group <nanog@nanog.org> Subject: Re: wow, lots of akamai Matt: I am going to disagree with your characterization of how Akamai - and many other CDNs - manage things. First, to be blunt, if you really think Akamai nodes are “sitting idle for weeks” before CoD comes out with a new game, you are clearly confused. More importantly, I know for a fact Akamai has spent ungodly amounts of money & resources putting content precisely where the ISPs ask them to put it, deliver it over the pipes the ISPs ask them to deliver it, at precisely the capacity the ISPs tell them. On the other hand, I agree with your characterization of residential broadband. It is ridiculous to expect a neighborhood with 1,000 homes each with 1 Gbps links to have a terabit of uplink capacity. But it also should have a lot more than 10 Gbps, IMHO. Unfortunately, most neighborhoods I have seen are closer to the latter than the former. Finally, this could quickly devolve into finger pointing. You say the CDNs bear some responsibility? They may well respond that the large broadband providers ask for cash to interconnect - but still require the CDNs to do all the work. The CDNs did not create the content, or tell the users which content to pull. When I pay $NATIONAL_PROVIDER, I expect them to provide me with access to the Internet. Not just to the content that pays that provider. Personally, I have zero problems with the ISPs saying “give me a cache to put here with this sized uplink” or “please deliver to these users over this xconn / IX / whatever”. I have a huge problem with the ISPs blaming the ISPs for delivering what the ISP’s users request. Of course, this could all be solved if there were more competition in broadband in the US (and many other countries). But that is a totally different 10,000 post thread (that we have had many dozens of times). -- TTFN, patrick On Apr 1, 2021, at 3:53 PM, Matt Erculiani <merculiani@gmail.com <mailto:merculiani@gmail.com> > wrote: Niels, I think to clarify Jean's point, when you buy a 300mbps circuit, you're paying for 300mbps of internet access. That does not mean that a network should (and in this case small-medium ones simply can't) build all of their capacity to service a large number of customer circuits at line rate at the same time for an extended period, ESPECIALLY to the exact same endpoint. It's just not economically reasonable to expect that. Remember we're talking about residential service here, not enterprise circuits. Therefore, how do you prevent this spike of [insert large number here] gigabits traversing the network at the same time from causing issues? Build more network? That sounds easy, but there are plenty of legitimate reasons why ISPs can't or don't want to do that, particularly for an event that only occurs once per quarter or so. Does Akamai bear some burden here to make these rollouts less troublesome for the ISPs they traverse through the last mile(s)? IMO yes, yes they do. When you're doing something new and unprecedented, as Akamai frequently brags about on Twitter, like having rapid, bursty growth of traffic, you need to consider that just because you can generate it, doesn't mean it can be delivered. They've gotta be more sophisticated than a bunch of servers with SSD arrays, ramdisks, and 100 gig interfaces, so there's no excuse for them here to just blindly fill every link they have after sitting idle for weeks/months at a time and expect everything to come out alright and nobody to complain about it. On Thu, Apr 1, 2021 at 1:21 PM Niels Bakker <niels=nanog@bakker.net <mailto:nanog@bakker.net> > wrote: * nanog@nanog.org <mailto:nanog@nanog.org> (Jean St-Laurent via NANOG) [Thu 01 Apr 2021, 21:03 CEST]:
An artificial roll out penalty somehow? Probably not at the ISP level, but more at the game level. Well, ISP could also have some mechanisms to reduce the impact or even Akamai could force a progressive roll out.
It's an online game. You can't play the game with outdated assets. You'd not see walls where other players would, for example. What you're suggesting is the ability of ISPs to market Internet access at a certain speed but not have to deliver it based on conditions they create. -- Niels. -- Matt Erculiani ERCUL-ARIN
Patrick, “Personally, I have zero problems with the ISPs saying “give me a cache to put here with this sized uplink” or “please deliver to these users over this xconn / IX / whatever”. I have a huge problem with the ISPs blaming the ISPs for delivering what the ISP’s users request.” We had to beg to get more local CDN resources which still doesn’t even deliver half of their traffic. “More importantly, I know for a fact Akamai has spent ungodly amounts of money & resources putting content precisely where the ISPs ask them to put it, deliver it over the pipes the ISPs ask them to deliver it, at precisely the capacity the ISPs tell them.” From our experience this hasn’t been the case, can’t get PNI, can’t get them to add resources on their end to send more IX traffic. Don’t take that as me thinking they’re not doing things on their end to try and make things better, I’m not blind that it takes a massive number of resources. We just haven’t see things the way you have is all. From: NANOG <nanog-bounces+lguillory=reservetele.com@nanog.org> On Behalf Of Patrick W. Gilmore Sent: Thursday, April 1, 2021 3:09 PM To: North American Operators' Group <nanog@nanog.org> Subject: Re: wow, lots of akamai *External Email: Use Caution* Matt: I am going to disagree with your characterization of how Akamai - and many other CDNs - manage things. First, to be blunt, if you really think Akamai nodes are “sitting idle for weeks” before CoD comes out with a new game, you are clearly confused. More importantly, I know for a fact Akamai has spent ungodly amounts of money & resources putting content precisely where the ISPs ask them to put it, deliver it over the pipes the ISPs ask them to deliver it, at precisely the capacity the ISPs tell them. On the other hand, I agree with your characterization of residential broadband. It is ridiculous to expect a neighborhood with 1,000 homes each with 1 Gbps links to have a terabit of uplink capacity. But it also should have a lot more than 10 Gbps, IMHO. Unfortunately, most neighborhoods I have seen are closer to the latter than the former. Finally, this could quickly devolve into finger pointing. You say the CDNs bear some responsibility? They may well respond that the large broadband providers ask for cash to interconnect - but still require the CDNs to do all the work. The CDNs did not create the content, or tell the users which content to pull. When I pay $NATIONAL_PROVIDER, I expect them to provide me with access to the Internet. Not just to the content that pays that provider. Personally, I have zero problems with the ISPs saying “give me a cache to put here with this sized uplink” or “please deliver to these users over this xconn / IX / whatever”. I have a huge problem with the ISPs blaming the ISPs for delivering what the ISP’s users request. Of course, this could all be solved if there were more competition in broadband in the US (and many other countries). But that is a totally different 10,000 post thread (that we have had many dozens of times). -- TTFN, patrick On Apr 1, 2021, at 3:53 PM, Matt Erculiani <merculiani@gmail.com<mailto:merculiani@gmail.com>> wrote: Niels, I think to clarify Jean's point, when you buy a 300mbps circuit, you're paying for 300mbps of internet access. That does not mean that a network should (and in this case small-medium ones simply can't) build all of their capacity to service a large number of customer circuits at line rate at the same time for an extended period, ESPECIALLY to the exact same endpoint. It's just not economically reasonable to expect that. Remember we're talking about residential service here, not enterprise circuits. Therefore, how do you prevent this spike of [insert large number here] gigabits traversing the network at the same time from causing issues? Build more network? That sounds easy, but there are plenty of legitimate reasons why ISPs can't or don't want to do that, particularly for an event that only occurs once per quarter or so. Does Akamai bear some burden here to make these rollouts less troublesome for the ISPs they traverse through the last mile(s)? IMO yes, yes they do. When you're doing something new and unprecedented, as Akamai frequently brags about on Twitter, like having rapid, bursty growth of traffic, you need to consider that just because you can generate it, doesn't mean it can be delivered. They've gotta be more sophisticated than a bunch of servers with SSD arrays, ramdisks, and 100 gig interfaces, so there's no excuse for them here to just blindly fill every link they have after sitting idle for weeks/months at a time and expect everything to come out alright and nobody to complain about it. On Thu, Apr 1, 2021 at 1:21 PM Niels Bakker <niels=nanog@bakker.net<mailto:nanog@bakker.net>> wrote: * nanog@nanog.org<mailto:nanog@nanog.org> (Jean St-Laurent via NANOG) [Thu 01 Apr 2021, 21:03 CEST]:
An artificial roll out penalty somehow? Probably not at the ISP level, but more at the game level. Well, ISP could also have some mechanisms to reduce the impact or even Akamai could force a progressive roll out.
It's an online game. You can't play the game with outdated assets. You'd not see walls where other players would, for example. What you're suggesting is the ability of ISPs to market Internet access at a certain speed but not have to deliver it based on conditions they create. -- Niels. -- Matt Erculiani ERCUL-ARIN
Patrick,
First, to be blunt, if you really think Akamai nodes are “sitting idle for weeks” before CoD comes out with a new game, you are clearly confused.
"Idle" in the sense that when you look at a graph of traffic before and after a large push such as this makes the rest of the week's traffic look like a horizontal line at the bottom, admittedly poor word choice, yes, but far from "confused" as to what CDNs do under relatively normal circumstances. Otherwise very valid points you've raised. Tom,
Akamai, and other CDNs, do not **generate** traffic ; they serve the requests generated by users.
A user sends a few megabytes of request and receives 50 gigs of reply. They aren't DDoSing the network, but they're amplifying a single 50 gig copy they receive from the mothership and turning it into likely tens of terabytes of traffic. Yes, that's a CDN's job, but that volume of legitimate traffic and the very tiny window with which it is transmitted is likely to be a burden for even the largest residential ISPs. -Matt On Thu, Apr 1, 2021 at 2:09 PM Patrick W. Gilmore <patrick@ianai.net> wrote:
Matt:
I am going to disagree with your characterization of how Akamai - and many other CDNs - manage things. First, to be blunt, if you really think Akamai nodes are “sitting idle for weeks” before CoD comes out with a new game, you are clearly confused.
More importantly, I know for a fact Akamai has spent ungodly amounts of money & resources putting content precisely where the ISPs ask them to put it, deliver it over the pipes the ISPs ask them to deliver it, at precisely the capacity the ISPs tell them.
On the other hand, I agree with your characterization of residential broadband. It is ridiculous to expect a neighborhood with 1,000 homes each with 1 Gbps links to have a terabit of uplink capacity. But it also should have a lot more than 10 Gbps, IMHO. Unfortunately, most neighborhoods I have seen are closer to the latter than the former.
Finally, this could quickly devolve into finger pointing. You say the CDNs bear some responsibility? They may well respond that the large broadband providers ask for cash to interconnect - but still require the CDNs to do all the work. The CDNs did not create the content, or tell the users which content to pull. When I pay $NATIONAL_PROVIDER, I expect them to provide me with access to the Internet. Not just to the content that pays that provider.
Personally, I have zero problems with the ISPs saying “give me a cache to put here with this sized uplink” or “please deliver to these users over this xconn / IX / whatever”. I have a huge problem with the ISPs blaming the ISPs for delivering what the ISP’s users request.
Of course, this could all be solved if there were more competition in broadband in the US (and many other countries). But that is a totally different 10,000 post thread (that we have had many dozens of times).
-- TTFN, patrick
On Apr 1, 2021, at 3:53 PM, Matt Erculiani <merculiani@gmail.com> wrote:
Niels,
I think to clarify Jean's point, when you buy a 300mbps circuit, you're paying for 300mbps of *internet *access.
That does not mean that a network should (and in this case small-medium ones simply can't) build all of their capacity to service a large number of customer circuits at line rate at the same time for an extended period, ESPECIALLY to the exact same endpoint. It's just not economically reasonable to expect that. Remember we're talking about residential service here, not enterprise circuits.
Therefore, how do you prevent this spike of [insert large number here] gigabits traversing the network at the same time from causing issues? Build more network? That sounds easy, but there are plenty of legitimate reasons why ISPs can't or don't want to do that, particularly for an event that only occurs once per quarter or so.
Does Akamai bear some burden here to make these rollouts less troublesome for the ISPs they traverse through the last mile(s)? IMO yes, yes they do. When you're doing something new and unprecedented, as Akamai frequently brags about on Twitter, like having rapid, bursty growth of traffic, you need to consider that just because you can generate it, doesn't mean it can be delivered. They've gotta be more sophisticated than a bunch of servers with SSD arrays, ramdisks, and 100 gig interfaces, so there's no excuse for them here to just blindly fill every link they have after sitting idle for weeks/months at a time and expect everything to come out alright and nobody to complain about it.
On Thu, Apr 1, 2021 at 1:21 PM Niels Bakker <niels=nanog@bakker.net> wrote:
* nanog@nanog.org (Jean St-Laurent via NANOG) [Thu 01 Apr 2021, 21:03 CEST]:
An artificial roll out penalty somehow? Probably not at the ISP level, but more at the game level. Well, ISP could also have some mechanisms to reduce the impact or even Akamai could force a progressive roll out.
It's an online game. You can't play the game with outdated assets. You'd not see walls where other players would, for example.
What you're suggesting is the ability of ISPs to market Internet access at a certain speed but not have to deliver it based on conditions they create.
-- Niels.
-- Matt Erculiani ERCUL-ARIN
-- Matt Erculiani ERCUL-ARIN
A user sends a few megabytes of request and receives 50 gigs of reply. They aren't DDoSing the network, but they're amplifying a single 50 gig copy they receive from the mothership and turning it into likely tens of terabytes of traffic. Yes, that's a CDN's job, but that volume of legitimate traffic and the very tiny window with which it is transmitted is likely to be a burden for even the largest residential ISPs.
I'm sitting at home, and I could send a 50k request for a 50G file right now from a source not fronted by a CDN. What do? My ISP is still has to deliver it to me. The fact that the 50G file does or does not come from a CDN is irrelevant. The CDN just happens to be a point source that a lot of users happen to connect to. CDNs want to have the best performance to users because that's what brings them business. A poorly performing CDN will lose customers to a better performing one. The trend for years has been instead of ISPs investing in infrastructure to effectively handle the traffic that their users request, they turf that to CDNs. In many cases, a CDN will put a cache box in or extend a circuit at a loss to them, because they know if the performance metrics get bad, business will be taken elsewhere, even if the CAUSE of the poor performance is actually at the edge of, or inside , the ISPs network. ISPs in the US can get away with this because their users are captive and rarely have an alternative choice of provider. On Thu, Apr 1, 2021 at 4:33 PM Matt Erculiani <merculiani@gmail.com> wrote:
Patrick,
First, to be blunt, if you really think Akamai nodes are “sitting idle for weeks” before CoD comes out with a new game, you are clearly confused.
"Idle" in the sense that when you look at a graph of traffic before and after a large push such as this makes the rest of the week's traffic look like a horizontal line at the bottom, admittedly poor word choice, yes, but far from "confused" as to what CDNs do under relatively normal circumstances. Otherwise very valid points you've raised.
Tom,
Akamai, and other CDNs, do not **generate** traffic ; they serve the requests generated by users.
A user sends a few megabytes of request and receives 50 gigs of reply. They aren't DDoSing the network, but they're amplifying a single 50 gig copy they receive from the mothership and turning it into likely tens of terabytes of traffic. Yes, that's a CDN's job, but that volume of legitimate traffic and the very tiny window with which it is transmitted is likely to be a burden for even the largest residential ISPs.
-Matt
On Thu, Apr 1, 2021 at 2:09 PM Patrick W. Gilmore <patrick@ianai.net> wrote:
Matt:
I am going to disagree with your characterization of how Akamai - and many other CDNs - manage things. First, to be blunt, if you really think Akamai nodes are “sitting idle for weeks” before CoD comes out with a new game, you are clearly confused.
More importantly, I know for a fact Akamai has spent ungodly amounts of money & resources putting content precisely where the ISPs ask them to put it, deliver it over the pipes the ISPs ask them to deliver it, at precisely the capacity the ISPs tell them.
On the other hand, I agree with your characterization of residential broadband. It is ridiculous to expect a neighborhood with 1,000 homes each with 1 Gbps links to have a terabit of uplink capacity. But it also should have a lot more than 10 Gbps, IMHO. Unfortunately, most neighborhoods I have seen are closer to the latter than the former.
Finally, this could quickly devolve into finger pointing. You say the CDNs bear some responsibility? They may well respond that the large broadband providers ask for cash to interconnect - but still require the CDNs to do all the work. The CDNs did not create the content, or tell the users which content to pull. When I pay $NATIONAL_PROVIDER, I expect them to provide me with access to the Internet. Not just to the content that pays that provider.
Personally, I have zero problems with the ISPs saying “give me a cache to put here with this sized uplink” or “please deliver to these users over this xconn / IX / whatever”. I have a huge problem with the ISPs blaming the ISPs for delivering what the ISP’s users request.
Of course, this could all be solved if there were more competition in broadband in the US (and many other countries). But that is a totally different 10,000 post thread (that we have had many dozens of times).
-- TTFN, patrick
On Apr 1, 2021, at 3:53 PM, Matt Erculiani <merculiani@gmail.com> wrote:
Niels,
I think to clarify Jean's point, when you buy a 300mbps circuit, you're paying for 300mbps of *internet *access.
That does not mean that a network should (and in this case small-medium ones simply can't) build all of their capacity to service a large number of customer circuits at line rate at the same time for an extended period, ESPECIALLY to the exact same endpoint. It's just not economically reasonable to expect that. Remember we're talking about residential service here, not enterprise circuits.
Therefore, how do you prevent this spike of [insert large number here] gigabits traversing the network at the same time from causing issues? Build more network? That sounds easy, but there are plenty of legitimate reasons why ISPs can't or don't want to do that, particularly for an event that only occurs once per quarter or so.
Does Akamai bear some burden here to make these rollouts less troublesome for the ISPs they traverse through the last mile(s)? IMO yes, yes they do. When you're doing something new and unprecedented, as Akamai frequently brags about on Twitter, like having rapid, bursty growth of traffic, you need to consider that just because you can generate it, doesn't mean it can be delivered. They've gotta be more sophisticated than a bunch of servers with SSD arrays, ramdisks, and 100 gig interfaces, so there's no excuse for them here to just blindly fill every link they have after sitting idle for weeks/months at a time and expect everything to come out alright and nobody to complain about it.
On Thu, Apr 1, 2021 at 1:21 PM Niels Bakker <niels=nanog@bakker.net> wrote:
* nanog@nanog.org (Jean St-Laurent via NANOG) [Thu 01 Apr 2021, 21:03 CEST]:
An artificial roll out penalty somehow? Probably not at the ISP level, but more at the game level. Well, ISP could also have some mechanisms to reduce the impact or even Akamai could force a progressive roll out.
It's an online game. You can't play the game with outdated assets. You'd not see walls where other players would, for example.
What you're suggesting is the ability of ISPs to market Internet access at a certain speed but not have to deliver it based on conditions they create.
-- Niels.
-- Matt Erculiani ERCUL-ARIN
-- Matt Erculiani ERCUL-ARIN
Tom, All due respect, but there is a massive difference between one user downloading 50G and thousands of users each downloading 50G when they all go to play their videogame of choice at around the same time. -Matt On Thu, Apr 1, 2021 at 2:46 PM Tom Beecher <beecher@beecher.cc> wrote:
A user sends a few megabytes of request and receives 50 gigs of reply.
They aren't DDoSing the network, but they're amplifying a single 50 gig copy they receive from the mothership and turning it into likely tens of terabytes of traffic. Yes, that's a CDN's job, but that volume of legitimate traffic and the very tiny window with which it is transmitted is likely to be a burden for even the largest residential ISPs.
I'm sitting at home, and I could send a 50k request for a 50G file right now from a source not fronted by a CDN. What do? My ISP is still has to deliver it to me. The fact that the 50G file does or does not come from a CDN is irrelevant. The CDN just happens to be a point source that a lot of users happen to connect to.
CDNs want to have the best performance to users because that's what brings them business. A poorly performing CDN will lose customers to a better performing one. The trend for years has been instead of ISPs investing in infrastructure to effectively handle the traffic that their users request, they turf that to CDNs. In many cases, a CDN will put a cache box in or extend a circuit at a loss to them, because they know if the performance metrics get bad, business will be taken elsewhere, even if the CAUSE of the poor performance is actually at the edge of, or inside , the ISPs network.
ISPs in the US can get away with this because their users are captive and rarely have an alternative choice of provider.
On Thu, Apr 1, 2021 at 4:33 PM Matt Erculiani <merculiani@gmail.com> wrote:
Patrick,
First, to be blunt, if you really think Akamai nodes are “sitting idle for weeks” before CoD comes out with a new game, you are clearly confused.
"Idle" in the sense that when you look at a graph of traffic before and after a large push such as this makes the rest of the week's traffic look like a horizontal line at the bottom, admittedly poor word choice, yes, but far from "confused" as to what CDNs do under relatively normal circumstances. Otherwise very valid points you've raised.
Tom,
Akamai, and other CDNs, do not **generate** traffic ; they serve the requests generated by users.
A user sends a few megabytes of request and receives 50 gigs of reply. They aren't DDoSing the network, but they're amplifying a single 50 gig copy they receive from the mothership and turning it into likely tens of terabytes of traffic. Yes, that's a CDN's job, but that volume of legitimate traffic and the very tiny window with which it is transmitted is likely to be a burden for even the largest residential ISPs.
-Matt
On Thu, Apr 1, 2021 at 2:09 PM Patrick W. Gilmore <patrick@ianai.net> wrote:
Matt:
I am going to disagree with your characterization of how Akamai - and many other CDNs - manage things. First, to be blunt, if you really think Akamai nodes are “sitting idle for weeks” before CoD comes out with a new game, you are clearly confused.
More importantly, I know for a fact Akamai has spent ungodly amounts of money & resources putting content precisely where the ISPs ask them to put it, deliver it over the pipes the ISPs ask them to deliver it, at precisely the capacity the ISPs tell them.
On the other hand, I agree with your characterization of residential broadband. It is ridiculous to expect a neighborhood with 1,000 homes each with 1 Gbps links to have a terabit of uplink capacity. But it also should have a lot more than 10 Gbps, IMHO. Unfortunately, most neighborhoods I have seen are closer to the latter than the former.
Finally, this could quickly devolve into finger pointing. You say the CDNs bear some responsibility? They may well respond that the large broadband providers ask for cash to interconnect - but still require the CDNs to do all the work. The CDNs did not create the content, or tell the users which content to pull. When I pay $NATIONAL_PROVIDER, I expect them to provide me with access to the Internet. Not just to the content that pays that provider.
Personally, I have zero problems with the ISPs saying “give me a cache to put here with this sized uplink” or “please deliver to these users over this xconn / IX / whatever”. I have a huge problem with the ISPs blaming the ISPs for delivering what the ISP’s users request.
Of course, this could all be solved if there were more competition in broadband in the US (and many other countries). But that is a totally different 10,000 post thread (that we have had many dozens of times).
-- TTFN, patrick
On Apr 1, 2021, at 3:53 PM, Matt Erculiani <merculiani@gmail.com> wrote:
Niels,
I think to clarify Jean's point, when you buy a 300mbps circuit, you're paying for 300mbps of *internet *access.
That does not mean that a network should (and in this case small-medium ones simply can't) build all of their capacity to service a large number of customer circuits at line rate at the same time for an extended period, ESPECIALLY to the exact same endpoint. It's just not economically reasonable to expect that. Remember we're talking about residential service here, not enterprise circuits.
Therefore, how do you prevent this spike of [insert large number here] gigabits traversing the network at the same time from causing issues? Build more network? That sounds easy, but there are plenty of legitimate reasons why ISPs can't or don't want to do that, particularly for an event that only occurs once per quarter or so.
Does Akamai bear some burden here to make these rollouts less troublesome for the ISPs they traverse through the last mile(s)? IMO yes, yes they do. When you're doing something new and unprecedented, as Akamai frequently brags about on Twitter, like having rapid, bursty growth of traffic, you need to consider that just because you can generate it, doesn't mean it can be delivered. They've gotta be more sophisticated than a bunch of servers with SSD arrays, ramdisks, and 100 gig interfaces, so there's no excuse for them here to just blindly fill every link they have after sitting idle for weeks/months at a time and expect everything to come out alright and nobody to complain about it.
On Thu, Apr 1, 2021 at 1:21 PM Niels Bakker <niels=nanog@bakker.net> wrote:
* nanog@nanog.org (Jean St-Laurent via NANOG) [Thu 01 Apr 2021, 21:03 CEST]:
An artificial roll out penalty somehow? Probably not at the ISP level, but more at the game level. Well, ISP could also have some mechanisms to reduce the impact or even Akamai could force a progressive roll out.
It's an online game. You can't play the game with outdated assets. You'd not see walls where other players would, for example.
What you're suggesting is the ability of ISPs to market Internet access at a certain speed but not have to deliver it based on conditions they create.
-- Niels.
-- Matt Erculiani ERCUL-ARIN
-- Matt Erculiani ERCUL-ARIN
-- Matt Erculiani ERCUL-ARIN
No disrespect taken, or intended back in your direction, but again, I disagree. If thousands of users are downloading 50G files at the same time, it really doesn't matter if they are pulling from a CDN or the origin directly. The volume of traffic still has to be handled. Yes, it's a burden on the ISP, but it's a burden created by the usage created by their subscribers. On Thu, Apr 1, 2021 at 4:57 PM Matt Erculiani <merculiani@gmail.com> wrote:
Tom,
All due respect, but there is a massive difference between one user downloading 50G and thousands of users each downloading 50G when they all go to play their videogame of choice at around the same time.
-Matt
On Thu, Apr 1, 2021 at 2:46 PM Tom Beecher <beecher@beecher.cc> wrote:
A user sends a few megabytes of request and receives 50 gigs of reply.
They aren't DDoSing the network, but they're amplifying a single 50 gig copy they receive from the mothership and turning it into likely tens of terabytes of traffic. Yes, that's a CDN's job, but that volume of legitimate traffic and the very tiny window with which it is transmitted is likely to be a burden for even the largest residential ISPs.
I'm sitting at home, and I could send a 50k request for a 50G file right now from a source not fronted by a CDN. What do? My ISP is still has to deliver it to me. The fact that the 50G file does or does not come from a CDN is irrelevant. The CDN just happens to be a point source that a lot of users happen to connect to.
CDNs want to have the best performance to users because that's what brings them business. A poorly performing CDN will lose customers to a better performing one. The trend for years has been instead of ISPs investing in infrastructure to effectively handle the traffic that their users request, they turf that to CDNs. In many cases, a CDN will put a cache box in or extend a circuit at a loss to them, because they know if the performance metrics get bad, business will be taken elsewhere, even if the CAUSE of the poor performance is actually at the edge of, or inside , the ISPs network.
ISPs in the US can get away with this because their users are captive and rarely have an alternative choice of provider.
On Thu, Apr 1, 2021 at 4:33 PM Matt Erculiani <merculiani@gmail.com> wrote:
Patrick,
First, to be blunt, if you really think Akamai nodes are “sitting idle for weeks” before CoD comes out with a new game, you are clearly confused.
"Idle" in the sense that when you look at a graph of traffic before and after a large push such as this makes the rest of the week's traffic look like a horizontal line at the bottom, admittedly poor word choice, yes, but far from "confused" as to what CDNs do under relatively normal circumstances. Otherwise very valid points you've raised.
Tom,
Akamai, and other CDNs, do not **generate** traffic ; they serve the requests generated by users.
A user sends a few megabytes of request and receives 50 gigs of reply. They aren't DDoSing the network, but they're amplifying a single 50 gig copy they receive from the mothership and turning it into likely tens of terabytes of traffic. Yes, that's a CDN's job, but that volume of legitimate traffic and the very tiny window with which it is transmitted is likely to be a burden for even the largest residential ISPs.
-Matt
On Thu, Apr 1, 2021 at 2:09 PM Patrick W. Gilmore <patrick@ianai.net> wrote:
Matt:
I am going to disagree with your characterization of how Akamai - and many other CDNs - manage things. First, to be blunt, if you really think Akamai nodes are “sitting idle for weeks” before CoD comes out with a new game, you are clearly confused.
More importantly, I know for a fact Akamai has spent ungodly amounts of money & resources putting content precisely where the ISPs ask them to put it, deliver it over the pipes the ISPs ask them to deliver it, at precisely the capacity the ISPs tell them.
On the other hand, I agree with your characterization of residential broadband. It is ridiculous to expect a neighborhood with 1,000 homes each with 1 Gbps links to have a terabit of uplink capacity. But it also should have a lot more than 10 Gbps, IMHO. Unfortunately, most neighborhoods I have seen are closer to the latter than the former.
Finally, this could quickly devolve into finger pointing. You say the CDNs bear some responsibility? They may well respond that the large broadband providers ask for cash to interconnect - but still require the CDNs to do all the work. The CDNs did not create the content, or tell the users which content to pull. When I pay $NATIONAL_PROVIDER, I expect them to provide me with access to the Internet. Not just to the content that pays that provider.
Personally, I have zero problems with the ISPs saying “give me a cache to put here with this sized uplink” or “please deliver to these users over this xconn / IX / whatever”. I have a huge problem with the ISPs blaming the ISPs for delivering what the ISP’s users request.
Of course, this could all be solved if there were more competition in broadband in the US (and many other countries). But that is a totally different 10,000 post thread (that we have had many dozens of times).
-- TTFN, patrick
On Apr 1, 2021, at 3:53 PM, Matt Erculiani <merculiani@gmail.com> wrote:
Niels,
I think to clarify Jean's point, when you buy a 300mbps circuit, you're paying for 300mbps of *internet *access.
That does not mean that a network should (and in this case small-medium ones simply can't) build all of their capacity to service a large number of customer circuits at line rate at the same time for an extended period, ESPECIALLY to the exact same endpoint. It's just not economically reasonable to expect that. Remember we're talking about residential service here, not enterprise circuits.
Therefore, how do you prevent this spike of [insert large number here] gigabits traversing the network at the same time from causing issues? Build more network? That sounds easy, but there are plenty of legitimate reasons why ISPs can't or don't want to do that, particularly for an event that only occurs once per quarter or so.
Does Akamai bear some burden here to make these rollouts less troublesome for the ISPs they traverse through the last mile(s)? IMO yes, yes they do. When you're doing something new and unprecedented, as Akamai frequently brags about on Twitter, like having rapid, bursty growth of traffic, you need to consider that just because you can generate it, doesn't mean it can be delivered. They've gotta be more sophisticated than a bunch of servers with SSD arrays, ramdisks, and 100 gig interfaces, so there's no excuse for them here to just blindly fill every link they have after sitting idle for weeks/months at a time and expect everything to come out alright and nobody to complain about it.
On Thu, Apr 1, 2021 at 1:21 PM Niels Bakker <niels=nanog@bakker.net> wrote:
* nanog@nanog.org (Jean St-Laurent via NANOG) [Thu 01 Apr 2021, 21:03 CEST]:
An artificial roll out penalty somehow? Probably not at the ISP level, but more at the game level. Well, ISP could also have some mechanisms to reduce the impact or even Akamai could force a progressive roll out.
It's an online game. You can't play the game with outdated assets. You'd not see walls where other players would, for example.
What you're suggesting is the ability of ISPs to market Internet access at a certain speed but not have to deliver it based on conditions they create.
-- Niels.
-- Matt Erculiani ERCUL-ARIN
-- Matt Erculiani ERCUL-ARIN
-- Matt Erculiani ERCUL-ARIN
In terms of dollar flows, yes, the subscriber makes all requests. They make the requests of the ISP and of the game developer\publisher\whatever. However, the game publisher queues those requests. I'm meaning request generically, not a GET request or anything like that. The game publisher that contracts to the CDNs decides when to fulfill those requests, in the big picture. The game publisher is the one that then tells 100 million devices "Content Available". The rate that they do that is at their discretion. Me deciding to download 50 gigs of GIS imagery because I requested it at that moment isn't the same situation as 100 million people downloading COD because the publisher released it. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Tom Beecher" <beecher@beecher.cc> To: "Matt Erculiani" <merculiani@gmail.com> Cc: "North American Operators' Group" <nanog@nanog.org> Sent: Thursday, April 1, 2021 4:04:34 PM Subject: Re: wow, lots of akamai No disrespect taken, or intended back in your direction, but again, I disagree. If thousands of users are downloading 50G files at the same time, it really doesn't matter if they are pulling from a CDN or the origin directly. The volume of traffic still has to be handled. Yes, it's a burden on the ISP, but it's a burden created by the usage created by their subscribers. On Thu, Apr 1, 2021 at 4:57 PM Matt Erculiani < merculiani@gmail.com > wrote: Tom, All due respect, but there is a massive difference between one user downloading 50G and thousands of users each downloading 50G when they all go to play their videogame of choice at around the same time. -Matt On Thu, Apr 1, 2021 at 2:46 PM Tom Beecher <beecher@beecher.cc> wrote: <blockquote> <blockquote> A user sends a few megabytes of request and receives 50 gigs of reply. They aren't DDoSing the network, but they're amplifying a single 50 gig copy they receive from the mothership and turning it into likely tens of terabytes of traffic. Yes, that's a CDN's job, but that volume of legitimate traffic and the very tiny window with which it is transmitted is likely to be a burden for even the largest residential ISPs. I'm sitting at home, and I could send a 50k request for a 50G file right now from a source not fronted by a CDN. What do? My ISP is still has to deliver it to me. The fact that the 50G file does or does not come from a CDN is irrelevant. The CDN just happens to be a point source that a lot of users happen to connect to. CDNs want to have the best performance to users because that's what brings them business. A poorly performing CDN will lose customers to a better performing one. The trend for years has been instead of ISPs investing in infrastructure to effectively handle the traffic that their users request, they turf that to CDNs. In many cases, a CDN will put a cache box in or extend a circuit at a loss to them, because they know if the performance metrics get bad, business will be taken elsewhere, even if the CAUSE of the poor performance is actually at the edge of, or inside , the ISPs network. ISPs in the US can get away with this because their users are captive and rarely have an alternative choice of provider. On Thu, Apr 1, 2021 at 4:33 PM Matt Erculiani < merculiani@gmail.com > wrote: <blockquote> Patrick,
First, to be blunt, if you really think Akamai nodes are “sitting idle for weeks” before CoD comes out with a new game, you are clearly confused.
"Idle" in the sense that when you look at a graph of traffic before and after a large push such as this makes the rest of the week's traffic look like a horizontal line at the bottom, admittedly poor word choice, yes, but far from "confused" as to what CDNs do under relatively normal circumstances. Otherwise very valid points you've raised. Tom,
Akamai, and other CDNs, do not **generate** traffic ; they serve the requests generated by users.
A user sends a few megabytes of request and receives 50 gigs of reply. They aren't DDoSing the network, but they're amplifying a single 50 gig copy they receive from the mothership and turning it into likely tens of terabytes of traffic. Yes, that's a CDN's job, but that volume of legitimate traffic and the very tiny window with which it is transmitted is likely to be a burden for even the largest residential ISPs. -Matt On Thu, Apr 1, 2021 at 2:09 PM Patrick W. Gilmore < patrick@ianai.net > wrote: <blockquote> Matt: I am going to disagree with your characterization of how Akamai - and many other CDNs - manage things. First, to be blunt, if you really think Akamai nodes are “sitting idle for weeks” before CoD comes out with a new game, you are clearly confused. More importantly, I know for a fact Akamai has spent ungodly amounts of money & resources putting content precisely where the ISPs ask them to put it, deliver it over the pipes the ISPs ask them to deliver it, at precisely the capacity the ISPs tell them. On the other hand, I agree with your characterization of residential broadband. It is ridiculous to expect a neighborhood with 1,000 homes each with 1 Gbps links to have a terabit of uplink capacity. But it also should have a lot more than 10 Gbps, IMHO. Unfortunately, most neighborhoods I have seen are closer to the latter than the former. Finally, this could quickly devolve into finger pointing. You say the CDNs bear some responsibility? They may well respond that the large broadband providers ask for cash to interconnect - but still require the CDNs to do all the work. The CDNs did not create the content, or tell the users which content to pull. When I pay $NATIONAL_PROVIDER, I expect them to provide me with access to the Internet. Not just to the content that pays that provider. Personally, I have zero problems with the ISPs saying “give me a cache to put here with this sized uplink” or “please deliver to these users over this xconn / IX / whatever”. I have a huge problem with the ISPs blaming the ISPs for delivering what the ISP’s users request. Of course, this could all be solved if there were more competition in broadband in the US (and many other countries). But that is a totally different 10,000 post thread (that we have had many dozens of times). -- TTFN, patrick <blockquote> On Apr 1, 2021, at 3:53 PM, Matt Erculiani < merculiani@gmail.com > wrote: Niels, I think to clarify Jean's point, when you buy a 300mbps circuit, you're paying for 300mbps of internet access. That does not mean that a network should (and in this case small-medium ones simply can't) build all of their capacity to service a large number of customer circuits at line rate at the same time for an extended period, ESPECIALLY to the exact same endpoint. It's just not economically reasonable to expect that. Remember we're talking about residential service here, not enterprise circuits. Therefore, how do you prevent this spike of [insert large number here] gigabits traversing the network at the same time from causing issues? Build more network? That sounds easy, but there are plenty of legitimate reasons why ISPs can't or don't want to do that, particularly for an event that only occurs once per quarter or so. Does Akamai bear some burden here to make these rollouts less troublesome for the ISPs they traverse through the last mile(s)? IMO yes, yes they do. When you're doing something new and unprecedented, as Akamai frequently brags about on Twitter, like having rapid, bursty growth of traffic, you need to consider that just because you can generate it, doesn't mean it can be delivered. They've gotta be more sophisticated than a bunch of servers with SSD arrays, ramdisks, and 100 gig interfaces, so there's no excuse for them here to just blindly fill every link they have after sitting idle for weeks/months at a time and expect everything to come out alright and nobody to complain about it. On Thu, Apr 1, 2021 at 1:21 PM Niels Bakker <niels= nanog@bakker.net > wrote: <blockquote> * nanog@nanog.org (Jean St-Laurent via NANOG) [Thu 01 Apr 2021, 21:03 CEST]:
An artificial roll out penalty somehow? Probably not at the ISP level, but more at the game level. Well, ISP could also have some mechanisms to reduce the impact or even Akamai could force a progressive roll out.
It's an online game. You can't play the game with outdated assets. You'd not see walls where other players would, for example. What you're suggesting is the ability of ISPs to market Internet access at a certain speed but not have to deliver it based on conditions they create. -- Niels. </blockquote> -- Matt Erculiani ERCUL-ARIN </blockquote> </blockquote> -- Matt Erculiani ERCUL-ARIN </blockquote> </blockquote> -- Matt Erculiani ERCUL-ARIN </blockquote>
* nanog@ics-il.net (Mike Hammett) [Thu 01 Apr 2021, 23:17 CEST]:
However, the game publisher queues those requests. I'm meaning request generically, not a GET request or anything like that. The game publisher that contracts to the CDNs decides when to fulfill those requests, in the big picture. The game publisher is the one that then tells 100 million devices "Content Available". The rate that they do that is at their discretion.
Keep in mind that the publisher doesn't just create a bunch of new assets to give away for free from the goodness of their hearts. Every update comes with new hats, weapon finishes, heroes, trading cards etc. that players are expected to buy for real money. Any delay could kill the hype and impact revenue. There is a lot of "But muh discard counters!" in this thread that is perfectly valid and understandable and that I absolutely relate to, but it's not the only perspective. -- Niels.
Maybe? 6 months? 12 months? Okay, maybe I'll buy it. 36 hours? No. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Niels Bakker" <niels=nanog@bakker.net> To: nanog@nanog.org Sent: Thursday, April 1, 2021 5:40:41 PM Subject: Re: wow, lots of akamai * nanog@ics-il.net (Mike Hammett) [Thu 01 Apr 2021, 23:17 CEST]:
However, the game publisher queues those requests. I'm meaning request generically, not a GET request or anything like that. The game publisher that contracts to the CDNs decides when to fulfill those requests, in the big picture. The game publisher is the one that then tells 100 million devices "Content Available". The rate that they do that is at their discretion.
Keep in mind that the publisher doesn't just create a bunch of new assets to give away for free from the goodness of their hearts. Every update comes with new hats, weapon finishes, heroes, trading cards etc. that players are expected to buy for real money. Any delay could kill the hype and impact revenue. There is a lot of "But muh discard counters!" in this thread that is perfectly valid and understandable and that I absolutely relate to, but it's not the only perspective. -- Niels.
I am sorry, maybe I misunderstand. Matt: Are you arguing the CDNs are at fault because the game companies tell everyone to download simultaneously, and the ISPs sold the users connectivity to do that download? If so, are you really arguing “I sold my users XXX Mbps, but if they try to use it, I want *YOU* to tell them no”? Because that is what it sounds like to me. Imagine a gym sold 10,000 memberships with 10 machines because they figured everyone would sit on their ass. They would be right most of the time - and rake in that sweet, sweet monthly cash for zero effort after the initial sale. But if Oprah or Cher or Biden or some other person famous enough to go by one name tweets “get your ass to the gym!!", does the gym really think getting mad at Oprah is the solution? Or do they expect Oprah to pay for the extra machines they have to buy now? Selling a service you know will not work if everyone uses it simultaneously can be profitable, but there is risk. Do not blame third parties when you lose that bet. -- TTFN, patrick
On Apr 1, 2021, at 5:04 PM, Tom Beecher <beecher@beecher.cc> wrote:
No disrespect taken, or intended back in your direction, but again, I disagree.
If thousands of users are downloading 50G files at the same time, it really doesn't matter if they are pulling from a CDN or the origin directly. The volume of traffic still has to be handled. Yes, it's a burden on the ISP, but it's a burden created by the usage created by their subscribers.
On Thu, Apr 1, 2021 at 4:57 PM Matt Erculiani <merculiani@gmail.com <mailto:merculiani@gmail.com>> wrote: Tom,
All due respect, but there is a massive difference between one user downloading 50G and thousands of users each downloading 50G when they all go to play their videogame of choice at around the same time.
-Matt
On Thu, Apr 1, 2021 at 2:46 PM Tom Beecher <beecher@beecher.cc> wrote: A user sends a few megabytes of request and receives 50 gigs of reply. They aren't DDoSing the network, but they're amplifying a single 50 gig copy they receive from the mothership and turning it into likely tens of terabytes of traffic. Yes, that's a CDN's job, but that volume of legitimate traffic and the very tiny window with which it is transmitted is likely to be a burden for even the largest residential ISPs.
I'm sitting at home, and I could send a 50k request for a 50G file right now from a source not fronted by a CDN. What do? My ISP is still has to deliver it to me. The fact that the 50G file does or does not come from a CDN is irrelevant. The CDN just happens to be a point source that a lot of users happen to connect to.
CDNs want to have the best performance to users because that's what brings them business. A poorly performing CDN will lose customers to a better performing one. The trend for years has been instead of ISPs investing in infrastructure to effectively handle the traffic that their users request, they turf that to CDNs. In many cases, a CDN will put a cache box in or extend a circuit at a loss to them, because they know if the performance metrics get bad, business will be taken elsewhere, even if the CAUSE of the poor performance is actually at the edge of, or inside , the ISPs network.
ISPs in the US can get away with this because their users are captive and rarely have an alternative choice of provider.
On Thu, Apr 1, 2021 at 4:33 PM Matt Erculiani <merculiani@gmail.com <mailto:merculiani@gmail.com>> wrote: Patrick,
First, to be blunt, if you really think Akamai nodes are “sitting idle for weeks” before CoD comes out with a new game, you are clearly confused.
"Idle" in the sense that when you look at a graph of traffic before and after a large push such as this makes the rest of the week's traffic look like a horizontal line at the bottom, admittedly poor word choice, yes, but far from "confused" as to what CDNs do under relatively normal circumstances. Otherwise very valid points you've raised.
Tom,
Akamai, and other CDNs, do not **generate** traffic ; they serve the requests generated by users.
A user sends a few megabytes of request and receives 50 gigs of reply. They aren't DDoSing the network, but they're amplifying a single 50 gig copy they receive from the mothership and turning it into likely tens of terabytes of traffic. Yes, that's a CDN's job, but that volume of legitimate traffic and the very tiny window with which it is transmitted is likely to be a burden for even the largest residential ISPs.
-Matt
On Thu, Apr 1, 2021 at 2:09 PM Patrick W. Gilmore <patrick@ianai.net <mailto:patrick@ianai.net>> wrote: Matt:
I am going to disagree with your characterization of how Akamai - and many other CDNs - manage things. First, to be blunt, if you really think Akamai nodes are “sitting idle for weeks” before CoD comes out with a new game, you are clearly confused.
More importantly, I know for a fact Akamai has spent ungodly amounts of money & resources putting content precisely where the ISPs ask them to put it, deliver it over the pipes the ISPs ask them to deliver it, at precisely the capacity the ISPs tell them.
On the other hand, I agree with your characterization of residential broadband. It is ridiculous to expect a neighborhood with 1,000 homes each with 1 Gbps links to have a terabit of uplink capacity. But it also should have a lot more than 10 Gbps, IMHO. Unfortunately, most neighborhoods I have seen are closer to the latter than the former.
Finally, this could quickly devolve into finger pointing. You say the CDNs bear some responsibility? They may well respond that the large broadband providers ask for cash to interconnect - but still require the CDNs to do all the work. The CDNs did not create the content, or tell the users which content to pull. When I pay $NATIONAL_PROVIDER, I expect them to provide me with access to the Internet. Not just to the content that pays that provider.
Personally, I have zero problems with the ISPs saying “give me a cache to put here with this sized uplink” or “please deliver to these users over this xconn / IX / whatever”. I have a huge problem with the ISPs blaming the ISPs for delivering what the ISP’s users request.
Of course, this could all be solved if there were more competition in broadband in the US (and many other countries). But that is a totally different 10,000 post thread (that we have had many dozens of times).
-- TTFN, patrick
On Apr 1, 2021, at 3:53 PM, Matt Erculiani <merculiani@gmail.com <mailto:merculiani@gmail.com>> wrote:
Niels,
I think to clarify Jean's point, when you buy a 300mbps circuit, you're paying for 300mbps of internet access.
That does not mean that a network should (and in this case small-medium ones simply can't) build all of their capacity to service a large number of customer circuits at line rate at the same time for an extended period, ESPECIALLY to the exact same endpoint. It's just not economically reasonable to expect that. Remember we're talking about residential service here, not enterprise circuits.
Therefore, how do you prevent this spike of [insert large number here] gigabits traversing the network at the same time from causing issues? Build more network? That sounds easy, but there are plenty of legitimate reasons why ISPs can't or don't want to do that, particularly for an event that only occurs once per quarter or so.
Does Akamai bear some burden here to make these rollouts less troublesome for the ISPs they traverse through the last mile(s)? IMO yes, yes they do. When you're doing something new and unprecedented, as Akamai frequently brags about on Twitter, like having rapid, bursty growth of traffic, you need to consider that just because you can generate it, doesn't mean it can be delivered. They've gotta be more sophisticated than a bunch of servers with SSD arrays, ramdisks, and 100 gig interfaces, so there's no excuse for them here to just blindly fill every link they have after sitting idle for weeks/months at a time and expect everything to come out alright and nobody to complain about it.
On Thu, Apr 1, 2021 at 1:21 PM Niels Bakker <niels=nanog@bakker.net <mailto:nanog@bakker.net>> wrote: * nanog@nanog.org <mailto:nanog@nanog.org> (Jean St-Laurent via NANOG) [Thu 01 Apr 2021, 21:03 CEST]:
An artificial roll out penalty somehow? Probably not at the ISP level, but more at the game level. Well, ISP could also have some mechanisms to reduce the impact or even Akamai could force a progressive roll out.
It's an online game. You can't play the game with outdated assets. You'd not see walls where other players would, for example.
What you're suggesting is the ability of ISPs to market Internet access at a certain speed but not have to deliver it based on conditions they create.
-- Niels.
-- Matt Erculiani ERCUL-ARIN
-- Matt Erculiani ERCUL-ARIN
-- Matt Erculiani ERCUL-ARIN
Patrick,
Matt: Are you arguing the CDNs are at fault because the game companies tell everyone to download simultaneously, and the ISPs sold the users connectivity to do that download?
While a gross oversimplification, yes, that's basically what I'm saying; I know it may not be a popular opinion, but I stand by it. There aren't any villains here though, just lots of good suggestions in this thread to make the internet work better for everyone, without spending large swaths of money to cover the demand of an infrequent, large, update for a single game. CNDs do, however, have a responsibility to be good netizens and get this data out in a manner that doesn't cause disruption. They know the technical challenges of distributing that much data to the masses, the game company does not, that's why they outsourced it to a CDN. If the CDN knows what the gaming company is asking for is pushing the limits of our current infrastructure, they have a responsibility to relay those limitations that are outside of their control to their customer, as any responsible vendor should. Instead, there may be an element of "oh yeah sure, we can do that" or "the customer is always right" going on here and modern limitations are being disregarded. The idea behind the internet is not that every user can always have their entire capacity available for a single destination *regardless of what everyone else is doing *(and *especially *if they're all going to the same place too), the user has purchased that capacity into their provider's network as a whole, gaining access to all of their connections to all of the various endpoints on the internet at a backbone and peering capacity that is economically viable given normal peak demand with some cushion built-in for redundancy. If that's the desire to have full capacity available to Akamai available at all times, then everyone needs dedicated P2P circuits direct to Akamai, but that's not practical. If you own an ISP and you're not oversubscribing, you're not making money, period. To use your analogy, if you've ever been to a gym in January, you've seen a similar phenomenon first-hand. There aren't enough machines for everyone, and the gym isn't going to add them because this is a once-a-year thing and it goes away after a few weeks when many people get tired of fulfilling their new year's resolutions. Why should the gym limit its sales to exactly the capacity it has available (or add a lot more machines), when it knows that for the overwhelming majority of the year, there will always be dozens of empty machines across the floor? If Akamai was doing these updates more frequently (weekly for example) then sure, it is on the ISP to augment, because this has become the "new normal". But these updates for a single game that happen once per quarter are hardly able to be considered normal. Sure, some day 50GB updates will be the norm, but that's not today, and when it is, somebody else will be pushing out 250GB updates quarterly. This problem isn't going away soon, and it can't be fixed permanently by just adding more capacity, it's a complex technical challenge that CDN's ought to give some more thought, and game publishers should start considering too. -Matt On Thu, Apr 1, 2021 at 3:30 PM Patrick W. Gilmore <patrick@ianai.net> wrote:
I am sorry, maybe I misunderstand.
Matt: Are you arguing the CDNs are at fault because the game companies tell everyone to download simultaneously, and the ISPs sold the users connectivity to do that download?
If so, are you really arguing “I sold my users XXX Mbps, but if they try to use it, I want *YOU* to tell them no”? Because that is what it sounds like to me.
Imagine a gym sold 10,000 memberships with 10 machines because they figured everyone would sit on their ass. They would be right most of the time - and rake in that sweet, sweet monthly cash for zero effort after the initial sale. But if Oprah or Cher or Biden or some other person famous enough to go by one name tweets “get your ass to the gym!!", does the gym really think getting mad at Oprah is the solution? Or do they expect Oprah to pay for the extra machines they have to buy now?
Selling a service you know will not work if everyone uses it simultaneously can be profitable, but there is risk. Do not blame third parties when you lose that bet.
-- TTFN, patrick
On Apr 1, 2021, at 5:04 PM, Tom Beecher <beecher@beecher.cc> wrote:
No disrespect taken, or intended back in your direction, but again, I disagree.
If thousands of users are downloading 50G files at the same time, it really doesn't matter if they are pulling from a CDN or the origin directly. The volume of traffic still has to be handled. Yes, it's a burden on the ISP, but it's a burden created by the usage created by their subscribers.
On Thu, Apr 1, 2021 at 4:57 PM Matt Erculiani <merculiani@gmail.com> wrote:
Tom,
All due respect, but there is a massive difference between one user downloading 50G and thousands of users each downloading 50G when they all go to play their videogame of choice at around the same time.
-Matt
On Thu, Apr 1, 2021 at 2:46 PM Tom Beecher <beecher@beecher.cc> wrote:
A user sends a few megabytes of request and receives 50 gigs of reply.
They aren't DDoSing the network, but they're amplifying a single 50 gig copy they receive from the mothership and turning it into likely tens of terabytes of traffic. Yes, that's a CDN's job, but that volume of legitimate traffic and the very tiny window with which it is transmitted is likely to be a burden for even the largest residential ISPs.
I'm sitting at home, and I could send a 50k request for a 50G file right now from a source not fronted by a CDN. What do? My ISP is still has to deliver it to me. The fact that the 50G file does or does not come from a CDN is irrelevant. The CDN just happens to be a point source that a lot of users happen to connect to.
CDNs want to have the best performance to users because that's what brings them business. A poorly performing CDN will lose customers to a better performing one. The trend for years has been instead of ISPs investing in infrastructure to effectively handle the traffic that their users request, they turf that to CDNs. In many cases, a CDN will put a cache box in or extend a circuit at a loss to them, because they know if the performance metrics get bad, business will be taken elsewhere, even if the CAUSE of the poor performance is actually at the edge of, or inside , the ISPs network.
ISPs in the US can get away with this because their users are captive and rarely have an alternative choice of provider.
On Thu, Apr 1, 2021 at 4:33 PM Matt Erculiani <merculiani@gmail.com> wrote:
Patrick,
First, to be blunt, if you really think Akamai nodes are “sitting idle for weeks” before CoD comes out with a new game, you are clearly confused.
"Idle" in the sense that when you look at a graph of traffic before and after a large push such as this makes the rest of the week's traffic look like a horizontal line at the bottom, admittedly poor word choice, yes, but far from "confused" as to what CDNs do under relatively normal circumstances. Otherwise very valid points you've raised.
Tom,
Akamai, and other CDNs, do not **generate** traffic ; they serve the requests generated by users.
A user sends a few megabytes of request and receives 50 gigs of reply. They aren't DDoSing the network, but they're amplifying a single 50 gig copy they receive from the mothership and turning it into likely tens of terabytes of traffic. Yes, that's a CDN's job, but that volume of legitimate traffic and the very tiny window with which it is transmitted is likely to be a burden for even the largest residential ISPs.
-Matt
On Thu, Apr 1, 2021 at 2:09 PM Patrick W. Gilmore <patrick@ianai.net> wrote:
Matt:
I am going to disagree with your characterization of how Akamai - and many other CDNs - manage things. First, to be blunt, if you really think Akamai nodes are “sitting idle for weeks” before CoD comes out with a new game, you are clearly confused.
More importantly, I know for a fact Akamai has spent ungodly amounts of money & resources putting content precisely where the ISPs ask them to put it, deliver it over the pipes the ISPs ask them to deliver it, at precisely the capacity the ISPs tell them.
On the other hand, I agree with your characterization of residential broadband. It is ridiculous to expect a neighborhood with 1,000 homes each with 1 Gbps links to have a terabit of uplink capacity. But it also should have a lot more than 10 Gbps, IMHO. Unfortunately, most neighborhoods I have seen are closer to the latter than the former.
Finally, this could quickly devolve into finger pointing. You say the CDNs bear some responsibility? They may well respond that the large broadband providers ask for cash to interconnect - but still require the CDNs to do all the work. The CDNs did not create the content, or tell the users which content to pull. When I pay $NATIONAL_PROVIDER, I expect them to provide me with access to the Internet. Not just to the content that pays that provider.
Personally, I have zero problems with the ISPs saying “give me a cache to put here with this sized uplink” or “please deliver to these users over this xconn / IX / whatever”. I have a huge problem with the ISPs blaming the ISPs for delivering what the ISP’s users request.
Of course, this could all be solved if there were more competition in broadband in the US (and many other countries). But that is a totally different 10,000 post thread (that we have had many dozens of times).
-- TTFN, patrick
On Apr 1, 2021, at 3:53 PM, Matt Erculiani <merculiani@gmail.com> wrote:
Niels,
I think to clarify Jean's point, when you buy a 300mbps circuit, you're paying for 300mbps of *internet *access.
That does not mean that a network should (and in this case small-medium ones simply can't) build all of their capacity to service a large number of customer circuits at line rate at the same time for an extended period, ESPECIALLY to the exact same endpoint. It's just not economically reasonable to expect that. Remember we're talking about residential service here, not enterprise circuits.
Therefore, how do you prevent this spike of [insert large number here] gigabits traversing the network at the same time from causing issues? Build more network? That sounds easy, but there are plenty of legitimate reasons why ISPs can't or don't want to do that, particularly for an event that only occurs once per quarter or so.
Does Akamai bear some burden here to make these rollouts less troublesome for the ISPs they traverse through the last mile(s)? IMO yes, yes they do. When you're doing something new and unprecedented, as Akamai frequently brags about on Twitter, like having rapid, bursty growth of traffic, you need to consider that just because you can generate it, doesn't mean it can be delivered. They've gotta be more sophisticated than a bunch of servers with SSD arrays, ramdisks, and 100 gig interfaces, so there's no excuse for them here to just blindly fill every link they have after sitting idle for weeks/months at a time and expect everything to come out alright and nobody to complain about it.
On Thu, Apr 1, 2021 at 1:21 PM Niels Bakker <niels=nanog@bakker.net> wrote:
* nanog@nanog.org (Jean St-Laurent via NANOG) [Thu 01 Apr 2021, 21:03 CEST]: >An artificial roll out penalty somehow? Probably not at the ISP >level, but more at the game level. Well, ISP could also have some >mechanisms to reduce the impact or even Akamai could force a >progressive roll out.
It's an online game. You can't play the game with outdated assets. You'd not see walls where other players would, for example.
What you're suggesting is the ability of ISPs to market Internet access at a certain speed but not have to deliver it based on conditions they create.
-- Niels.
-- Matt Erculiani ERCUL-ARIN
-- Matt Erculiani ERCUL-ARIN
-- Matt Erculiani ERCUL-ARIN
-- Matt Erculiani ERCUL-ARIN
I am a bit worried about phrases like "If Akamai was doing these updates more frequently”. Akamai does not decide these things. You may as well say “if the fiber carriers sent the bits over several hours instead of all at once.” And please do not say you were just using shorthand. You have blamed the CDNs and Akamai by name several times in this thread. I know first hand that Akamai has explained to large customers the possible problems with multi-GB updates to millions of users simultaneously. If the game company does not care, then I do not see what you expect the CDN to do about it. Most CDNs do their best to deliver traffic optimally. It is in their own best interest. They want to avoid dropped packets even more than you do. If you do not like the way a CDN will deliver the traffic, talk to them. Perhaps there is a compromise, perhaps not. But most of them will at least kick ideas around to see what can be done. And after all that, I still do not see what we are arguing about? You want the game companies to change their business model, but you do not want to change yours. Please do not say something like “but if they just ….” Unless you want the game companies to say “but if the ISPs just ….” Either way, stop trying to say someone else - the game provider, the CDN, the user, whoever - should change their model or spend their money to keep your business above water. -- TTFN, patrick P.S. It is not 1995. “The Internet” is a bit more mature, and users expect a bit more.
On Apr 1, 2021, at 6:27 PM, Matt Erculiani <merculiani@gmail.com> wrote:
Patrick,
Matt: Are you arguing the CDNs are at fault because the game companies tell everyone to download simultaneously, and the ISPs sold the users connectivity to do that download?
While a gross oversimplification, yes, that's basically what I'm saying; I know it may not be a popular opinion, but I stand by it. There aren't any villains here though, just lots of good suggestions in this thread to make the internet work better for everyone, without spending large swaths of money to cover the demand of an infrequent, large, update for a single game.
CNDs do, however, have a responsibility to be good netizens and get this data out in a manner that doesn't cause disruption. They know the technical challenges of distributing that much data to the masses, the game company does not, that's why they outsourced it to a CDN. If the CDN knows what the gaming company is asking for is pushing the limits of our current infrastructure, they have a responsibility to relay those limitations that are outside of their control to their customer, as any responsible vendor should. Instead, there may be an element of "oh yeah sure, we can do that" or "the customer is always right" going on here and modern limitations are being disregarded.
The idea behind the internet is not that every user can always have their entire capacity available for a single destination regardless of what everyone else is doing (and especially if they're all going to the same place too), the user has purchased that capacity into their provider's network as a whole, gaining access to all of their connections to all of the various endpoints on the internet at a backbone and peering capacity that is economically viable given normal peak demand with some cushion built-in for redundancy. If that's the desire to have full capacity available to Akamai available at all times, then everyone needs dedicated P2P circuits direct to Akamai, but that's not practical.
If you own an ISP and you're not oversubscribing, you're not making money, period. To use your analogy, if you've ever been to a gym in January, you've seen a similar phenomenon first-hand. There aren't enough machines for everyone, and the gym isn't going to add them because this is a once-a-year thing and it goes away after a few weeks when many people get tired of fulfilling their new year's resolutions. Why should the gym limit its sales to exactly the capacity it has available (or add a lot more machines), when it knows that for the overwhelming majority of the year, there will always be dozens of empty machines across the floor?
If Akamai was doing these updates more frequently (weekly for example) then sure, it is on the ISP to augment, because this has become the "new normal". But these updates for a single game that happen once per quarter are hardly able to be considered normal. Sure, some day 50GB updates will be the norm, but that's not today, and when it is, somebody else will be pushing out 250GB updates quarterly. This problem isn't going away soon, and it can't be fixed permanently by just adding more capacity, it's a complex technical challenge that CDN's ought to give some more thought, and game publishers should start considering too.
-Matt
On Thu, Apr 1, 2021 at 3:30 PM Patrick W. Gilmore <patrick@ianai.net <mailto:patrick@ianai.net>> wrote: I am sorry, maybe I misunderstand.
Matt: Are you arguing the CDNs are at fault because the game companies tell everyone to download simultaneously, and the ISPs sold the users connectivity to do that download?
If so, are you really arguing “I sold my users XXX Mbps, but if they try to use it, I want *YOU* to tell them no”? Because that is what it sounds like to me.
Imagine a gym sold 10,000 memberships with 10 machines because they figured everyone would sit on their ass. They would be right most of the time - and rake in that sweet, sweet monthly cash for zero effort after the initial sale. But if Oprah or Cher or Biden or some other person famous enough to go by one name tweets “get your ass to the gym!!", does the gym really think getting mad at Oprah is the solution? Or do they expect Oprah to pay for the extra machines they have to buy now?
Selling a service you know will not work if everyone uses it simultaneously can be profitable, but there is risk. Do not blame third parties when you lose that bet.
-- TTFN, patrick
On Apr 1, 2021, at 5:04 PM, Tom Beecher <beecher@beecher.cc <mailto:beecher@beecher.cc>> wrote:
No disrespect taken, or intended back in your direction, but again, I disagree.
If thousands of users are downloading 50G files at the same time, it really doesn't matter if they are pulling from a CDN or the origin directly. The volume of traffic still has to be handled. Yes, it's a burden on the ISP, but it's a burden created by the usage created by their subscribers.
On Thu, Apr 1, 2021 at 4:57 PM Matt Erculiani <merculiani@gmail.com <mailto:merculiani@gmail.com>> wrote: Tom,
All due respect, but there is a massive difference between one user downloading 50G and thousands of users each downloading 50G when they all go to play their videogame of choice at around the same time.
-Matt
On Thu, Apr 1, 2021 at 2:46 PM Tom Beecher <beecher@beecher.cc <mailto:beecher@beecher.cc>> wrote: A user sends a few megabytes of request and receives 50 gigs of reply. They aren't DDoSing the network, but they're amplifying a single 50 gig copy they receive from the mothership and turning it into likely tens of terabytes of traffic. Yes, that's a CDN's job, but that volume of legitimate traffic and the very tiny window with which it is transmitted is likely to be a burden for even the largest residential ISPs.
I'm sitting at home, and I could send a 50k request for a 50G file right now from a source not fronted by a CDN. What do? My ISP is still has to deliver it to me. The fact that the 50G file does or does not come from a CDN is irrelevant. The CDN just happens to be a point source that a lot of users happen to connect to.
CDNs want to have the best performance to users because that's what brings them business. A poorly performing CDN will lose customers to a better performing one. The trend for years has been instead of ISPs investing in infrastructure to effectively handle the traffic that their users request, they turf that to CDNs. In many cases, a CDN will put a cache box in or extend a circuit at a loss to them, because they know if the performance metrics get bad, business will be taken elsewhere, even if the CAUSE of the poor performance is actually at the edge of, or inside , the ISPs network.
ISPs in the US can get away with this because their users are captive and rarely have an alternative choice of provider.
On Thu, Apr 1, 2021 at 4:33 PM Matt Erculiani <merculiani@gmail.com <mailto:merculiani@gmail.com>> wrote: Patrick,
First, to be blunt, if you really think Akamai nodes are “sitting idle for weeks” before CoD comes out with a new game, you are clearly confused.
"Idle" in the sense that when you look at a graph of traffic before and after a large push such as this makes the rest of the week's traffic look like a horizontal line at the bottom, admittedly poor word choice, yes, but far from "confused" as to what CDNs do under relatively normal circumstances. Otherwise very valid points you've raised.
Tom,
Akamai, and other CDNs, do not **generate** traffic ; they serve the requests generated by users.
A user sends a few megabytes of request and receives 50 gigs of reply. They aren't DDoSing the network, but they're amplifying a single 50 gig copy they receive from the mothership and turning it into likely tens of terabytes of traffic. Yes, that's a CDN's job, but that volume of legitimate traffic and the very tiny window with which it is transmitted is likely to be a burden for even the largest residential ISPs.
-Matt
On Thu, Apr 1, 2021 at 2:09 PM Patrick W. Gilmore <patrick@ianai.net <mailto:patrick@ianai.net>> wrote: Matt:
I am going to disagree with your characterization of how Akamai - and many other CDNs - manage things. First, to be blunt, if you really think Akamai nodes are “sitting idle for weeks” before CoD comes out with a new game, you are clearly confused.
More importantly, I know for a fact Akamai has spent ungodly amounts of money & resources putting content precisely where the ISPs ask them to put it, deliver it over the pipes the ISPs ask them to deliver it, at precisely the capacity the ISPs tell them.
On the other hand, I agree with your characterization of residential broadband. It is ridiculous to expect a neighborhood with 1,000 homes each with 1 Gbps links to have a terabit of uplink capacity. But it also should have a lot more than 10 Gbps, IMHO. Unfortunately, most neighborhoods I have seen are closer to the latter than the former.
Finally, this could quickly devolve into finger pointing. You say the CDNs bear some responsibility? They may well respond that the large broadband providers ask for cash to interconnect - but still require the CDNs to do all the work. The CDNs did not create the content, or tell the users which content to pull. When I pay $NATIONAL_PROVIDER, I expect them to provide me with access to the Internet. Not just to the content that pays that provider.
Personally, I have zero problems with the ISPs saying “give me a cache to put here with this sized uplink” or “please deliver to these users over this xconn / IX / whatever”. I have a huge problem with the ISPs blaming the ISPs for delivering what the ISP’s users request.
Of course, this could all be solved if there were more competition in broadband in the US (and many other countries). But that is a totally different 10,000 post thread (that we have had many dozens of times).
-- TTFN, patrick
On Apr 1, 2021, at 3:53 PM, Matt Erculiani <merculiani@gmail.com <mailto:merculiani@gmail.com>> wrote:
Niels,
I think to clarify Jean's point, when you buy a 300mbps circuit, you're paying for 300mbps of internet access.
That does not mean that a network should (and in this case small-medium ones simply can't) build all of their capacity to service a large number of customer circuits at line rate at the same time for an extended period, ESPECIALLY to the exact same endpoint. It's just not economically reasonable to expect that. Remember we're talking about residential service here, not enterprise circuits.
Therefore, how do you prevent this spike of [insert large number here] gigabits traversing the network at the same time from causing issues? Build more network? That sounds easy, but there are plenty of legitimate reasons why ISPs can't or don't want to do that, particularly for an event that only occurs once per quarter or so.
Does Akamai bear some burden here to make these rollouts less troublesome for the ISPs they traverse through the last mile(s)? IMO yes, yes they do. When you're doing something new and unprecedented, as Akamai frequently brags about on Twitter, like having rapid, bursty growth of traffic, you need to consider that just because you can generate it, doesn't mean it can be delivered. They've gotta be more sophisticated than a bunch of servers with SSD arrays, ramdisks, and 100 gig interfaces, so there's no excuse for them here to just blindly fill every link they have after sitting idle for weeks/months at a time and expect everything to come out alright and nobody to complain about it.
On Thu, Apr 1, 2021 at 1:21 PM Niels Bakker <niels=nanog@bakker.net <mailto:nanog@bakker.net>> wrote: * nanog@nanog.org <mailto:nanog@nanog.org> (Jean St-Laurent via NANOG) [Thu 01 Apr 2021, 21:03 CEST]:
An artificial roll out penalty somehow? Probably not at the ISP level, but more at the game level. Well, ISP could also have some mechanisms to reduce the impact or even Akamai could force a progressive roll out.
It's an online game. You can't play the game with outdated assets. You'd not see walls where other players would, for example.
What you're suggesting is the ability of ISPs to market Internet access at a certain speed but not have to deliver it based on conditions they create.
-- Niels.
-- Matt Erculiani ERCUL-ARIN
-- Matt Erculiani ERCUL-ARIN
-- Matt Erculiani ERCUL-ARIN
-- Matt Erculiani ERCUL-ARIN
* patrick@ianai.net (Patrick W. Gilmore) [Fri 02 Apr 2021, 01:01 CEST]:
I know first hand that Akamai has explained to large customers the possible problems with multi-GB updates to millions of users simultaneously. If the game company does not care, then I do not see what you expect the CDN to do about it.
Thanks, Patrick. In fact, Akamai has been a partner in this dialogue: https://blogs.akamai.com/2020/03/working-together-to-manage-global-internet-... One of the positive results is described here: https://www.callofduty.com/blog/2021/03/Season-Two-Reloaded-Warzone-File-Siz... "Enhancements to the overall content management system has been made possible through data optimization and streamlining content packs needed for individual game modes. This will come after a larger than usual, one-time update for Season Two Reloaded, which will include these optimizations and is necessary in order to reduce the overall footprint; future patch sizes for Modern Warfare and Warzone are expected to be smaller than the one set to release on March 30 at 11PM PST." So the ISPs as well as the CDNs and the players are being listened to by at least this publisher. -- Niels.
On 4/2/21 00:56, Patrick W. Gilmore wrote:
And after all that, I still do not see what we are arguing about? You want the game companies to change their business model, but you do not want to change yours. Please do not say something like “but if they just ….” Unless you want the game companies to say “but if the ISPs just ….” Either way, stop trying to say someone else - the game provider, the CDN, the user, whoever - should change their model or spend their money to keep your business above water.
I know - don't sign up customers who smell like they have gaming consoles, gaming PC's, or gaming hand-held devices :-). #ProblemSolved Mark.
This is not actually (as in yes it does matter) the case, if a file comes from a CDN it is often a close and low latency source that will run up to very high speeds. For example in our case we connect to local peering exchanges (or PNI’s/local caches) at 100G or Nx10G with latency to the end user in the 1-30ms range resulting in very large peaks of local backhaul traffic. If a file is delivers from source or from remote CDN’s/exchanges these are located in other countries with between 25ms (New Zealand to Australia) and 130-200ms (New Zealand to LA/SJC or Singapore) latency, this results in a much slower and normally barely noticeable traffic blip. Yes as an ISP we need to carry the traffic in both cases but the first case can result in a 20-30% local backhaul increase for a couple of hours and in the second case its just BAU traffic for a day or two. Local CDN is obviously the better option for cost and the consumer, but you certainly do notice the traffic in local backhaul. From: NANOG <nanog-bounces+tony=wicks.co.nz@nanog.org> On Behalf Of Tom Beecher Sent: Friday, 2 April 2021 10:05 am To: Matt Erculiani <merculiani@gmail.com> Cc: North American Operators' Group <nanog@nanog.org> Subject: Re: wow, lots of akamai If thousands of users are downloading 50G files at the same time, it really doesn't matter if they are pulling from a CDN or the origin directly. The volume of traffic still has to be handled. Yes, it's a burden on the ISP, but it's a burden created by the usage created by their subscribers.
Just so I am clear, you are saying “I would rather have it come over my undersea cables than from inside the datacenter”? And you are assuming TCP transport. -- TTFN, patrick
On Apr 1, 2021, at 6:23 PM, Tony Wicks <tony@wicks.co.nz> wrote:
This is not actually (as in yes it does matter) the case, if a file comes from a CDN it is often a close and low latency source that will run up to very high speeds. For example in our case we connect to local peering exchanges (or PNI’s/local caches) at 100G or Nx10G with latency to the end user in the 1-30ms range resulting in very large peaks of local backhaul traffic. If a file is delivers from source or from remote CDN’s/exchanges these are located in other countries with between 25ms (New Zealand to Australia) and 130-200ms (New Zealand to LA/SJC or Singapore) latency, this results in a much slower and normally barely noticeable traffic blip. Yes as an ISP we need to carry the traffic in both cases but the first case can result in a 20-30% local backhaul increase for a couple of hours and in the second case its just BAU traffic for a day or two. Local CDN is obviously the better option for cost and the consumer, but you certainly do notice the traffic in local backhaul.
From: NANOG <nanog-bounces+tony=wicks.co.nz@nanog.org <mailto:nanog-bounces+tony=wicks.co.nz@nanog.org>> On Behalf Of Tom Beecher Sent: Friday, 2 April 2021 10:05 am To: Matt Erculiani <merculiani@gmail.com <mailto:merculiani@gmail.com>> Cc: North American Operators' Group <nanog@nanog.org <mailto:nanog@nanog.org>> Subject: Re: wow, lots of akamai
If thousands of users are downloading 50G files at the same time, it really doesn't matter if they are pulling from a CDN or the origin directly. The volume of traffic still has to be handled. Yes, it's a burden on the ISP, but it's a burden created by the usage created by their subscribers.
No absolutely not, having the traffic coming from local CDN’s and the shorter but higher traffic is very much preferred. My comment was just to point out that yes there is a significant difference on ISP traffic between delivery via CDN/PNI/Peering than transit as in our case transit is a long way away. Local backhaul is plentiful and relatively cheap where as subsea wavelengths are extremely expensive and require months of planning. I’m not assuming transport just going on real world traffic affects. From: NANOG <nanog-bounces+tony=wicks.co.nz@nanog.org> On Behalf Of Patrick W. Gilmore Sent: Friday, 2 April 2021 11:32 am To: North American Operators' Group <nanog@nanog.org> Subject: Re: wow, lots of akamai Just so I am clear, you are saying “I would rather have it come over my undersea cables than from inside the datacenter”? And you are assuming TCP transport.
On 2 Apr 2021, at 11:47, Mark Tinka <mark@tinka.africa> wrote:
On 4/2/21 01:41, Tony Wicks wrote:
Local backhaul is plentiful and relatively cheap where as subsea wavelengths are extremely expensive and require months of planning.
Funny, it's the exact opposite for us. Yup, it is...
Mark.
Darwin-.
Does Akamai bear some burden here to make these rollouts less troublesome for the ISPs they traverse through the last mile(s)? IMO yes, yes they do. When you're doing something new and unprecedented, as Akamai frequently brags about on Twitter, like having rapid, bursty growth of traffic, you need to consider that just because you can generate it, doesn't mean it can be delivered.
Akamai, and other CDNs, do not **generate** traffic ; they serve the requests generated by users. On Thu, Apr 1, 2021 at 3:54 PM Matt Erculiani <merculiani@gmail.com> wrote:
Niels,
I think to clarify Jean's point, when you buy a 300mbps circuit, you're paying for 300mbps of *internet *access.
That does not mean that a network should (and in this case small-medium ones simply can't) build all of their capacity to service a large number of customer circuits at line rate at the same time for an extended period, ESPECIALLY to the exact same endpoint. It's just not economically reasonable to expect that. Remember we're talking about residential service here, not enterprise circuits.
Therefore, how do you prevent this spike of [insert large number here] gigabits traversing the network at the same time from causing issues? Build more network? That sounds easy, but there are plenty of legitimate reasons why ISPs can't or don't want to do that, particularly for an event that only occurs once per quarter or so.
Does Akamai bear some burden here to make these rollouts less troublesome for the ISPs they traverse through the last mile(s)? IMO yes, yes they do. When you're doing something new and unprecedented, as Akamai frequently brags about on Twitter, like having rapid, bursty growth of traffic, you need to consider that just because you can generate it, doesn't mean it can be delivered. They've gotta be more sophisticated than a bunch of servers with SSD arrays, ramdisks, and 100 gig interfaces, so there's no excuse for them here to just blindly fill every link they have after sitting idle for weeks/months at a time and expect everything to come out alright and nobody to complain about it.
On Thu, Apr 1, 2021 at 1:21 PM Niels Bakker <niels=nanog@bakker.net> wrote:
* nanog@nanog.org (Jean St-Laurent via NANOG) [Thu 01 Apr 2021, 21:03 CEST]:
An artificial roll out penalty somehow? Probably not at the ISP level, but more at the game level. Well, ISP could also have some mechanisms to reduce the impact or even Akamai could force a progressive roll out.
It's an online game. You can't play the game with outdated assets. You'd not see walls where other players would, for example.
What you're suggesting is the ability of ISPs to market Internet access at a certain speed but not have to deliver it based on conditions they create.
-- Niels.
-- Matt Erculiani ERCUL-ARIN
They add a cookie. This generate traffic From: NANOG <nanog-bounces+jean=ddostest.me@nanog.org> On Behalf Of Tom Beecher Sent: April 1, 2021 4:12 PM To: Matt Erculiani <merculiani@gmail.com> Cc: nanog@nanog.org list <nanog@nanog.org> Subject: Re: wow, lots of akamai Does Akamai bear some burden here to make these rollouts less troublesome for the ISPs they traverse through the last mile(s)? IMO yes, yes they do. When you're doing something new and unprecedented, as Akamai frequently brags about on Twitter, like having rapid, bursty growth of traffic, you need to consider that just because you can generate it, doesn't mean it can be delivered. Akamai, and other CDNs, do not **generate** traffic ; they serve the requests generated by users. On Thu, Apr 1, 2021 at 3:54 PM Matt Erculiani <merculiani@gmail.com <mailto:merculiani@gmail.com> > wrote: Niels, I think to clarify Jean's point, when you buy a 300mbps circuit, you're paying for 300mbps of internet access. That does not mean that a network should (and in this case small-medium ones simply can't) build all of their capacity to service a large number of customer circuits at line rate at the same time for an extended period, ESPECIALLY to the exact same endpoint. It's just not economically reasonable to expect that. Remember we're talking about residential service here, not enterprise circuits. Therefore, how do you prevent this spike of [insert large number here] gigabits traversing the network at the same time from causing issues? Build more network? That sounds easy, but there are plenty of legitimate reasons why ISPs can't or don't want to do that, particularly for an event that only occurs once per quarter or so. Does Akamai bear some burden here to make these rollouts less troublesome for the ISPs they traverse through the last mile(s)? IMO yes, yes they do. When you're doing something new and unprecedented, as Akamai frequently brags about on Twitter, like having rapid, bursty growth of traffic, you need to consider that just because you can generate it, doesn't mean it can be delivered. They've gotta be more sophisticated than a bunch of servers with SSD arrays, ramdisks, and 100 gig interfaces, so there's no excuse for them here to just blindly fill every link they have after sitting idle for weeks/months at a time and expect everything to come out alright and nobody to complain about it. On Thu, Apr 1, 2021 at 1:21 PM Niels Bakker <niels=nanog@bakker.net <mailto:nanog@bakker.net> > wrote: * nanog@nanog.org <mailto:nanog@nanog.org> (Jean St-Laurent via NANOG) [Thu 01 Apr 2021, 21:03 CEST]:
An artificial roll out penalty somehow? Probably not at the ISP level, but more at the game level. Well, ISP could also have some mechanisms to reduce the impact or even Akamai could force a progressive roll out.
It's an online game. You can't play the game with outdated assets. You'd not see walls where other players would, for example. What you're suggesting is the ability of ISPs to market Internet access at a certain speed but not have to deliver it based on conditions they create. -- Niels. -- Matt Erculiani ERCUL-ARIN
There are a couple things going on that all combine together. - Competition between CDNs has pushed $/byte numbers down a lot. (Good or bad, depending on which side you're on. :) ) - Game developers are under constant pressure to deliver content to users quicker - Games are graphically much higher resolution and multi resolution, which means more assets that don't compress well. The only real pressure on a developer to shrink their file sized comes from users running out of disk space on their consoles. Otherwise it's cheaper to just pay for the content delivery than hire more developers to improve the file sizes. On Thu, Apr 1, 2021 at 3:02 PM Jean St-Laurent via NANOG <nanog@nanog.org> wrote:
I remembered working for a big ISP in Europe offering cable tv + internet with +20M subscribers
Every time there was a huge power outage in major cities, all tv`s would go off at the same time. I don`t have stats on power grid stability in Europe Vs N/A.
The problem, was when the power was coming back in big cities, all the tv subscribers would come back online at the exact same second or minute. More or less the same 2 or 3 minutes.
What happened is that it would create a kind of internal DDoS and they would all timed out and give a weird error message. Something very useful like Error Code 0x8098808 Please call our support line at this phone number.
The server sysadmins would go on a panic because all systems were overloaded. They often needed to do overtime because DB crashed, key servers there crashed, DB here crashed, whatever... there was always something crashing. This was before the cloud when you could just push a slider and have tons of VMs or containers to absorb the load in real time. (in my dream)
This would every time create frustration from the clients, the help desk, the support teams and also the upper management. Every time the teams were really tired after that. It was draining juice.
Anyway, after some years of talking internally (red tape), we finally managed to install a random artificial penalty in the setup boxes when they boot after a power outage. Nothing like 20 minutes, but just enough to spread the load over a longer period of time. For the end user, it went transparent for them because, if the setup box would boot in 206 seconds instead of the super aggressive 34 seconds, well it booted and they could watch tv. Vs
my system is totally frozen and it`s been like that for 20 minutes with weird messages because all your systems are down and the error msg said to call the help desk.
This simple change to add 3 lines of code to add a random artificial boot penalty of few seconds, completely solve the problem. This way, when a city would black out, we wouldn't be self DDoS, because the systems would slowly rampup. The setup boxes would all reboot but, wait randomly before asking for the DRM package to unlock the cable TV service and validate whether billing is right.
I`m no Call of Duty expert nor Akamai, but it's been many times that I observe the same question here:
What's happening? Call of Duty! Okay.
Would a kind of throttle help here?
An artificial roll out penalty somehow? Probably not at the ISP level, but more at the game level. Well, ISP could also have some mechanisms to reduce the impact or even Akamai could force a progressive roll out.
I`m not sure that the proposed solutions could work, but it seems to impact NANOG frequently and/or at least generate a call overnight/weekend. It seems to also happens just before long holidays when operations are sometimes on reduce personnel.
Are big games roll out really impacting NANOG? or it's more a: Hey I was curious what happened and I thought to ask here on NANOG?
#JustCurious
Jean
-----Original Message----- From: NANOG <nanog-bounces+jean=ddostest.me@nanog.org> On Behalf Of aaron1@gvtc.com Sent: April 1, 2021 12:12 PM To: 'Jared Mauch' <jared@puck.nether.net>; 'Töma Gavrichenkov' < ximaera@gmail.com> Cc: 'NANOG' <nanog@nanog.org> Subject: RE: wow, lots of akamai
Gaming update... I had a feeling. Thanks for the feedback folks.
Thanks Jared, it's running well, before, during and after. We have a lot of capacity there.
-Aaron
Ummmm, throw bandwidth at it. ...which reminds me... I actually want a t-shirt that says.... "Bandwidth solves a lot" -aaron -----Original Message----- From: Jean St-Laurent <jean@ddostest.me> Sent: Thursday, April 1, 2021 2:01 PM To: aaron1@gvtc.com; 'Jared Mauch' <jared@puck.nether.net>; 'Töma Gavrichenkov' <ximaera@gmail.com> Cc: 'NANOG' <nanog@nanog.org> Subject: RE: wow, lots of akamai I remembered working for a big ISP in Europe offering cable tv + internet with +20M subscribers Every time there was a huge power outage in major cities, all tv`s would go off at the same time. I don`t have stats on power grid stability in Europe Vs N/A. The problem, was when the power was coming back in big cities, all the tv subscribers would come back online at the exact same second or minute. More or less the same 2 or 3 minutes. What happened is that it would create a kind of internal DDoS and they would all timed out and give a weird error message. Something very useful like Error Code 0x8098808 Please call our support line at this phone number. The server sysadmins would go on a panic because all systems were overloaded. They often needed to do overtime because DB crashed, key servers there crashed, DB here crashed, whatever... there was always something crashing. This was before the cloud when you could just push a slider and have tons of VMs or containers to absorb the load in real time. (in my dream) This would every time create frustration from the clients, the help desk, the support teams and also the upper management. Every time the teams were really tired after that. It was draining juice. Anyway, after some years of talking internally (red tape), we finally managed to install a random artificial penalty in the setup boxes when they boot after a power outage. Nothing like 20 minutes, but just enough to spread the load over a longer period of time. For the end user, it went transparent for them because, if the setup box would boot in 206 seconds instead of the super aggressive 34 seconds, well it booted and they could watch tv. Vs my system is totally frozen and it`s been like that for 20 minutes with weird messages because all your systems are down and the error msg said to call the help desk. This simple change to add 3 lines of code to add a random artificial boot penalty of few seconds, completely solve the problem. This way, when a city would black out, we wouldn't be self DDoS, because the systems would slowly rampup. The setup boxes would all reboot but, wait randomly before asking for the DRM package to unlock the cable TV service and validate whether billing is right. I`m no Call of Duty expert nor Akamai, but it's been many times that I observe the same question here: What's happening? Call of Duty! Okay. Would a kind of throttle help here? An artificial roll out penalty somehow? Probably not at the ISP level, but more at the game level. Well, ISP could also have some mechanisms to reduce the impact or even Akamai could force a progressive roll out. I`m not sure that the proposed solutions could work, but it seems to impact NANOG frequently and/or at least generate a call overnight/weekend. It seems to also happens just before long holidays when operations are sometimes on reduce personnel. Are big games roll out really impacting NANOG? or it's more a: Hey I was curious what happened and I thought to ask here on NANOG? #JustCurious Jean -----Original Message----- From: NANOG <nanog-bounces+jean=ddostest.me@nanog.org> On Behalf Of aaron1@gvtc.com Sent: April 1, 2021 12:12 PM To: 'Jared Mauch' <jared@puck.nether.net>; 'Töma Gavrichenkov' <ximaera@gmail.com> Cc: 'NANOG' <nanog@nanog.org> Subject: RE: wow, lots of akamai Gaming update... I had a feeling. Thanks for the feedback folks. Thanks Jared, it's running well, before, during and after. We have a lot of capacity there. -Aaron
Yes, I was reaching out to my NANOG folks to find out as you stated... "Hey I was curious what happened and I thought to ask here on NANOG?" I appreciate the membership with you all and value your position and visibility in regional, continental and global operations. Thanks for your insights, and I hope I contribute occasionally as well. -Aaron
participants (16)
-
aaron1＠gvtc.com -
Dave Brockman - DVS -
dc＠darwincosta.com -
james jones -
Jared Mauch -
Jean St-Laurent -
Luke Guillory -
Mark Tinka -
Matt Erculiani -
Mehmet Akcin -
Mike Hammett -
Niels Bakker -
Patrick W. Gilmore -
Tom Beecher -
Tony Wicks -
Töma Gavrichenkov