Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this... Thanks in adavnce, - ferg On 9/19/2013 10:23 AM, Nick Olsen wrote:
We also saw a huge spike in traffic. Still pretty high today as well. We saw a ~60% above average hit yesterday, And we're at ~20-30% above average today as well. Being an android user, It didn't dawn on me until some of the IOS users in the office started jumping up and down about IOS7 Nick Olsen Network Operations (855) FLSPEED x106
---------------------------------------- From: "Justin M. Streiner" <streiner@cluebyfour.org> Sent: Wednesday, September 18, 2013 6:19 PM To: "NANOG" <nanog@nanog.org> Subject: Re: iOS 7 update traffic
On Wed, 18 Sep 2013, Tassos Chatzithomaoglou wrote:
We also noticed an interesting spike (+ ~40%), mostly in akamai. The same happened on previous iOS too.
I see it here, too. At its peak, our traffic levels were roughly double what we would see on a normal weekday.
jms
Zachary McGibbon wrote on 18/9/2013 20:38:
So iOS 7 just came out, here's the spike in our graphs going to our ISP here at McGill, anyone else noticing a big spike?
[image: internet-sw1 - Traffic - Te0/7 - To Internet1-srp (IR Canet) - TenGigabitEthernet0/7]
Zachary McGibbon
-- Paul Ferguson Vice President, Threat Intelligence Internet Identity, Tacoma, Washington USA IID --> "Connect and Collaborate" --> www.internetidentity.com
On Thu, 19 Sep 2013, Paul Ferguson wrote:
Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this...
I think this was just the traffic to download iOS 7 to everyones' relevant Apple devices. I don't know how large the update was (maybe a few hundred MB per device?), but I guess everyone got the notification or their devices started automatically downloading around the same time. The vast majority of the traffic here (large .edu) happened between about 1 and 5 PM yesterday. jms
Apple actually tries to rate-limit the notifications to prevent this, but you can just manually go check and hit the upgrade button yourself. It's pretty well-known that Apple likes to release ~10am, so tens (hundreds?) of millions of users did just that. Since this update is available for all iThingies made in the last 4-ish years that means a lot of extra traffic. On Thu, Sep 19, 2013 at 7:13 AM, Justin M. Streiner <streiner@cluebyfour.org
wrote:
On Thu, 19 Sep 2013, Paul Ferguson wrote:
Can someone please explain to a non-Apple person what the hell happened
that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this...
I think this was just the traffic to download iOS 7 to everyones' relevant Apple devices. I don't know how large the update was (maybe a few hundred MB per device?), but I guess everyone got the notification or their devices started automatically downloading around the same time. The vast majority of the traffic here (large .edu) happened between about 1 and 5 PM yesterday.
jms
On Thu, 19 Sep 2013, Paul Ferguson wrote:
Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this...
The IOS7 upgrade is ~750 megabyte download for the phones/pods, and ~950 megabytes for ipad. There are quite a few devices out there times these amounts to download... -- Mikael Abrahamsson email: swmike@swm.pp.se
I don't see how operators could tolerate this, honestly. I can't think of a single provider who does not oversubscribe their access platform... Which leads me to this question : Why does apple feel it is okay to send every mobile device an update on a single day? Never mind the fact that we are we ones on the last mile responsible for getting it to their customers, 1gb per sub is pretty serious.. Why are they not caching at their head ends, dslams, etc? Sent from my Mobile Device. -------- Original message -------- From: Mikael Abrahamsson <swmike@swm.pp.se> Date: 09/19/2013 11:08 AM (GMT-08:00) To: Paul Ferguson <fergdawgster@mykolab.com> Cc: NANOG <nanog@nanog.org> Subject: Re: iOS 7 update traffic On Thu, 19 Sep 2013, Paul Ferguson wrote:
Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this...
The IOS7 upgrade is ~750 megabyte download for the phones/pods, and ~950 megabytes for ipad. There are quite a few devices out there times these amounts to download... -- Mikael Abrahamsson email: swmike@swm.pp.se
On Thu, 19 Sep 2013, Warren Bailey wrote:
I don't see how operators could tolerate this, honestly. I can't think of a single provider who does not oversubscribe their access platform... Which leads me to this question :
The vast majority of the traffic I saw was served from the Akamai farm at an upstream provider, so the pain that was felt 'on the backbone' was mitigated somewhat by that. jms
On Thu, 19 Sep 2013, Warren Bailey wrote:
Why does apple feel it is okay to send every mobile device an update on a single day?
They don't, these are users who actively goes into the software upgrade menu and pressing "upgrade". I believe the nagging won't start for quite some time. -- Mikael Abrahamsson email: swmike@swm.pp.se
I own a galaxy note 2..tmo ran an update that pushed to unique IMEI's sequentially. That way, you do not.. 1. Murder your last mike packet network, which is your bandwidth bottleneck. 2. Murder your ggsn/whateverpacketnodeyouwant closer to the core. 3. Anger your paying customers who would like to use packet data successfully on an ios download day. These people (Apple) represent themselves as smart guys, but their actions reflect otherwise. I bet this would be a larger deal to Nanog people if your Internet stopped working as the result of 100% Linux adoption. That is very close to what this is.. Tens of millions of people trying to update their 13 ios devices at the same time. Who owns a single ios device? A household could do 5-10gb worth of updates in a single day.. I personally do not own an ios device, and I see close to 3 gigs worth of update traffic at my house. These things are everywhere, and this problem will not stop. Sent from my Mobile Device. -------- Original message -------- From: Mikael Abrahamsson <swmike@swm.pp.se> Date: 09/19/2013 11:16 AM (GMT-08:00) To: Warren Bailey <wbailey@satelliteintelligencegroup.com> Cc: Paul Ferguson <fergdawgster@mykolab.com>,NANOG <nanog@nanog.org> Subject: Re: iOS 7 update traffic On Thu, 19 Sep 2013, Warren Bailey wrote:
Why does apple feel it is okay to send every mobile device an update on a single day?
They don't, these are users who actively goes into the software upgrade menu and pressing "upgrade". I believe the nagging won't start for quite some time. -- Mikael Abrahamsson email: swmike@swm.pp.se
To be honest, I don't see this as a problem at all. Use it as an excuse to upgrade your pipes, talk Akamai or CDN of choice into putting a cache on your network, or implement your own caching solution. As operators of the Internet we should be looking for ways to enable things like this, not be up in arms at Apple for releasing an update to their phone OS or making it available in a way that's inconvenient to our oversubscription policies. As a side note, how are some of you not aware of this? This has happened with every single Apple OS update since the iPhone was released in 2007. This isn't a new phenomenon. I realize some of you are too cool for Apple, but paying attention to traffic trends and keeping abreast of how new software releases might affect your utilization is part of properly running a network. /Ryan Ryan Harden Senior Network Engineer University of Chicago - AS160 P: 773-834-5441 On Sep 19, 2013, at 1:22 PM, Warren Bailey <wbailey@satelliteintelligencegroup.com> wrote:
I own a galaxy note 2..tmo ran an update that pushed to unique IMEI's sequentially. That way, you do not..
1. Murder your last mike packet network, which is your bandwidth bottleneck.
2. Murder your ggsn/whateverpacketnodeyouwant closer to the core.
3. Anger your paying customers who would like to use packet data successfully on an ios download day.
These people (Apple) represent themselves as smart guys, but their actions reflect otherwise. I bet this would be a larger deal to Nanog people if your Internet stopped working as the result of 100% Linux adoption. That is very close to what this is.. Tens of millions of people trying to update their 13 ios devices at the same time. Who owns a single ios device? A household could do 5-10gb worth of updates in a single day..
I personally do not own an ios device, and I see close to 3 gigs worth of update traffic at my house. These things are everywhere, and this problem will not stop.
Sent from my Mobile Device.
-------- Original message -------- From: Mikael Abrahamsson <swmike@swm.pp.se> Date: 09/19/2013 11:16 AM (GMT-08:00) To: Warren Bailey <wbailey@satelliteintelligencegroup.com> Cc: Paul Ferguson <fergdawgster@mykolab.com>,NANOG <nanog@nanog.org> Subject: Re: iOS 7 update traffic
On Thu, 19 Sep 2013, Warren Bailey wrote:
Why does apple feel it is okay to send every mobile device an update on a single day?
They don't, these are users who actively goes into the software upgrade menu and pressing "upgrade".
I believe the nagging won't start for quite some time.
-- Mikael Abrahamsson email: swmike@swm.pp.se
I certainly don't want to put words in his mouth, but I thin Warren's problem is that he can't upgrade his pipes. Physics limits the bandwidth available, as I think he is a satellite provider. My argument is that if I'm a satellite user I should be well aware, particularly because this is not a new phenomenon, that there are times when my bandwidth will suck. It is what it is. On 9/19/13 3:06 PM, "Ryan Harden" <hardenrm@uchicago.edu> wrote:
To be honest, I don't see this as a problem at all. Use it as an excuse to upgrade your pipes, talk Akamai or CDN of choice into putting a cache on your network, or implement your own caching solution. As operators of the Internet we should be looking for ways to enable things like this, not be up in arms at Apple for releasing an update to their phone OS or making it available in a way that's inconvenient to our oversubscription policies.
As a side note, how are some of you not aware of this? This has happened with every single Apple OS update since the iPhone was released in 2007. This isn't a new phenomenon. I realize some of you are too cool for Apple, but paying attention to traffic trends and keeping abreast of how new software releases might affect your utilization is part of properly running a network.
/Ryan
Ryan Harden Senior Network Engineer University of Chicago - AS160 P: 773-834-5441
On Sep 19, 2013, at 1:22 PM, Warren Bailey <wbailey@satelliteintelligencegroup.com> wrote:
I own a galaxy note 2..tmo ran an update that pushed to unique IMEI's sequentially. That way, you do not..
1. Murder your last mike packet network, which is your bandwidth bottleneck.
2. Murder your ggsn/whateverpacketnodeyouwant closer to the core.
3. Anger your paying customers who would like to use packet data successfully on an ios download day.
These people (Apple) represent themselves as smart guys, but their actions reflect otherwise. I bet this would be a larger deal to Nanog people if your Internet stopped working as the result of 100% Linux adoption. That is very close to what this is.. Tens of millions of people trying to update their 13 ios devices at the same time. Who owns a single ios device? A household could do 5-10gb worth of updates in a single day..
I personally do not own an ios device, and I see close to 3 gigs worth of update traffic at my house. These things are everywhere, and this problem will not stop.
Sent from my Mobile Device.
-------- Original message -------- From: Mikael Abrahamsson <swmike@swm.pp.se> Date: 09/19/2013 11:16 AM (GMT-08:00) To: Warren Bailey <wbailey@satelliteintelligencegroup.com> Cc: Paul Ferguson <fergdawgster@mykolab.com>,NANOG <nanog@nanog.org> Subject: Re: iOS 7 update traffic
On Thu, 19 Sep 2013, Warren Bailey wrote:
Why does apple feel it is okay to send every mobile device an update on a single day?
They don't, these are users who actively goes into the software upgrade menu and pressing "upgrade".
I believe the nagging won't start for quite some time.
-- Mikael Abrahamsson email: swmike@swm.pp.se
Absolutely correct. Large file updates etc are not an issue for wide band communications networks. If you have 10mbps to your house with a 30ms delay to your first hop.. You're sitting pretty. If you have a 1mbps/512kbps pipe with a built in 750ms latency, things get a little more complicated. Add in the fact that our bandwidth is insanely expensive (link budgets put 1mbps are at about 1MHz *1bits/hz* which costs roughly $3800 a month in sky access ONLY). We have some challenges, and we try (sometimes actually succeed) to get data from point A to point B in an acceptable time frame. The issue is.. When a user begins to EXCEED their general usage things in contention land get pretty hairy. If my 500 remote locations are running on 10MHz worth of capacity (about 10mbps down, 2mbps up) at a 10:1 or 20:1 oversub things get tight sometimes. Imagine an entire user base trying to update their electronic devices on something that feels like 56k. All things on the internet are not created equal, and I can accept that. I can accept that I/we have challenges, but I would figure that those of you who had easier lives would want to share that ease with us who are doing things a little differently. I'm not saying poor us, we need love.. I'm just saying.. I would really appreciate if those of you in the industry who are responsible for these types of scenarios could think about the people who are paying a lot of money, for a very little amount of bandwidth. We don't tier our service by usage cap, but if Apple/Microsoft/whoever increases their update rate we may have to end up looking at something like that. Big files, over a little pipe, with little margin for improvement, is hard. It wouldn't be *THAT* hard to help out those who are trying to make communications available to a much larger customer base than a traditional ISP. Group Hug. //warren On 9/19/13 1:08 PM, "Fred Reimer" <freimer@freimer.org> wrote:
I certainly don't want to put words in his mouth, but I thin Warren's problem is that he can't upgrade his pipes. Physics limits the bandwidth available, as I think he is a satellite provider. My argument is that if I'm a satellite user I should be well aware, particularly because this is not a new phenomenon, that there are times when my bandwidth will suck. It is what it is.
On 9/19/13 3:06 PM, "Ryan Harden" <hardenrm@uchicago.edu> wrote:
To be honest, I don't see this as a problem at all. Use it as an excuse to upgrade your pipes, talk Akamai or CDN of choice into putting a cache on your network, or implement your own caching solution. As operators of the Internet we should be looking for ways to enable things like this, not be up in arms at Apple for releasing an update to their phone OS or making it available in a way that's inconvenient to our oversubscription policies.
As a side note, how are some of you not aware of this? This has happened with every single Apple OS update since the iPhone was released in 2007. This isn't a new phenomenon. I realize some of you are too cool for Apple, but paying attention to traffic trends and keeping abreast of how new software releases might affect your utilization is part of properly running a network.
/Ryan
Ryan Harden Senior Network Engineer University of Chicago - AS160 P: 773-834-5441
On Sep 19, 2013, at 1:22 PM, Warren Bailey <wbailey@satelliteintelligencegroup.com> wrote:
I own a galaxy note 2..tmo ran an update that pushed to unique IMEI's sequentially. That way, you do not..
1. Murder your last mike packet network, which is your bandwidth bottleneck.
2. Murder your ggsn/whateverpacketnodeyouwant closer to the core.
3. Anger your paying customers who would like to use packet data successfully on an ios download day.
These people (Apple) represent themselves as smart guys, but their actions reflect otherwise. I bet this would be a larger deal to Nanog people if your Internet stopped working as the result of 100% Linux adoption. That is very close to what this is.. Tens of millions of people trying to update their 13 ios devices at the same time. Who owns a single ios device? A household could do 5-10gb worth of updates in a single day..
I personally do not own an ios device, and I see close to 3 gigs worth of update traffic at my house. These things are everywhere, and this problem will not stop.
Sent from my Mobile Device.
-------- Original message -------- From: Mikael Abrahamsson <swmike@swm.pp.se> Date: 09/19/2013 11:16 AM (GMT-08:00) To: Warren Bailey <wbailey@satelliteintelligencegroup.com> Cc: Paul Ferguson <fergdawgster@mykolab.com>,NANOG <nanog@nanog.org> Subject: Re: iOS 7 update traffic
On Thu, 19 Sep 2013, Warren Bailey wrote:
Why does apple feel it is okay to send every mobile device an update on a single day?
They don't, these are users who actively goes into the software upgrade menu and pressing "upgrade".
I believe the nagging won't start for quite some time.
-- Mikael Abrahamsson email: swmike@swm.pp.se
On 09/19/2013 12:06 PM, Ryan Harden wrote:
As a side note, how are some of you not aware of this? This has happened with every single Apple OS update since the iPhone was released in 2007.
The difference is there are now a "couple" more million devices out there than there were in 2007. And in 2007 there was just the one phone, now you have tablets and what have you.
This isn't a new phenomenon. I realize some of you are too cool for Apple
Lame low ball remark, however I thought it was the opposite, Apple==coolness? Regards, Jeroen -- Earthquake Magnitude: 5.3 Date: 2013-09-19 17:25:09.350 UTC Location: 19km ESE of Ishikawa, Japan Latitude: 37.0716; Longitude: 140.6495 Depth: 22.22 km | e-quake.org
On Sep 19, 2013, at 3:11 PM, Jeroen van Aart <jeroen@mompl.net> wrote:
On 09/19/2013 12:06 PM, Ryan Harden wrote:
As a side note, how are some of you not aware of this? This has happened with every single Apple OS update since the iPhone was released in 2007.
The difference is there are now a "couple" more million devices out there than there were in 2007. And in 2007 there was just the one phone, now you have tablets and what have you.
The effect has been relatively the same regardless of how many iDevices there are. Network Operators have seen spikes during Apple OS releases since they started. The only leeway I'll give you is that the original iPhone only supported 802.11b. With .11n and someday .11ac, the ability for these devices to consume data at a faster rate is also increasing.
This isn't a new phenomenon. I realize some of you are too cool for Apple
Lame low ball remark, however I thought it was the opposite, Apple==coolness?
This was in no way meant to be a lowball remark. But it doesn't take much searching to find people exclaiming how they have zero Apple devices or how they don't pay attention to Apple's "iJunk". I assumed (probably mistakenly) that the lack of knowing this is going to happen roughly 2-3 times a year was due to being 'too cool' to keep up with the stuff Apple puts out.
Regards, Jeroen
-- Earthquake Magnitude: 5.3 Date: 2013-09-19 17:25:09.350 UTC Location: 19km ESE of Ishikawa, Japan Latitude: 37.0716; Longitude: 140.6495 Depth: 22.22 km | e-quake.org
Your software updates (you meaning a user of the Internet) should not affect my experience. I'm not advocating we go back to 5.25 floppies and never look back. I'm asking.. Is there a way for a COMPUTER and PHONE manufacturer to distribute their software without destroying most last mile connectivity? Who else has had traffic surges like this? And who else has a Nanog strike team coming in screaming buy more bandwidth? ;) Sent from my Mobile Device. -------- Original message -------- From: Ryan Harden <hardenrm@uchicago.edu> Date: 09/19/2013 3:04 PM (GMT-08:00) To: Jeroen van Aart <jeroen@mompl.net> Cc: "<nanog@nanog.org>" <nanog@nanog.org> Subject: Re: iOS 7 update traffic On Sep 19, 2013, at 3:11 PM, Jeroen van Aart <jeroen@mompl.net> wrote:
On 09/19/2013 12:06 PM, Ryan Harden wrote:
As a side note, how are some of you not aware of this? This has happened with every single Apple OS update since the iPhone was released in 2007.
The difference is there are now a "couple" more million devices out there than there were in 2007. And in 2007 there was just the one phone, now you have tablets and what have you.
The effect has been relatively the same regardless of how many iDevices there are. Network Operators have seen spikes during Apple OS releases since they started. The only leeway I'll give you is that the original iPhone only supported 802.11b. With .11n and someday .11ac, the ability for these devices to consume data at a faster rate is also increasing.
This isn't a new phenomenon. I realize some of you are too cool for Apple
Lame low ball remark, however I thought it was the opposite, Apple==coolness?
This was in no way meant to be a lowball remark. But it doesn't take much searching to find people exclaiming how they have zero Apple devices or how they don't pay attention to Apple's "iJunk". I assumed (probably mistakenly) that the lack of knowing this is going to happen roughly 2-3 times a year was due to being 'too cool' to keep up with the stuff Apple puts out.
Regards, Jeroen
-- Earthquake Magnitude: 5.3 Date: 2013-09-19 17:25:09.350 UTC Location: 19km ESE of Ishikawa, Japan Latitude: 37.0716; Longitude: 140.6495 Depth: 22.22 km | e-quake.org
On 9/19/13 3:29 PM, Warren Bailey wrote:
Your software updates (you meaning a user of the Internet) should not affect my experience. I'm not advocating we go back to 5.25 floppies and never look back. I'm asking..
Is there a way for a COMPUTER and PHONE manufacturer to distribute their software without destroying most last mile connectivity?
Who else has had traffic surges like this?
Flash traffic occurs, sometimes people fly planes into things, sometimes nuclear reactors melt down, earthquakes or hurricanes occur or cables are segmented due to underwater landslides. and what infrastructure that is left shifts abruptly from terrestrial to sattelite or gets droppped on the floor. the best you can ask for on an instantanious basis is graceful degredation under load. this happens to not be weather.so maybe you can do something about it. but ultimately a certain number of bytes have to be transfered and given the architecture, the flash was driven by the consumer and not by software automation, if we want the later to control it consumer choice has to be taken out of the loop, which may or may not be palatable.
And who else has a Nanog strike team coming in screaming buy more bandwidth? ;)
Sent from my Mobile Device.
-------- Original message -------- From: Ryan Harden <hardenrm@uchicago.edu> Date: 09/19/2013 3:04 PM (GMT-08:00) To: Jeroen van Aart <jeroen@mompl.net> Cc: "<nanog@nanog.org>" <nanog@nanog.org> Subject: Re: iOS 7 update traffic
On Sep 19, 2013, at 3:11 PM, Jeroen van Aart <jeroen@mompl.net> wrote:
On 09/19/2013 12:06 PM, Ryan Harden wrote:
As a side note, how are some of you not aware of this? This has happened with every single Apple OS update since the iPhone was released in 2007.
The difference is there are now a "couple" more million devices out there than there were in 2007. And in 2007 there was just the one phone, now you have tablets and what have you.
The effect has been relatively the same regardless of how many iDevices there are. Network Operators have seen spikes during Apple OS releases since they started. The only leeway I'll give you is that the original iPhone only supported 802.11b. With .11n and someday .11ac, the ability for these devices to consume data at a faster rate is also increasing.
This isn't a new phenomenon. I realize some of you are too cool for Apple
Lame low ball remark, however I thought it was the opposite, Apple==coolness?
This was in no way meant to be a lowball remark. But it doesn't take much searching to find people exclaiming how they have zero Apple devices or how they don't pay attention to Apple's "iJunk". I assumed (probably mistakenly) that the lack of knowing this is going to happen roughly 2-3 times a year was due to being 'too cool' to keep up with the stuff Apple puts out.
Regards, Jeroen
-- Earthquake Magnitude: 5.3 Date: 2013-09-19 17:25:09.350 UTC Location: 19km ESE of Ishikawa, Japan Latitude: 37.0716; Longitude: 140.6495 Depth: 22.22 km | e-quake.org
1) Rate limit the software update download ("Us") 2) Have device OS download the update in the background, and be resilient to failures with retries ("Manufacturer") 3) Don't present the update notification to the user until the update blob is already cached on the device ("Manufacturer") Only in a perfect world though. On Thu, Sep 19, 2013 at 5:49 PM, joel jaeggli <joelja@bogus.com> wrote:
On 9/19/13 3:29 PM, Warren Bailey wrote:
Your software updates (you meaning a user of the Internet) should not affect my experience. I'm not advocating we go back to 5.25 floppies and never look back. I'm asking..
Is there a way for a COMPUTER and PHONE manufacturer to distribute their software without destroying most last mile connectivity?
Who else has had traffic surges like this?
Flash traffic occurs, sometimes people fly planes into things, sometimes nuclear reactors melt down, earthquakes or hurricanes occur or cables are segmented due to underwater landslides. and what infrastructure that is left shifts abruptly from terrestrial to sattelite or gets droppped on the floor. the best you can ask for on an instantanious basis is graceful degredation under load.
this happens to not be weather.so maybe you can do something about it. but ultimately a certain number of bytes have to be transfered and given the architecture, the flash was driven by the consumer and not by software automation, if we want the later to control it consumer choice has to be taken out of the loop, which may or may not be palatable.
And who else has a Nanog strike team coming in screaming buy more bandwidth? ;)
Sent from my Mobile Device.
-------- Original message -------- From: Ryan Harden <hardenrm@uchicago.edu> Date: 09/19/2013 3:04 PM (GMT-08:00) To: Jeroen van Aart <jeroen@mompl.net> Cc: "<nanog@nanog.org>" <nanog@nanog.org> Subject: Re: iOS 7 update traffic
On Sep 19, 2013, at 3:11 PM, Jeroen van Aart <jeroen@mompl.net> wrote:
On 09/19/2013 12:06 PM, Ryan Harden wrote:
As a side note, how are some of you not aware of this? This has happened with every single Apple OS update since the iPhone was released in
The difference is there are now a "couple" more million devices out
there than there were in 2007. And in 2007 there was just the one phone, now you have tablets and what have you.
The effect has been relatively the same regardless of how many iDevices there are. Network Operators have seen spikes during Apple OS releases since they started. The only leeway I'll give you is that the original iPhone only supported 802.11b. With .11n and someday .11ac, the ability for these devices to consume data at a faster rate is also increasing.
This isn't a new phenomenon. I realize some of you are too cool for
Apple
Lame low ball remark, however I thought it was the opposite,
Apple==coolness?
This was in no way meant to be a lowball remark. But it doesn't take much searching to find people exclaiming how they have zero Apple devices or how they don't pay attention to Apple's "iJunk". I assumed (probably mistakenly) that the lack of knowing this is going to happen roughly 2-3 times a year was due to being 'too cool' to keep up with the stuff Apple puts out.
Regards, Jeroen
-- Earthquake Magnitude: 5.3 Date: 2013-09-19 17:25:09.350 UTC Location: 19km ESE of Ishikawa, Japan Latitude: 37.0716; Longitude: 140.6495 Depth: 22.22 km | e-quake.org
On 9/19/13 3:57 PM, Brandon Galbraith wrote:
1) Rate limit the software update download ("Us")
2) Have device OS download the update in the background, and be resilient to failures with retries ("Manufacturer")
3) Don't present the update notification to the user until the update blob is already cached on the device ("Manufacturer")
Even in this case, this comes down to a whole bunch of consumers going to the general menu selecting software update and selecting check for software update. So insuring that consumers don't have the opportunity to retrieve something that they want when they want it would be part of a solution. some large fraction of the devices will be soaking this up over the next couple days rather than yesterday some won't get it for quite some time. Software distribution can handle this, they could have been pushing unreleased software blobs for a couple weeks for example, as some steam game launches do for example. But, if you support near instantanious gratification then, when somebody asks for something, then you start fulfilling it.
Only in a perfect world though.
On Thu, Sep 19, 2013 at 5:49 PM, joel jaeggli <joelja@bogus.com <mailto:joelja@bogus.com>> wrote:
On 9/19/13 3:29 PM, Warren Bailey wrote: > Your software updates (you meaning a user of the Internet) should not affect my experience. I'm not advocating we go back to 5.25 floppies and never look back. I'm asking.. > > Is there a way for a COMPUTER and PHONE manufacturer to distribute their software without destroying most last mile connectivity? > > Who else has had traffic surges like this?
Flash traffic occurs, sometimes people fly planes into things, sometimes nuclear reactors melt down, earthquakes or hurricanes occur or cables are segmented due to underwater landslides. and what infrastructure that is left shifts abruptly from terrestrial to sattelite or gets droppped on the floor. the best you can ask for on an instantanious basis is graceful degredation under load.
this happens to not be weather.so maybe you can do something about it. but ultimately a certain number of bytes have to be transfered and given the architecture, the flash was driven by the consumer and not by software automation, if we want the later to control it consumer choice has to be taken out of the loop, which may or may not be palatable.
> And who else has a Nanog strike team coming in screaming buy more bandwidth? ;) > > > Sent from my Mobile Device. > > > -------- Original message -------- > From: Ryan Harden <hardenrm@uchicago.edu <mailto:hardenrm@uchicago.edu>> > Date: 09/19/2013 3:04 PM (GMT-08:00) > To: Jeroen van Aart <jeroen@mompl.net <mailto:jeroen@mompl.net>> > Cc: "<nanog@nanog.org <mailto:nanog@nanog.org>>" <nanog@nanog.org <mailto:nanog@nanog.org>> > Subject: Re: iOS 7 update traffic > > > > On Sep 19, 2013, at 3:11 PM, Jeroen van Aart <jeroen@mompl.net <mailto:jeroen@mompl.net>> wrote: > >> On 09/19/2013 12:06 PM, Ryan Harden wrote: >>> As a side note, how are some of you not aware of this? This has happened with every single Apple OS update since the iPhone was released in 2007. >> >> The difference is there are now a "couple" more million devices out there than there were in 2007. And in 2007 there was just the one phone, now you have tablets and what have you. > > The effect has been relatively the same regardless of how many iDevices there are. Network Operators have seen spikes during Apple OS releases since they started. The only leeway I'll give you is that the original iPhone only supported 802.11b. With .11n and someday .11ac, the ability for these devices to consume data at a faster rate is also increasing. > >> >>> This isn't a new phenomenon. I realize some of you are too cool for Apple >> >> Lame low ball remark, however I thought it was the opposite, Apple==coolness? > > This was in no way meant to be a lowball remark. But it doesn't take much searching to find people exclaiming how they have zero Apple devices or how they don't pay attention to Apple's "iJunk". I assumed (probably mistakenly) that the lack of knowing this is going to happen roughly 2-3 times a year was due to being 'too cool' to keep up with the stuff Apple puts out. > >> >> Regards, >> Jeroen >> >> -- >> Earthquake Magnitude: 5.3 >> Date: 2013-09-19 17:25:09.350 UTC >> Location: 19km ESE of Ishikawa, Japan >> Latitude: 37.0716; Longitude: 140.6495 >> Depth: 22.22 km | e-quake.org <http://e-quake.org> >> > >
Just as a note. On Sep 19, 2013, at 6:57 PM, Brandon Galbraith wrote:
1) Rate limit the software update download ("Us")
2) Have device OS download the update in the background, and be resilient to failures with retries ("Manufacturer")
Apple already does this in the iTunes update the ios device mode.
3) Don't present the update notification to the user until the update blob is already cached on the device ("Manufacturer")
Apple also already does this. However, manual checks/updates can be done. When there is so much buzz on the news and given Apple customers zeal a large percentage manually invoke the update.
Only in a perfect world though.
On Thu, Sep 19, 2013 at 5:49 PM, joel jaeggli <joelja@bogus.com> wrote:
On 9/19/13 3:29 PM, Warren Bailey wrote:
Your software updates (you meaning a user of the Internet) should not affect my experience. I'm not advocating we go back to 5.25 floppies and never look back. I'm asking..
Is there a way for a COMPUTER and PHONE manufacturer to distribute their software without destroying most last mile connectivity?
Who else has had traffic surges like this?
Flash traffic occurs, sometimes people fly planes into things, sometimes nuclear reactors melt down, earthquakes or hurricanes occur or cables are segmented due to underwater landslides. and what infrastructure that is left shifts abruptly from terrestrial to sattelite or gets droppped on the floor. the best you can ask for on an instantanious basis is graceful degredation under load.
this happens to not be weather.so maybe you can do something about it. but ultimately a certain number of bytes have to be transfered and given the architecture, the flash was driven by the consumer and not by software automation, if we want the later to control it consumer choice has to be taken out of the loop, which may or may not be palatable.
And who else has a Nanog strike team coming in screaming buy more bandwidth? ;)
Sent from my Mobile Device.
-------- Original message -------- From: Ryan Harden <hardenrm@uchicago.edu> Date: 09/19/2013 3:04 PM (GMT-08:00) To: Jeroen van Aart <jeroen@mompl.net> Cc: "<nanog@nanog.org>" <nanog@nanog.org> Subject: Re: iOS 7 update traffic
On Sep 19, 2013, at 3:11 PM, Jeroen van Aart <jeroen@mompl.net> wrote:
On 09/19/2013 12:06 PM, Ryan Harden wrote:
As a side note, how are some of you not aware of this? This has happened with every single Apple OS update since the iPhone was released in
The difference is there are now a "couple" more million devices out
there than there were in 2007. And in 2007 there was just the one phone, now you have tablets and what have you.
The effect has been relatively the same regardless of how many iDevices there are. Network Operators have seen spikes during Apple OS releases since they started. The only leeway I'll give you is that the original iPhone only supported 802.11b. With .11n and someday .11ac, the ability for these devices to consume data at a faster rate is also increasing.
This isn't a new phenomenon. I realize some of you are too cool for
Apple
Lame low ball remark, however I thought it was the opposite,
Apple==coolness?
This was in no way meant to be a lowball remark. But it doesn't take much searching to find people exclaiming how they have zero Apple devices or how they don't pay attention to Apple's "iJunk". I assumed (probably mistakenly) that the lack of knowing this is going to happen roughly 2-3 times a year was due to being 'too cool' to keep up with the stuff Apple puts out.
Regards, Jeroen
-- Earthquake Magnitude: 5.3 Date: 2013-09-19 17:25:09.350 UTC Location: 19km ESE of Ishikawa, Japan Latitude: 37.0716; Longitude: 140.6495 Depth: 22.22 km | e-quake.org
Again, as others have said: complain to the ISP that most probably oversubscribed their links. On 19/09/13 15:29, Warren Bailey wrote:
Your software updates (you meaning a user of the Internet) should not affect my experience. I'm not advocating we go back to 5.25 floppies and never look back. I'm asking..
Is there a way for a COMPUTER and PHONE manufacturer to distribute their software without destroying most last mile connectivity?
Who else has had traffic surges like this? And who else has a Nanog strike team coming in screaming buy more bandwidth? ;)
On Thu, 19 Sep 2013 22:29:28 -0000, Warren Bailey said:
Who else has had traffic surges like this?
Most are reporting a doubling or so of bandwidth, for an event that you had a week's advance notice. April 16 a few years ago, we had a much higher bandwidth impact on much shorter notice. Oh, and a protip: Trying to remediate a slashdotting is hard. Trying to remediate one while you're still under lockdown due to the cause of the slashdotting is even harder. But our efforts were nowhere near as valiant as the guys who were boots on the ground in Haiti after the quake, or the guys who kept 60 Hudson online. In the greater scheme of things, it was *just* a software release.
Why do you sell services to customers using iThings if you are incapable of supporting them? Are you sure that it is not you yourself who have used to much "bait and switch" selling a service you are unable to provide? What actions do you take to discourage iThings on your network?
-----Original Message----- From: Warren Bailey [mailto:wbailey@satelliteintelligencegroup.com] Sent: Thursday, 19 September, 2013 16:29 To: Ryan Harden; Jeroen van Aart Cc: <nanog@nanog.org> Subject: Re: iOS 7 update traffic
Your software updates (you meaning a user of the Internet) should not affect my experience. I'm not advocating we go back to 5.25 floppies and never look back. I'm asking..
Is there a way for a COMPUTER and PHONE manufacturer to distribute their software without destroying most last mile connectivity?
Who else has had traffic surges like this? And who else has a Nanog strike team coming in screaming buy more bandwidth? ;)
Sent from my Mobile Device.
-------- Original message -------- From: Ryan Harden <hardenrm@uchicago.edu> Date: 09/19/2013 3:04 PM (GMT-08:00) To: Jeroen van Aart <jeroen@mompl.net> Cc: "<nanog@nanog.org>" <nanog@nanog.org> Subject: Re: iOS 7 update traffic
On Sep 19, 2013, at 3:11 PM, Jeroen van Aart <jeroen@mompl.net> wrote:
On 09/19/2013 12:06 PM, Ryan Harden wrote:
As a side note, how are some of you not aware of this? This has happened with every single Apple OS update since the iPhone was released in 2007.
The difference is there are now a "couple" more million devices out there than there were in 2007. And in 2007 there was just the one phone, now you have tablets and what have you.
The effect has been relatively the same regardless of how many iDevices there are. Network Operators have seen spikes during Apple OS releases since they started. The only leeway I'll give you is that the original iPhone only supported 802.11b. With .11n and someday .11ac, the ability for these devices to consume data at a faster rate is also increasing.
This isn't a new phenomenon. I realize some of you are too cool for
Apple
Lame low ball remark, however I thought it was the opposite,
Apple==coolness?
This was in no way meant to be a lowball remark. But it doesn't take much searching to find people exclaiming how they have zero Apple devices or how they don't pay attention to Apple's "iJunk". I assumed (probably mistakenly) that the lack of knowing this is going to happen roughly 2-3 times a year was due to being 'too cool' to keep up with the stuff Apple puts out.
Regards, Jeroen
-- Earthquake Magnitude: 5.3 Date: 2013-09-19 17:25:09.350 UTC Location: 19km ESE of Ishikawa, Japan Latitude: 37.0716; Longitude: 140.6495 Depth: 22.22 km | e-quake.org
On 9/19/13 5:54 PM, Keith Medcalf wrote:
Why do you sell services to customers using iThings if you are incapable of supporting them? Are you sure that it is not you yourself who have used to much "bait and switch" selling a service you are unable to provide? What actions do you take to discourage iThings on your network?
Not all business models are able sustain some kinds of demands. Marketplaces aren't so elsatic that sudden changes in demand can be immediately addressed. Given enough time and incentive this addressed, at the margins however you may get hosed anyway. Physics is a stern taskmaster.
-----Original Message----- From: Warren Bailey [mailto:wbailey@satelliteintelligencegroup.com] Sent: Thursday, 19 September, 2013 16:29 To: Ryan Harden; Jeroen van Aart Cc: <nanog@nanog.org> Subject: Re: iOS 7 update traffic
Your software updates (you meaning a user of the Internet) should not affect my experience. I'm not advocating we go back to 5.25 floppies and never look back. I'm asking..
Is there a way for a COMPUTER and PHONE manufacturer to distribute their software without destroying most last mile connectivity?
Who else has had traffic surges like this? And who else has a Nanog strike team coming in screaming buy more bandwidth? ;)
Sent from my Mobile Device.
-------- Original message -------- From: Ryan Harden <hardenrm@uchicago.edu> Date: 09/19/2013 3:04 PM (GMT-08:00) To: Jeroen van Aart <jeroen@mompl.net> Cc: "<nanog@nanog.org>" <nanog@nanog.org> Subject: Re: iOS 7 update traffic
On Sep 19, 2013, at 3:11 PM, Jeroen van Aart <jeroen@mompl.net> wrote:
On 09/19/2013 12:06 PM, Ryan Harden wrote:
As a side note, how are some of you not aware of this? This has happened with every single Apple OS update since the iPhone was released in 2007.
The difference is there are now a "couple" more million devices out there than there were in 2007. And in 2007 there was just the one phone, now you have tablets and what have you.
The effect has been relatively the same regardless of how many iDevices there are. Network Operators have seen spikes during Apple OS releases since they started. The only leeway I'll give you is that the original iPhone only supported 802.11b. With .11n and someday .11ac, the ability for these devices to consume data at a faster rate is also increasing.
This isn't a new phenomenon. I realize some of you are too cool for
Apple
Lame low ball remark, however I thought it was the opposite,
Apple==coolness?
This was in no way meant to be a lowball remark. But it doesn't take much searching to find people exclaiming how they have zero Apple devices or how they don't pay attention to Apple's "iJunk". I assumed (probably mistakenly) that the lack of knowing this is going to happen roughly 2-3 times a year was due to being 'too cool' to keep up with the stuff Apple puts out.
Regards, Jeroen
-- Earthquake Magnitude: 5.3 Date: 2013-09-19 17:25:09.350 UTC Location: 19km ESE of Ishikawa, Japan Latitude: 37.0716; Longitude: 140.6495 Depth: 22.22 km | e-quake.org
That system by the way is annoying when your mobile network operator are so oversubscribed/old-fashioned that I had to wait over 6 months before I could update to Android ICS... I really don't want my ability to update the software on my phone to be controlled by a teleco, and these large teleco's really should have Akamai caches in place by now - if they even know what that is. On Thu, Sep 19, 2013 at 8:22 PM, Warren Bailey < wbailey@satelliteintelligencegroup.com> wrote:
I own a galaxy note 2..tmo ran an update that pushed to unique IMEI's sequentially. That way, you do not..
1. Murder your last mike packet network, which is your bandwidth bottleneck.
2. Murder your ggsn/whateverpacketnodeyouwant closer to the core.
3. Anger your paying customers who would like to use packet data successfully on an ios download day.
These people (Apple) represent themselves as smart guys, but their actions reflect otherwise. I bet this would be a larger deal to Nanog people if your Internet stopped working as the result of 100% Linux adoption. That is very close to what this is.. Tens of millions of people trying to update their 13 ios devices at the same time. Who owns a single ios device? A household could do 5-10gb worth of updates in a single day..
I personally do not own an ios device, and I see close to 3 gigs worth of update traffic at my house. These things are everywhere, and this problem will not stop.
Sent from my Mobile Device.
-------- Original message -------- From: Mikael Abrahamsson <swmike@swm.pp.se> Date: 09/19/2013 11:16 AM (GMT-08:00) To: Warren Bailey <wbailey@satelliteintelligencegroup.com> Cc: Paul Ferguson <fergdawgster@mykolab.com>,NANOG <nanog@nanog.org> Subject: Re: iOS 7 update traffic
On Thu, 19 Sep 2013, Warren Bailey wrote:
Why does apple feel it is okay to send every mobile device an update on a single day?
They don't, these are users who actively goes into the software upgrade menu and pressing "upgrade".
I believe the nagging won't start for quite some time.
-- Mikael Abrahamsson email: swmike@swm.pp.se
Picked this off www.jaluri.com (network and Cisco blog aggregator): http://routingfreak.wordpress.com/2013/09/23/ios7s-impact-on-networks-worldw... The consensus seems to be for providers to install CDN servers, if they arent able to cope up with an occasional OS update traffic. http://news.idg.no/cw/art.cfm?id=391B4B64-F693-41B7-6BBAC6D7017C3B8A John ________________________________ From: Colin Alston <colin.alston@gmail.com> To: Warren Bailey <wbailey@satelliteintelligencegroup.com> Cc: NANOG <nanog@nanog.org> Sent: Monday, 23 September 2013, 1:05 Subject: Re: iOS 7 update traffic That system by the way is annoying when your mobile network operator are so oversubscribed/old-fashioned that I had to wait over 6 months before I could update to Android ICS... I really don't want my ability to update the software on my phone to be controlled by a teleco, and these large teleco's really should have Akamai caches in place by now - if they even know what that is. On Thu, Sep 19, 2013 at 8:22 PM, Warren Bailey < wbailey@satelliteintelligencegroup.com> wrote:
I own a galaxy note 2..tmo ran an update that pushed to unique IMEI's sequentially. That way, you do not..
1. Murder your last mike packet network, which is your bandwidth bottleneck.
2. Murder your ggsn/whateverpacketnodeyouwant closer to the core.
3. Anger your paying customers who would like to use packet data successfully on an ios download day.
These people (Apple) represent themselves as smart guys, but their actions reflect otherwise. I bet this would be a larger deal to Nanog people if your Internet stopped working as the result of 100% Linux adoption. That is very close to what this is.. Tens of millions of people trying to update their 13 ios devices at the same time. Who owns a single ios device? A household could do 5-10gb worth of updates in a single day..
I personally do not own an ios device, and I see close to 3 gigs worth of update traffic at my house. These things are everywhere, and this problem will not stop.
Sent from my Mobile Device.
-------- Original message -------- From: Mikael Abrahamsson <swmike@swm.pp.se> Date: 09/19/2013 11:16 AM (GMT-08:00) To: Warren Bailey <wbailey@satelliteintelligencegroup.com> Cc: Paul Ferguson <fergdawgster@mykolab.com>,NANOG <nanog@nanog.org> Subject: Re: iOS 7 update traffic
On Thu, 19 Sep 2013, Warren Bailey wrote:
Why does apple feel it is okay to send every mobile device an update on a single day?
They don't, these are users who actively goes into the software upgrade menu and pressing "upgrade".
I believe the nagging won't start for quite some time.
-- Mikael Abrahamsson email: swmike@swm.pp.se
On 23/09/13 10:32, John Smith wrote:
Picked this off www.jaluri.com (network and Cisco blog aggregator):
http://routingfreak.wordpress.com/2013/09/23/ios7s-impact-on-networks-worldw...
The consensus seems to be for providers to install CDN servers, if they arent able to cope up with an occasional OS update traffic.
http://news.idg.no/cw/art.cfm?id=391B4B64-F693-41B7-6BBAC6D7017C3B8A
John
Perhaps Apple, Microsoft etc. should consider using Bittorrent as a way of distributing their updates? If ISPs were to run their own Bittorrent servers (with appropriate restrictions, see below), this would then create an instant CDN, with no need to define any other protocols or pay any third parties. The hard bit would be to create a way for Apple etc. to be able to authoritatively say "we are the content owners, and are happy for you to replicate this locally": but perhaps this could be as simple serving the initial seed from an HTTPS server with a valid certificate? It would then be trivial to create a whitelist of the domains of the top 10 or so distributors of patches, and then everything would work automatically from then on. -- N.
One of the earlier posts seems to suggest that if iOS updates were cached on the ISPs CDN server then the traffic would have been manageable since everybody would only contact the local sever to get the image. Is this assumption correct? Do most big service providers maintain their own content servers? Is this what we're heading to these days? Glen On Mon, Sep 23, 2013 at 4:29 PM, Neil Harris <neil@tonal.clara.co.uk> wrote:
On 23/09/13 10:32, John Smith wrote:
Picked this off www.jaluri.com (network and Cisco blog aggregator):
http://routingfreak.wordpress.**com/2013/09/23/ios7s-impact-** on-networks-worldwide/<http://routingfreak.wordpress.com/2013/09/23/ios7s-impact-on-networks-worldwide/>
The consensus seems to be for providers to install CDN servers, if they arent able to cope up with an occasional OS update traffic.
John
Perhaps Apple, Microsoft etc. should consider using Bittorrent as a way of distributing their updates? If ISPs were to run their own Bittorrent servers (with appropriate restrictions, see below), this would then create an instant CDN, with no need to define any other protocols or pay any third parties.
The hard bit would be to create a way for Apple etc. to be able to authoritatively say "we are the content owners, and are happy for you to replicate this locally": but perhaps this could be as simple serving the initial seed from an HTTPS server with a valid certificate? It would then be trivial to create a whitelist of the domains of the top 10 or so distributors of patches, and then everything would work automatically from then on.
-- N.
Glen Kent writes:
One of the earlier posts seems to suggest that if iOS updates were cached on the ISPs CDN server then the traffic would have been manageable since everybody would only contact the local sever to get the image. Is this assumption correct?
Not necessarily. I think most of the iOS 7 update traffic WAS in fact delivered from CDN servers (in particular Akamai). And many/most large service providers already have Akamai servers in their networks. But they may not have enough spare capacity for such a sudden demand - either in terms of CDN (Akamai) servers or in terms of capacity between their CDN servers and their customers.
Do most big service providers maintain their own content servers? Is this what we're heading to these days?
Depends on what you mean by "their own". As I said, these days Akamai has servers in many of the big networks. Google and possibly others (Limelight, ...?) might have that as well. But I wouldn't call them "their [the SPs'] own". Some SPs are also built their own CDNs (Level 3) or are talking about it. But that model seems to be less popular with the content owners and the other SPs. -- Simon.
On Sep 23, 2013, at 8:10 AM, Simon Leinen <simon.leinen@switch.ch> wrote:
Not necessarily. I think most of the iOS 7 update traffic WAS in fact delivered from CDN servers (in particular Akamai). And many/most large service providers already have Akamai servers in their networks. But they may not have enough spare capacity for such a sudden demand - either in terms of CDN (Akamai) servers or in terms of capacity between their CDN servers and their customers.
Apple claims 200 million[1] IOS devices upgrade to version 7 in the past week. A typical download was on the order of 800MB. At the same time, Apple released some other updates, like OSX 10.8.5[2] (275MB) and XCode 5.0[3] (2GB). They also made the iWork and iLife applications (Pages, Numbers, Keynote iMovie, and iPhoto) free to download[4] for all new IOS purchasers. Oh, and they sold 9 million iPhone 5s/c devices[1], most of which needed an update to IOS 7.0.1[5] which was a 1.2GB download. With all of that going on the grumbling on NANOG has pretty much been limited to "yeah, we saw a spike", and a few providers of alternative technologies (like Satellite) grousing a bit. I'm not saying the industry can't do better, but I'm finding it hard to describe what happened as anything besides a success for CDN's and most consumer facing ISP's. I only hope the various CDN's and ISP's study what happened so they can be prepared for the next event, which will no doubt be bigger. We're all in an "up and to the right" industry. 1: http://9to5mac.com/2013/09/23/apple-announces-9-million-iphone-sales-over-fi... 2: http://support.apple.com/kb/DL1675 3: http://9to5mac.com/2013/09/18/xcode-5-0-released-with-ios-7-sdk-64-bit-app-c... 4: http://9to5mac.com/2013/09/10/apple-makes-iwork-apps-iphoto-and-imovie-free-... 5: http://support.apple.com/kb/DL1683 -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/
On 2013-09-23, at 09:10, Simon Leinen <simon.leinen@switch.ch> wrote:
Glen Kent writes:
One of the earlier posts seems to suggest that if iOS updates were cached on the ISPs CDN server then the traffic would have been manageable since everybody would only contact the local sever to get the image. Is this assumption correct?
Not necessarily. I think most of the iOS 7 update traffic WAS in fact delivered from CDN servers (in particular Akamai). And many/most large service providers already have Akamai servers in their networks. But they may not have enough spare capacity for such a sudden demand - either in terms of CDN (Akamai) servers or in terms of capacity between their CDN servers and their customers.
I think oversubscription in the access network (between the customer and the ISP network that might contain Akamai nodes) is the general concern, at least from the ISPs I have visibility into. Your access network doesn't have to be a narrowband satellite network for this kind of unexpected demand to hurt, and provisioning extra access bandwidth in anticipation of a one- or two-day possibility of increased demand is not practical. I don't doubt Apple are aware of the issue and will make changes if they can. The characterisation that Apple doesn't care, or is callously causing pain in other networks ignores the commercial reality that user experience is important to them. The user experience when an anticipated update can't be downloaded easily is not great. The suggestions on how to make things better next time that have appeared on this list seem helpful. I would imagine that any vendor with a huge and widely-distributed user base would do well to take note. Joe
On Sep 23, 2013, at 15:10, Simon Leinen <simon.leinen@switch.ch> wrote:
Glen Kent writes:
One of the earlier posts seems to suggest that if iOS updates were cached on the ISPs CDN server then the traffic would have been manageable since everybody would only contact the local sever to get the image. Is this assumption correct?
Not necessarily. I think most of the iOS 7 update traffic WAS in fact delivered from CDN servers (in particular Akamai). And many/most large service providers already have Akamai servers in their networks. But they may not have enough spare capacity for such a sudden demand - either in terms of CDN (Akamai) servers or in terms of capacity between their CDN servers and their customers.
I have some anecdotal evidence that a large swatch of Telekom land in Germany was fed from two (2) Limelight servers in Frankfurt (?). Of course, packet loss to them during Wednesday evening was around 50 %. (I VPNed out of Telekom land to get my iOS 7 update, which was then no problem at all; that clearly shows that the access infrastructure wasn't overloaded.) It doesn't help that Apple's update software has no way to make use of the results of a prematurely aborted transfer; this is a recipe for bistable behavior. Grüße, Carsten
Perhaps Apple, Microsoft etc. should consider using Bittorrent as a way of distributing their updates? If ISPs were to run their own Bittorrent servers (with appropriate restrictions, see below), this would then create an instant CDN, with no need to define any other protocols or pay any third parties.
They should do it like the game vendors. You are able to preload the game but it will only run on the day it is released. But, you still have to make sure not all your customers fetch the content at the same time with full speed. And what about thouse who just own a phone and no computer with iTunes? These customers will prefer to download over wifi as fast as possible to get the upgrade done. Preloading over anything other than wifi is way to expensive. Someone mentioned Win8/8.1 and Upgrades, there will also be OSX Mavericks quite soon. Fewer devices but way more content ... Access is sold by bandwith, so there will be oversubscrition. Sell it by traffic and all thouse iPhone users will walk to the neares Apple store to buy an USB stick with the upgrade since its cheaper than downloading it. rm
BTW Linux distributions are available to download via bittorrent, so we dont really need Akamai/Limelight here. Is there a reason why Apple has not adopted bit-torrent for distribution? Are there legal/commercial implications using bit-torrent? Glen On Mon, Sep 23, 2013 at 4:29 PM, Neil Harris <neil@tonal.clara.co.uk> wrote:
On 23/09/13 10:32, John Smith wrote:
Picked this off www.jaluri.com (network and Cisco blog aggregator):
http://routingfreak.wordpress.**com/2013/09/23/ios7s-impact-** on-networks-worldwide/<http://routingfreak.wordpress.com/2013/09/23/ios7s-impact-on-networks-worldwide/>
The consensus seems to be for providers to install CDN servers, if they arent able to cope up with an occasional OS update traffic.
John
Perhaps Apple, Microsoft etc. should consider using Bittorrent as a way of distributing their updates? If ISPs were to run their own Bittorrent servers (with appropriate restrictions, see below), this would then create an instant CDN, with no need to define any other protocols or pay any third parties.
The hard bit would be to create a way for Apple etc. to be able to authoritatively say "we are the content owners, and are happy for you to replicate this locally": but perhaps this could be as simple serving the initial seed from an HTTPS server with a valid certificate? It would then be trivial to create a whitelist of the domains of the top 10 or so distributors of patches, and then everything would work automatically from then on.
-- N.
On Sep 23, 2013, at 9:41 AM, Glen Kent <glen.kent@gmail.com> wrote:
BTW Linux distributions are available to download via bittorrent, so we dont really need Akamai/Limelight here. Is there a reason why Apple has not adopted bit-torrent for distribution? Are there legal/commercial implications using bit-torrent?
It's more about predictable results and outcome. I can pay a CDN and likely get some sort of reporting/SLA. I can't as easily insure that my torrent traffic will work as well. Some carriers dabbled in doing something about peer-to-peer/torrent type traffic in the past, such as the P4P stuff: http://www.datacenterknowledge.com/archives/2008/03/14/verizon-testing-p4p-f... But I think it died off like many other things. I think CNN.com video still wants the peer-to-peer octoshape thing, but I have always said NO. https://www.google.com/search?q=octoshape+peer+to+peer while an older article from 2009, here's why you should say no as well: http://arstechnica.com/business/2009/02/cnn-p2p-video-streaming-tech-raises-... - Jared
On 2013-09-23 15:41 , Glen Kent wrote:
BTW Linux distributions are available to download via bittorrent,
I am very sure that you will be happy to see your customer's UPSTREAM links filled with that traffic... next to you having a shiny CDN and then having to do traffic to ISPs who do not have one... IMHO the model with CDN is appropriate for this kind of traffic: the pusher is footing the bill. Greets, Jeroen
On 2013-09-23, at 09:41, Glen Kent <glen.kent@gmail.com> wrote:
BTW Linux distributions are available to download via bittorrent, so we dont really need Akamai/Limelight here. Is there a reason why Apple has not adopted bit-torrent for distribution? Are there legal/commercial implications using bit-torrent?
There are upstream congestion issues frequently associated with bittorrent. If you compare (a) five thousand students on a campus wifi network trying to download a 1GB image from a nearby Akamai cache, and (b) five thousand students on a campus wifi network seeding a 1GB image to people all over the world it's not obvious that more pain results from (a) than (b). Even given the ability of Apple to control the behaviour of the bittorrent agent (which presumably would be built into iOS) the impact of such a strategy on an event of this size seems very hard to predict, given a narrow time base and an unknowable number of local network constraints. It doesn't seem impossible to try and optimise the fan-out by giving network operators hooks to influence peer selection based on local topology. But it also doesn't sound like an easy general problem to solve (or a problem that anybody necessarily wants to spend money on if the relief is only going to be felt once per year on major iOS updates). (Remember as well that the scale here is very different. With iOS, Apple is the major Unix vendor on the planet by some margin. No other single Linux or other Unix/Unix-like distribution comes close, and I am guessing no single operating system triggers the update enthusiasm observed with iOS.) Joe
Bit torrent is a way to lighten the load on the originator, and to increase the speed of the acquisition from the receivers. It is not a tool to decrease network load, if anything it does the opposite most of the time. Every now and then, a client will find a local network peer, but its usually an accident more than anything from the algorithm it uses to try to find the fastest senders with pieces it needs. This is most often a product of far end congestion and what pieces are completed, and rarely upstream related barring major issues. The algorithim is self greedy, not altruistic, and definitely not written with ISP load issues in mind. I'd much prefer CDN content over bittorrent from the ISP side. -Blake On Mon, Sep 23, 2013 at 9:02 AM, Joe Abley <jabley@hopcount.ca> wrote:
On 2013-09-23, at 09:41, Glen Kent <glen.kent@gmail.com> wrote:
BTW Linux distributions are available to download via bittorrent, so we dont really need Akamai/Limelight here. Is there a reason why Apple has not adopted bit-torrent for distribution? Are there legal/commercial implications using bit-torrent?
There are upstream congestion issues frequently associated with bittorrent. If you compare
(a) five thousand students on a campus wifi network trying to download a 1GB image from a nearby Akamai cache, and
(b) five thousand students on a campus wifi network seeding a 1GB image to people all over the world
it's not obvious that more pain results from (a) than (b).
Even given the ability of Apple to control the behaviour of the bittorrent agent (which presumably would be built into iOS) the impact of such a strategy on an event of this size seems very hard to predict, given a narrow time base and an unknowable number of local network constraints.
It doesn't seem impossible to try and optimise the fan-out by giving network operators hooks to influence peer selection based on local topology. But it also doesn't sound like an easy general problem to solve (or a problem that anybody necessarily wants to spend money on if the relief is only going to be felt once per year on major iOS updates).
(Remember as well that the scale here is very different. With iOS, Apple is the major Unix vendor on the planet by some margin. No other single Linux or other Unix/Unix-like distribution comes close, and I am guessing no single operating system triggers the update enthusiasm observed with iOS.)
Joe
That's just the typical Bittorrent /client/, but the idea of using Bittorrent means the /protocol/. A special Bittorrent client could be written for ISPs with uploads disabled and Apple could also disable them on the update-downloading Bittorrent client for the phones. The clients (be it Bittorrent or not) would still download the MD5 hash after the download finishes to verify the integrity of the download, and Apple would still be able to measure the amount of downloaded images. For the ISP, this would mean minimal amount of effective downloads. On 09/23/2013 07:31 AM, Blake Dunlap wrote:
Bit torrent is a way to lighten the load on the originator, and to increase the speed of the acquisition from the receivers. It is not a tool to decrease network load, if anything it does the opposite most of the time.
Every now and then, a client will find a local network peer, but its usually an accident more than anything from the algorithm it uses to try to find the fastest senders with pieces it needs. This is most often a product of far end congestion and what pieces are completed, and rarely upstream related barring major issues. The algorithim is self greedy, not altruistic, and definitely not written with ISP load issues in mind.
I'd much prefer CDN content over bittorrent from the ISP side.
That's just the typical Bittorrent /client/, but the idea of using Bittorrent means the /protocol/. A special Bittorrent client could be written for ISPs with uploads disabled and Apple could also disable them on the update-downloading Bittorrent client for the phones.
The clients (be it Bittorrent or not) would still download the MD5 hash after the download finishes to verify the integrity of the download, and Apple would still be able to measure the amount of downloaded images.
So then all the networks that have done $things to BitTorrent to demote it to second-rate traffic will suddenly have a bunch of very angry Apple fans whose downloads are mysteriously having issues. And then - assuming you intend for more things than just Apple to go this route - all the CDN's would need to be redesigned to support BT too. It seems like it'd be simpler for Apple to figure out how to validate a partial download and then resume. It isn't like that would be cutting edge technology. I think I might even have seen it happen before. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
On 9/23/2013 9:36 PM, Joe Greco wrote:
So then all the networks that have done $things to BitTorrent to demote it to second-rate traffic will suddenly have a bunch of very angry Apple fans whose downloads are mysteriously having issues.
Just ask the Blizzard fans (World of Warcraft) about this phenomenon... Jeff
So then all the networks that have done $things to BitTorrent to demote it to second-rate traffic will suddenly have a bunch of very angry Apple fans whose downloads are mysteriously having issues. Just ask the Blizzard fans (World of Warcraft) about this phenomenon...
i love the business plan of preventing the users from getting what they want. i think all my competitors should follow it. randy, who uses a bunch of bittorrent, try btsync for more private dropbox
----- Original Message -----
From: "Randy Bush" <randy@psg.com>
i love the business plan of preventing the users from getting what they want. i think all my competitors should follow it.
Strawman, Randy. Clearly, the Internet is *not* up to the task of 1) updating several dozen million devices 2) on links of various quality, 3) with 650MB to 1.2GB downloads and 4) a client that doesn't understand how to restart 5) all at once, cause, over all, it went very poorly. The people negatively impacted by that poor engineering planning on Apple's part *are Apple's customers*, quite apart from any negative impact it had on Everyone Else. Fixing 4 (which is an easy engineering issue) and 5 (which is an operations policy issue that, by and large, most people in that situation understand), *would have had a direct positive effect on Apple's paying customers*. "Preventing them from getting what they want" is made up, and I'm pretty sure you know that. Staging FW update rollouts over networks is old hat. Even I know better than to make that mistake, and I don't know anything. Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA #natog +1 727 647 1274
On Tue, 24 Sep 2013, Jay Ashworth wrote:
Fixing 4 (which is an easy engineering issue) and 5 (which is an operations policy issue that, by and large, most people in that situation understand), *would have had a direct positive effect on Apple's paying customers*.
Fixing 4 is something apple should do. 5 is a marketing decision for them. People are used to queuing *for things*. If the apple updates break your network instead of just the apple updates, then that is your fault as an ISP. For me, nothing more than the apple updates were broken during those hours, that I could tell anyway. "The Internet" wasn't broken. -- Mikael Abrahamsson email: swmike@swm.pp.se
On Sep 24, 2013, at 12:45 AM, Jay Ashworth <jra@baylink.com> wrote:
Strawman, Randy.
Clearly, the Internet is *not* up to the task of
1) updating several dozen million devices 2) on links of various quality, 3) with 650MB to 1.2GB downloads and 4) a client that doesn't understand how to restart 5) all at once,
cause, over all, it went very poorly.
It went well for most users, it seems the 1-5% of people with "odd" configs are the problem. Keep in mind that on any average day about 3% of the networks out there are broken based on pre-ipv6 day measurements. That is, their IPv4 is completely busted. Having the error/problem rate being down in that area is reasonable to me. How many code-red/slammer scans do you still see a decade on? Overall this was surely a network traffic event, and those that observed the IOS6 impact a year ago realized it would occur again with IOS7 and monitored for it. Not everything will work for everyone, but for the majority of users it was fine. (This from surveying my non-geek friends). Traffic levels will lower about 7 days post-release. Also, NYC and other police departments are advocating people update immediately for the anti-theft upgrades provided in the new software. Let me know how your conversation with them goes. - Jared
----- Original Message -----
From: "Jared Mauch" <jared@puck.nether.net>
It went well for most users, it seems the 1-5% of people with "odd" configs are the problem. [ ... ] IOS7 and monitored for it. Not everything will work for everyone, but for the majority of users it was fine. (This from surveying my non-geek friends).
Hmmm. The number of unsolicited reports I saw and received -- both from upgraders and those who didn't even know it was happening, seemed much higher than would support that. When you're upgrading several hundred million devices, the error rate doesn't have to be that high... But the things you *should* do are still a larger list than the things you *must* do; tragedy of the commons and like that. Apple could *easily* have handled it better. That's my story, and I'm stickin' to it. Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA #natog +1 727 647 1274
Picked this off www.jaluri.com (network and Cisco blog aggregator):
http://routingfreak.wordpress.com/2013/09/23/ios7s-impact-on-networks-worldw...
The consensus seems to be for providers to install CDN servers, if they arent able to cope up with an occasional OS update traffic.
While his argument for installing CDNs is not entirely incorrect, you should take it with a pound of salt, as the author works for a CDN vendor! :-) http://blog.streamingmedia.com/2009/07/alcatellucent-acquires-cdn-technology...
On 24/09/2013 17:55, Glen Kent wrote:
Picked this off www.jaluri.com (network and Cisco blog aggregator):
http://routingfreak.wordpress.com/2013/09/23/ios7s-impact-on-networks-worldw...
The consensus seems to be for providers to install CDN servers, if they arent able to cope up with an occasional OS update traffic.
Hang on a minute..... That last paragraph in his blog sounds awfully similar to something I posted here the other day ! He says on the 23rd of September : Users are paying service providers to deliver their IP packets. If providers cant handle the volume of traffic that they’re being asked to deliver then its probably time for them to reevaluate their commercial structure (charge customers more) or to redesign/overhaul their networks (invest in CDNs, etc). Remember, with everyone connecting to the “cloud”, the traffic that providers will be asked to push is only going to increase with time .. I said on the 20th of September : Your user is paying you to push packets. If that's causing you a problem, you either need to review your commercial structure (i.e. charge people more) or your technical network design. Face the facts, what with everyone jumping on the "cloud" bandwagon, the future is only going to see you pushing more packets, not less ! So if you can't stand the heat, get out of the kitchen (or the xSP industry). Hmmmmmm.......... ;-(
You've been robbed! On 9/24/2013 1:36 PM, Ben wrote:
Hang on a minute.....
That last paragraph in his blog sounds awfully similar to something I posted here the other day !
He says on the 23rd of September : Users are paying service providers to deliver their IP packets. If providers cant handle the volume of traffic that they’re being asked to deliver then its probably time for them to reevaluate their commercial structure (charge customers more) or to redesign/overhaul their networks (invest in CDNs, etc). Remember, with everyone connecting to the “cloud”, the traffic that providers will be asked to push is only going to increase with time ..
I said on the 20th of September : Your user is paying you to push packets. If that's causing you a problem, you either need to review your commercial structure (i.e. charge people more) or your technical network design. Face the facts, what with everyone jumping on the "cloud" bandwagon, the future is only going to see you pushing more packets, not less ! So if you can't stand the heat, get out of the kitchen (or the xSP industry).
Hmmmmmm.......... ;-(
-- Jon Sands
On 09/23/2013 08:36 PM, Joe Greco wrote:
That's just the typical Bittorrent /client/, but the idea of using Bittorrent means the /protocol/. A special Bittorrent client could be written for ISPs with uploads disabled and Apple could also disable them on the update-downloading Bittorrent client for the phones.
The clients (be it Bittorrent or not) would still download the MD5 hash after the download finishes to verify the integrity of the download, and Apple would still be able to measure the amount of downloaded images.
So then all the networks that have done $things to BitTorrent to demote it to second-rate traffic will suddenly have a bunch of very angry Apple fans whose downloads are mysteriously having issues.
No, usually that traffic is demoted right before upstream (or in some way not very near to the provider-edge-to-customer device). Once the download is ready on the ISP, that would be a solved problem. And also, the phone could support two protocols as a transition. It's the easiest solution I've read so far. There are others but not as easy.
And then - assuming you intend for more things than just Apple to go this route - all the CDN's would need to be redesigned to support BT too.
Why can't it be implemented as an independent mean of delivering updates?
It seems like it'd be simpler for Apple to figure out how to validate a partial download and then resume. It isn't like that would be cutting edge technology. I think I might even have seen it happen before.
Validate partial download? How would that help to reduce the overall load on the ISP? That is only limited to reducing the "redundant" traffic, where "redundant" means "twice per device", not "twice per content", which is the real problem. Cheers.
On Mon, Sep 23, 2013 at 08:36:30PM -0500, Joe Greco wrote:
That's just the typical Bittorrent /client/, but the idea of using Bittorrent means the /protocol/. A special Bittorrent client could be written for ISPs with uploads disabled and Apple could also disable them on the update-downloading Bittorrent client for the phones.
The clients (be it Bittorrent or not) would still download the MD5 hash after the download finishes to verify the integrity of the download, and Apple would still be able to measure the amount of downloaded images.
So then all the networks that have done $things to BitTorrent to demote it to second-rate traffic will suddenly have a bunch of very angry Apple fans whose downloads are mysteriously having issues.
Sounds like a win to me. - Matt -- "Python is a rich scripting language offering a lot of the power of C++ while retaining the ease of use of VBscript." -- The PyWin32 documentation
On Thu, 19 Sep 2013, Warren Bailey wrote:
I don't see how operators could tolerate this, honestly. I can't think of a single provider who does not oversubscribe their access platform... Which leads me to this question :
Why does apple feel it is okay to send every mobile device an update on a single day?
Never mind the fact that we are we ones on the last mile responsible for getting it to their customers, 1gb per sub is pretty serious.. Why are they not caching at their head ends, dslams, etc?
As far as I was aware, it was at least staggered throughout the day, so there's some concession. Also a reason to have some CDNs in any large deployment, I guess. I saw a spike in our Akamai traffic, but only slight.
Sent from my Mobile Device.
On Thu, Sep 19, 2013 at 06:11:11PM +0000, Warren Bailey wrote:
I don't see how operators could tolerate this, honestly. I can't think of a single provider who does not oversubscribe their access platform... Which leads me to this question :
Why does apple feel it is okay to send every mobile device an update on a single day?
Never mind the fact that we are we ones on the last mile responsible for getting it to their customers, 1gb per sub is pretty serious.. Why are they not caching at their head ends, dslams, etc?
They didn't make it available to everyone on the same day. My iphone 5 didn't see it until today; I looked yesterday, and it wasn't available for that device. I'm busily contributing to the network stress now. -- Mike Andrews, W5EGO mikea@mikea.ath.cx Tired old sysadmin
On 2013-09-19, at 14:11, Warren Bailey <wbailey@satelliteintelligencegroup.com> wrote:
I don't see how operators could tolerate this, honestly. I can't think of a single provider who does not oversubscribe their access platform... Which leads me to this question :
Why does apple feel it is okay to send every mobile device an update on a single day?
How is this different from the flash crowds caused by hockey championships, or football games, or any of the other things that generate a lot of simultaneous interest every once in a while?
Never mind the fact that we are we ones on the last mile responsible for getting it to their customers, 1gb per sub is pretty serious.. Why are they not caching at their head ends, dslams, etc?
Given that the code is signed, I'm surprised that iDevices that have already upgraded the hard way don't advertise a "update available" service on local networks. Individual devices don't care where the updates come from, so long as the signatures are good. You'd think that'd have the potential to improve the user experience as well as avoid jamming the tubes, especially in highly multi-user environments like university campuses; it could probably halve the network load in a significant number of home networks, too. Joe
http://images.mirror.co.uk/upl/m4/jun2011/6/0/image-5-for-riots-break-out-a fter-vancouver-canucks-lose-the-nhl-stanley-cup-playoffs-to-the-boston-brui ns-gallery-116084753.jpg Good example of the flash crowds post hockey championship It's not all butterflies, Abley.. LOL On 9/19/13 11:42 AM, "Joe Abley" <jabley@hopcount.ca> wrote:
On 2013-09-19, at 14:11, Warren Bailey <wbailey@satelliteintelligencegroup.com> wrote:
I don't see how operators could tolerate this, honestly. I can't think of a single provider who does not oversubscribe their access platform... Which leads me to this question :
Why does apple feel it is okay to send every mobile device an update on a single day?
How is this different from the flash crowds caused by hockey championships, or football games, or any of the other things that generate a lot of simultaneous interest every once in a while?
Never mind the fact that we are we ones on the last mile responsible for getting it to their customers, 1gb per sub is pretty serious.. Why are they not caching at their head ends, dslams, etc?
Given that the code is signed, I'm surprised that iDevices that have already upgraded the hard way don't advertise a "update available" service on local networks. Individual devices don't care where the updates come from, so long as the signatures are good.
You'd think that'd have the potential to improve the user experience as well as avoid jamming the tubes, especially in highly multi-user environments like university campuses; it could probably halve the network load in a significant number of home networks, too.
Joe
On Thu, Sep 19, 2013 at 02:42:12PM -0400, Joe Abley wrote:
Given that the code is signed, I'm surprised that iDevices that have already upgraded the hard way don't advertise a "update available" service on local networks. Individual devices don't care where the updates come from, so long as the signatures are good.
Going the other way, Apple will have local update caching as part of MDM for iOS 7 when it is fully upgraded and rolled out to support iOS for enterprises. But the big push with BYOD is that employees manage their own iDevices..
On Sep 19, 2013, at 2:11 PM, Warren Bailey <wbailey@satelliteintelligencegroup.com> wrote:
Why does apple feel it is okay to send every mobile device an update on a single day?
Apple does not "send" updates. The user device must request an update. --As a side note, IOS 7 fixes/improves iDevices in multiple areas, making it a compelling upgrade. James R. Cutler james.cutler@consultant.com
A line, is a line, is a line, is a line. There's no difference. Updates are available to all devices on a "download day", and providers networks are drastically reduced in capacity as a result. Apple does not cut them checks to serve it up, why should that traffic be more important than anything else? I'd DSCP updates to best effort hell and tell Apple I'd like a small share of the revenue they've gained from all the devices *I* am responsible for updating. They're not getting these updates OTA often, they actually advocate (shocking, AT&T wanting to save bandwidth) using your home Wi-Fi to download it. Providers can handle peaks, but SURGES begin to cause problems quickly. On narrowband pipes, we actually KILL updates.. They screw us that hard. On 9/19/13 11:43 AM, "Cutler James R" <james.cutler@consultant.com> wrote:
On Sep 19, 2013, at 2:11 PM, Warren Bailey <wbailey@satelliteintelligencegroup.com> wrote:
Why does apple feel it is okay to send every mobile device an update on a single day?
Apple does not "send" updates. The user device must request an update.
--As a side note, IOS 7 fixes/improves iDevices in multiple areas, making it a compelling upgrade.
James R. Cutler james.cutler@consultant.com
Woah there. I think you are crossing another line, or at least opening another topic of discussion, when you start talking about transit or last mile providers charging companies for bandwidth that their customers are already paying for. I'd suggest a subject change if we want to open a discussion on that topic. On 9/19/13 2:46 PM, "Warren Bailey" <wbailey@satelliteintelligencegroup.com> wrote:
A line, is a line, is a line, is a line.
There's no difference. Updates are available to all devices on a "download day", and providers networks are drastically reduced in capacity as a result. Apple does not cut them checks to serve it up, why should that traffic be more important than anything else? I'd DSCP updates to best effort hell and tell Apple I'd like a small share of the revenue they've gained from all the devices *I* am responsible for updating. They're not getting these updates OTA often, they actually advocate (shocking, AT&T wanting to save bandwidth) using your home Wi-Fi to download it. Providers can handle peaks, but SURGES begin to cause problems quickly. On narrowband pipes, we actually KILL updates.. They screw us that hard.
On 9/19/13 11:43 AM, "Cutler James R" <james.cutler@consultant.com> wrote:
On Sep 19, 2013, at 2:11 PM, Warren Bailey <wbailey@satelliteintelligencegroup.com> wrote:
Why does apple feel it is okay to send every mobile device an update on a single day?
Apple does not "send" updates. The user device must request an update.
--As a side note, IOS 7 fixes/improves iDevices in multiple areas, making it a compelling upgrade.
James R. Cutler james.cutler@consultant.com
On 13-09-19 02:46 PM, Warren Bailey wrote:
A line, is a line, is a line, is a line.
There's no difference. Updates are available to all devices on a "download day", and providers networks are drastically reduced in capacity as a result. Apple does not cut them checks to serve it up, why should that traffic be more important than anything else? I'd DSCP updates to best effort hell and tell Apple I'd like a small share of the revenue they've gained from all the devices *I* am responsible for updating. They're not getting these updates OTA often, they actually advocate (shocking, AT&T wanting to save bandwidth) using your home Wi-Fi to download it. Providers can handle peaks, but SURGES begin to cause problems quickly. On narrowband pipes, we actually KILL updates.. They screw us that hard. You fail at internet, please try again later.
Sounds like a great plan. You could do it for Netflix, Hulu, amazon, Walmart, etc. Get a piece of the action. Am I talking to Verizon? On 9/19/13 1:46 PM, Warren Bailey wrote:
A line, is a line, is a line, is a line.
There's no difference. Updates are available to all devices on a "download day", and providers networks are drastically reduced in capacity as a result. Apple does not cut them checks to serve it up, why should that traffic be more important than anything else? I'd DSCP updates to best effort hell and tell Apple I'd like a small share of the revenue they've gained from all the devices *I* am responsible for updating. They're not getting these updates OTA often, they actually advocate (shocking, AT&T wanting to save bandwidth) using your home Wi-Fi to download it. Providers can handle peaks, but SURGES begin to cause problems quickly. On narrowband pipes, we actually KILL updates.. They screw us that hard.
On Thu, 19 Sep 2013, Cutler James R wrote:
--As a side note, IOS 7 fixes/improves iDevices in multiple areas, making it a compelling upgrade.
That's supposed to be the nature of upgrades. If that's where the matter ended then you'd have no argument. The problem is when it comes to the new bugs that get introduced, and whether they affect features you care about. I'm holding back until next week :) (And after previous bitter experience with over-the-air upgrade, I've downloaded it through iTunes to a desktop and will upgrade it by wire at my leisure). Jethro. . . . . . . . . . . . . . . . . . . . . . . . . . Jethro R Binks, Network Manager, Information Services Directorate, University Of Strathclyde, Glasgow, UK The University of Strathclyde is a charitable body, registered in Scotland, number SC015263.
Why should Apple care if providers have oversubscribed lines or not? As far as I know, Akamai delivers most of the data anyway, so it is not coming all from Apple. I don't know for sure, but I doubt they have enough bandwidth themselves to saturate so many links concurrently. Apple also does not push the updates, it is pulled to the device when the users tell the device to retrieve it. So blame your users, not Apple. It is also my understanding that any updates they do push are staged so they all don't go out the same time. On 9/19/13 2:11 PM, "Warren Bailey" <wbailey@satelliteintelligencegroup.com> wrote:
I don't see how operators could tolerate this, honestly. I can't think of a single provider who does not oversubscribe their access platform... Which leads me to this question :
Why does apple feel it is okay to send every mobile device an update on a single day?
Never mind the fact that we are we ones on the last mile responsible for getting it to their customers, 1gb per sub is pretty serious.. Why are they not caching at their head ends, dslams, etc?
Sent from my Mobile Device.
-------- Original message -------- From: Mikael Abrahamsson <swmike@swm.pp.se> Date: 09/19/2013 11:08 AM (GMT-08:00) To: Paul Ferguson <fergdawgster@mykolab.com> Cc: NANOG <nanog@nanog.org> Subject: Re: iOS 7 update traffic
On Thu, 19 Sep 2013, Paul Ferguson wrote:
Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this...
The IOS7 upgrade is ~750 megabyte download for the phones/pods, and ~950 megabytes for ipad. There are quite a few devices out there times these amounts to download...
-- Mikael Abrahamsson email: swmike@swm.pp.se
My.. Our.. Users expect one thing.. Internet. It is our job to make that happen. When a electronics manufacturer decides to enable updates for all of their phones world wide.. It breaks things. When the Internet breaks, it is my fault. Your Apple update sucked because of me.. There is no "it must be apple", as you pointed out earlier. I'm simply saying.. It's a dick move to globally enable updates on a single day and tell ISP's to deal with it. Sent from my Mobile Device. -------- Original message -------- From: Fred Reimer <freimer@freimer.org> Date: 09/19/2013 11:48 AM (GMT-08:00) To: Warren Bailey <wbailey@satelliteintelligencegroup.com>,Mikael Abrahamsson <swmike@swm.pp.se>,Paul Ferguson <fergdawgster@mykolab.com> Cc: NANOG <nanog@nanog.org> Subject: Re: iOS 7 update traffic Why should Apple care if providers have oversubscribed lines or not? As far as I know, Akamai delivers most of the data anyway, so it is not coming all from Apple. I don't know for sure, but I doubt they have enough bandwidth themselves to saturate so many links concurrently. Apple also does not push the updates, it is pulled to the device when the users tell the device to retrieve it. So blame your users, not Apple. It is also my understanding that any updates they do push are staged so they all don't go out the same time. On 9/19/13 2:11 PM, "Warren Bailey" <wbailey@satelliteintelligencegroup.com> wrote:
I don't see how operators could tolerate this, honestly. I can't think of a single provider who does not oversubscribe their access platform... Which leads me to this question :
Why does apple feel it is okay to send every mobile device an update on a single day?
Never mind the fact that we are we ones on the last mile responsible for getting it to their customers, 1gb per sub is pretty serious.. Why are they not caching at their head ends, dslams, etc?
Sent from my Mobile Device.
-------- Original message -------- From: Mikael Abrahamsson <swmike@swm.pp.se> Date: 09/19/2013 11:08 AM (GMT-08:00) To: Paul Ferguson <fergdawgster@mykolab.com> Cc: NANOG <nanog@nanog.org> Subject: Re: iOS 7 update traffic
On Thu, 19 Sep 2013, Paul Ferguson wrote:
Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this...
The IOS7 upgrade is ~750 megabyte download for the phones/pods, and ~950 megabytes for ipad. There are quite a few devices out there times these amounts to download...
-- Mikael Abrahamsson email: swmike@swm.pp.se
Dorian, It seems warren may work for a satellite internet provider. (Just guessing). The impact might be different with this type of a link. There isn't a good broadcast caching system for this compared with the way other content is. This may have that type of an impact, but there are likely ways to address this as well. I recall the sky cache and other systems of the day. Jared Mauch
On Sep 19, 2013, at 2:41 PM, Dorian Kim <dorian@blackrose.org> wrote:
On Thu, Sep 19, 2013 at 06:52:51PM +0000, Warren Bailey wrote: My.. Our.. Users expect one thing..
Internet.
Isn't the ability to download something that they want part of the Internet thing that users expect from their service providers?
-dorian
O.K., I understand. Yes, for the average user I suppose they would blame their ISP. I was making the wrong assumption that people understood how the Internet works. At the same time, people would probably be more upset, at least the Apple fanboys, if they metered the updates and some people had to wait two or three weeks for their update to keep the traffic manageable. The only general news stories I see in a quick search are complaints that the downloads are slow, not that the general Internet is slow because of the downloadsŠ From: Warren Bailey <wbailey@satelliteintelligencegroup.com> Reply-To: Warren Bailey <wbailey@satelliteintelligencegroup.com> Date: Thursday, September 19, 2013 2:52 PM To: Fred Reimer <freimer@freimer.org>, Mikael Abrahamsson <swmike@swm.pp.se>, Paul Ferguson <fergdawgster@mykolab.com> Cc: NANOG <nanog@nanog.org> Subject: Re: iOS 7 update traffic
My.. Our.. Users expect one thing..
Internet.
It is our job to make that happen. When a electronics manufacturer decides to enable updates for all of their phones world wide.. It breaks things.
When the Internet breaks, it is my fault. Your Apple update sucked because of me.. There is no "it must be apple", as you pointed out earlier. I'm simply saying.. It's a dick move to globally enable updates on a single day and tell ISP's to deal with it.
Sent from my Mobile Device.
-------- Original message -------- From: Fred Reimer <freimer@freimer.org> Date: 09/19/2013 11:48 AM (GMT-08:00) To: Warren Bailey <wbailey@satelliteintelligencegroup.com>,Mikael Abrahamsson <swmike@swm.pp.se>,Paul Ferguson <fergdawgster@mykolab.com>
Cc: NANOG <nanog@nanog.org> Subject: Re: iOS 7 update traffic
Why should Apple care if providers have oversubscribed lines or not? As far as I know, Akamai delivers most of the data anyway, so it is not coming all from Apple. I don't know for sure, but I doubt they have enough bandwidth themselves to saturate so many links concurrently. Apple also does not push the updates, it is pulled to the device when the users tell the device to retrieve it. So blame your users, not Apple. It is also my understanding that any updates they do push are staged so they all don't go out the same time.
On 9/19/13 2:11 PM, "Warren Bailey" <wbailey@satelliteintelligencegroup.com> wrote:
I don't see how operators could tolerate this, honestly. I can't think of a single provider who does not oversubscribe their access platform... Which leads me to this question :
Why does apple feel it is okay to send every mobile device an update on a single day?
Never mind the fact that we are we ones on the last mile responsible for getting it to their customers, 1gb per sub is pretty serious.. Why are they not caching at their head ends, dslams, etc?
Sent from my Mobile Device.
-------- Original message -------- From: Mikael Abrahamsson <swmike@swm.pp.se> Date: 09/19/2013 11:08 AM (GMT-08:00) To: Paul Ferguson <fergdawgster@mykolab.com> Cc: NANOG <nanog@nanog.org> Subject: Re: iOS 7 update traffic
On Thu, 19 Sep 2013, Paul Ferguson wrote:
Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this...
The IOS7 upgrade is ~750 megabyte download for the phones/pods, and ~950 megabytes for ipad. There are quite a few devices out there times these amounts to download...
-- Mikael Abrahamsson email: swmike@swm.pp.se
How about add a cache in your time capsule/thing that everyone connects to? I mean, would it be THAT hard to enable a bonjour update server on an apple router/computer/whatever and serve things up locally from there? I've had many replies to this email already, and people are talking about upgrading bandwidth and CDN's. We own and operate a satellite data network. It is narrow band.. It is very expensive. Oversub matters.. Time slots matter. When we (industry, collectively) have to deal with hundreds of gigs worth of traffic it's a tough day. I just can't understand why these big boys have forgotten most of the planet doesn't have a 1000000gbps pipe anywhere near them, a lot of the earth actually doesn't have communications infrastructure at all. The quick "WE HAVE ENOUGH INTERNETS NOW" doesn't apply to every system, nor does it explain why an update needs to be sent relentlessly to individual devices (requested or not) over the course of a product's evolution. Things are not created equal amongst internet providers, a transponder (90mbps ish) runs us close to 160k a month and that's not including gear costs, teleport, etc. We strive to provide a great customer experience, and when "Hardware Maker X" decides to roll updates .. It can screw us. In this case, can means absolutely will happen. My .02. Not trying to start a flame thing or tell people what's what.. Just trying to get a point across. You don't need to send things individually now.. We live in the future.. We should act like it. On 9/19/13 12:10 PM, "Fred Reimer" <freimer@freimer.org> wrote:
O.K., I understand. Yes, for the average user I suppose they would blame their ISP. I was making the wrong assumption that people understood how the Internet works. At the same time, people would probably be more upset, at least the Apple fanboys, if they metered the updates and some people had to wait two or three weeks for their update to keep the traffic manageable. The only general news stories I see in a quick search are complaints that the downloads are slow, not that the general Internet is slow because of the downloadsŠ
From: Warren Bailey <wbailey@satelliteintelligencegroup.com> Reply-To: Warren Bailey <wbailey@satelliteintelligencegroup.com> Date: Thursday, September 19, 2013 2:52 PM To: Fred Reimer <freimer@freimer.org>, Mikael Abrahamsson <swmike@swm.pp.se>, Paul Ferguson <fergdawgster@mykolab.com> Cc: NANOG <nanog@nanog.org> Subject: Re: iOS 7 update traffic
My.. Our.. Users expect one thing..
Internet.
It is our job to make that happen. When a electronics manufacturer decides to enable updates for all of their phones world wide.. It breaks things.
When the Internet breaks, it is my fault. Your Apple update sucked because of me.. There is no "it must be apple", as you pointed out earlier. I'm simply saying.. It's a dick move to globally enable updates on a single day and tell ISP's to deal with it.
Sent from my Mobile Device.
-------- Original message -------- From: Fred Reimer <freimer@freimer.org> Date: 09/19/2013 11:48 AM (GMT-08:00) To: Warren Bailey <wbailey@satelliteintelligencegroup.com>,Mikael Abrahamsson <swmike@swm.pp.se>,Paul Ferguson <fergdawgster@mykolab.com>
Cc: NANOG <nanog@nanog.org> Subject: Re: iOS 7 update traffic
Why should Apple care if providers have oversubscribed lines or not? As far as I know, Akamai delivers most of the data anyway, so it is not coming all from Apple. I don't know for sure, but I doubt they have enough bandwidth themselves to saturate so many links concurrently. Apple also does not push the updates, it is pulled to the device when the users tell the device to retrieve it. So blame your users, not Apple. It is also my understanding that any updates they do push are staged so they all don't go out the same time.
On 9/19/13 2:11 PM, "Warren Bailey" <wbailey@satelliteintelligencegroup.com> wrote:
I don't see how operators could tolerate this, honestly. I can't think of a single provider who does not oversubscribe their access platform... Which leads me to this question :
Why does apple feel it is okay to send every mobile device an update on a single day?
Never mind the fact that we are we ones on the last mile responsible for getting it to their customers, 1gb per sub is pretty serious.. Why are they not caching at their head ends, dslams, etc?
Sent from my Mobile Device.
-------- Original message -------- From: Mikael Abrahamsson <swmike@swm.pp.se> Date: 09/19/2013 11:08 AM (GMT-08:00) To: Paul Ferguson <fergdawgster@mykolab.com> Cc: NANOG <nanog@nanog.org> Subject: Re: iOS 7 update traffic
On Thu, 19 Sep 2013, Paul Ferguson wrote:
Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this...
The IOS7 upgrade is ~750 megabyte download for the phones/pods, and ~950 megabytes for ipad. There are quite a few devices out there times these amounts to download...
-- Mikael Abrahamsson email: swmike@swm.pp.se
On Thu, 19 Sep 2013 19:18:29 -0000, Warren Bailey said: Reversing a few paragraphs to make a point.
We strive to provide a great customer experience, and when "Hardware Maker X" decides to roll updates .. It can screw us. In this case, can means absolutely will happen.
I mean, would it be THAT hard to enable a bonjour update server on an apple router/computer/whatever and serve things up locally from there? I've had many replies to this email already, and people are talking about upgrading bandwidth and CDN's
So why didn't you?
Things are not created equal amongst internet providers, a transponder (90mbps ish) runs us close to 160k a month and that's not including gear costs, teleport, etc.
And you pay Apple *how* much to guarantee that they don't do things that upset the business model you consciously chose to use? Oh, you don't pay them? And your users pay *you* to ensure that when they hit 'Download', that magical things happen? And iOS downloads are user pulls, not Apple pushes? Sounds to me like you and your users need to have a chat about what they pay for and what their expectations should be.....
I'm willing and open to hear anyone who has successfully had that conversation with their users. When network congestion occurs, we typically see a mass exodus from whatever website was being used to Speedtest.. You know.. Just to make sure the internets are fast. I'm trying to highlight a point that not all of us have studly 1gbps connections to Akamai. Some of us have to move data into orbit and back.. Some of us are not like the rest of you. These types of situations should not happen in general.. We live in the future. This is like sending a bulk fax to every user on a switch, and when the other users get busy signals I somehow need to realign my view of reality. A single Internet point or software update should not cause all of this discussion. You guys are collectively posting hundreds of gbps for basically a single software update, and comparing it to point releases from vendors. Why do I feel like many of you are spoiled with all of this cheap and fast bandwidth? Do you guys not remember your 9600bps modem? Many of you would have suffered heart failure if I sent you a 100mb file only 10 years ago. Keep that in mind.. Not everyone has their Internet coming off the end of an sfp. Sent from my Mobile Device. -------- Original message -------- From: Valdis.Kletnieks@vt.edu Date: 09/19/2013 1:42 PM (GMT-08:00) To: Warren Bailey <wbailey@satelliteintelligencegroup.com> Cc: Fred Reimer <freimer@freimer.org>,Mikael Abrahamsson <swmike@swm.pp.se>,Paul Ferguson <fergdawgster@mykolab.com>,NANOG <nanog@nanog.org> Subject: Re: iOS 7 update traffic On Thu, 19 Sep 2013 19:18:29 -0000, Warren Bailey said: Reversing a few paragraphs to make a point.
We strive to provide a great customer experience, and when "Hardware Maker X" decides to roll updates .. It can screw us. In this case, can means absolutely will happen.
I mean, would it be THAT hard to enable a bonjour update server on an apple router/computer/whatever and serve things up locally from there? I've had many replies to this email already, and people are talking about upgrading bandwidth and CDN's
So why didn't you?
Things are not created equal amongst internet providers, a transponder (90mbps ish) runs us close to 160k a month and that's not including gear costs, teleport, etc.
And you pay Apple *how* much to guarantee that they don't do things that upset the business model you consciously chose to use? Oh, you don't pay them? And your users pay *you* to ensure that when they hit 'Download', that magical things happen? And iOS downloads are user pulls, not Apple pushes? Sounds to me like you and your users need to have a chat about what they pay for and what their expectations should be.....
On Thu, 19 Sep 2013, Warren Bailey wrote:
I'm trying to highlight a point that not all of us have studly 1gbps connections to Akamai. Some of us have to move data into orbit and back.. Some of us are not like the rest of you. These types of situations should not happen in general.. We live in the future. This is like sending a bulk fax to every user on a switch, and when the other users get busy signals I somehow need to realign my view of reality.
I suspect that Apple and others did not take the latency and bandwidth limitations of satellite Internet access into account when developing their software distribution model - if they took such things into account at all. If I had to guess, they probably used basic DSL, cable modem, or 4G LTE service as their lowest common denominator. I would guess their assumption is that the vast majority of people who own iThings have an equivalent kind of Internet connection at their disposal. I'm not saying that's right or wrong, but I wouldn't be surprised if that's their thought process. I fully understand that it's very difficult and expensive to evolve the orbiting pieces of your network to get more bandwidth. Designing, building, flying, and operating a new bird is obscenely expensive. Higher data rates impose limitations of their own, re: rain fade, etc, plus the CPE might need to be replaced.
A single Internet point or software update should not cause all of this discussion. You guys are collectively posting hundreds of gbps for basically a single software update, and comparing it to point releases from vendors.
What happened is not an unprecedented occurrence. I used to work at an ISP, and I remember our customers crushing our pipes during an infamous Victoria's Secret webcast many moons ago. Many websites for major news outlets seized up on 9/11 because they couldn't handle the number of requests that were coming in. Joe Schmoe's website blows up because he was featured in an article on Slashdot. That was before the days when geographically dispersed content distribution became commonplace and websites weren't nearly as content-heavy as they are today.
Why do I feel like many of you are spoiled with all of this cheap and fast bandwidth? Do you guys not remember your 9600bps modem?
I won't disagree (happy with my fios service, other than the complete inability to get a straight answer from anyone in charge re: when IPv6 will be available in my area but that's a topic for another thread), and yes, I do remember 9600 bps modems. I started at 1200 bps, and my first reaction when I got a 2400 bps modem was 'Holy crap is this fast!' ;) For some reason, I recall the jump from 2400 to 14400 'feeling' somewhat less awe-inspiring.
Many of you would have suffered heart failure if I sent you a 100mb file only 10 years ago. Keep that in mind.. Not everyone has their Internet coming off the end of an sfp.
I didn't have a hard drive with enough space to store 100 MB of anything at that point ;) jms
Actually, I started out with a 300 baud acoustic modem. You know, the kind where you take the handset and jam it into two cups? But I digress… From: Warren Bailey <wbailey@satelliteintelligencegroup.com<mailto:wbailey@satelliteintelligencegroup.com>> Reply-To: Warren Bailey <wbailey@satelliteintelligencegroup.com<mailto:wbailey@satelliteintelligencegroup.com>> Date: Thursday, September 19, 2013 5:00 PM To: Valdis Kletnieks <Valdis.Kletnieks@vt.edu<mailto:Valdis.Kletnieks@vt.edu>> Cc: Fred Reimer <freimer@freimer.org<mailto:freimer@freimer.org>>, Mikael Abrahamsson <swmike@swm.pp.se<mailto:swmike@swm.pp.se>>, Paul Ferguson <fergdawgster@mykolab.com<mailto:fergdawgster@mykolab.com>>, NANOG <nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Re: iOS 7 update traffic I'm willing and open to hear anyone who has successfully had that conversation with their users. When network congestion occurs, we typically see a mass exodus from whatever website was being used to Speedtest.. You know.. Just to make sure the internets are fast. I'm trying to highlight a point that not all of us have studly 1gbps connections to Akamai. Some of us have to move data into orbit and back.. Some of us are not like the rest of you. These types of situations should not happen in general.. We live in the future. This is like sending a bulk fax to every user on a switch, and when the other users get busy signals I somehow need to realign my view of reality. A single Internet point or software update should not cause all of this discussion. You guys are collectively posting hundreds of gbps for basically a single software update, and comparing it to point releases from vendors. Why do I feel like many of you are spoiled with all of this cheap and fast bandwidth? Do you guys not remember your 9600bps modem? Many of you would have suffered heart failure if I sent you a 100mb file only 10 years ago. Keep that in mind.. Not everyone has their Internet coming off the end of an sfp. Sent from my Mobile Device. -------- Original message -------- From: Valdis.Kletnieks@vt.edu<mailto:Valdis.Kletnieks@vt.edu> Date: 09/19/2013 1:42 PM (GMT-08:00) To: Warren Bailey <wbailey@satelliteintelligencegroup.com<mailto:wbailey@satelliteintelligencegroup.com>> Cc: Fred Reimer <freimer@freimer.org<mailto:freimer@freimer.org>>,Mikael Abrahamsson <swmike@swm.pp.se<mailto:swmike@swm.pp.se>>,Paul Ferguson <fergdawgster@mykolab.com<mailto:fergdawgster@mykolab.com>>,NANOG <nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Re: iOS 7 update traffic On Thu, 19 Sep 2013 19:18:29 -0000, Warren Bailey said: Reversing a few paragraphs to make a point.
We strive to provide a great customer experience, and when "Hardware Maker X" decides to roll updates .. It can screw us. In this case, can means absolutely will happen.
I mean, would it be THAT hard to enable a bonjour update server on an apple router/computer/whatever and serve things up locally from there? I've had many replies to this email already, and people are talking about upgrading bandwidth and CDN's
So why didn't you?
Things are not created equal amongst internet providers, a transponder (90mbps ish) runs us close to 160k a month and that's not including gear costs, teleport, etc.
And you pay Apple *how* much to guarantee that they don't do things that upset the business model you consciously chose to use? Oh, you don't pay them? And your users pay *you* to ensure that when they hit 'Download', that magical things happen? And iOS downloads are user pulls, not Apple pushes? Sounds to me like you and your users need to have a chat about what they pay for and what their expectations should be.....
So you understand things aren't always metro e.. That's what I was trying to say. I still have a coupler.. ;) Sent from my Mobile Device. -------- Original message -------- From: Fred Reimer <freimer@freimer.org> Date: 09/19/2013 2:14 PM (GMT-08:00) To: Warren Bailey <wbailey@satelliteintelligencegroup.com>,Valdis.Kletnieks@vt.edu Cc: Mikael Abrahamsson <swmike@swm.pp.se>,Paul Ferguson <fergdawgster@mykolab.com>,NANOG <nanog@nanog.org> Subject: Re: iOS 7 update traffic Actually, I started out with a 300 baud acoustic modem. You know, the kind where you take the handset and jam it into two cups? But I digress… From: Warren Bailey <wbailey@satelliteintelligencegroup.com<mailto:wbailey@satelliteintelligencegroup.com>> Reply-To: Warren Bailey <wbailey@satelliteintelligencegroup.com<mailto:wbailey@satelliteintelligencegroup.com>> Date: Thursday, September 19, 2013 5:00 PM To: Valdis Kletnieks <Valdis.Kletnieks@vt.edu<mailto:Valdis.Kletnieks@vt.edu>> Cc: Fred Reimer <freimer@freimer.org<mailto:freimer@freimer.org>>, Mikael Abrahamsson <swmike@swm.pp.se<mailto:swmike@swm.pp.se>>, Paul Ferguson <fergdawgster@mykolab.com<mailto:fergdawgster@mykolab.com>>, NANOG <nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Re: iOS 7 update traffic I'm willing and open to hear anyone who has successfully had that conversation with their users. When network congestion occurs, we typically see a mass exodus from whatever website was being used to Speedtest.. You know.. Just to make sure the internets are fast. I'm trying to highlight a point that not all of us have studly 1gbps connections to Akamai. Some of us have to move data into orbit and back.. Some of us are not like the rest of you. These types of situations should not happen in general.. We live in the future. This is like sending a bulk fax to every user on a switch, and when the other users get busy signals I somehow need to realign my view of reality. A single Internet point or software update should not cause all of this discussion. You guys are collectively posting hundreds of gbps for basically a single software update, and comparing it to point releases from vendors. Why do I feel like many of you are spoiled with all of this cheap and fast bandwidth? Do you guys not remember your 9600bps modem? Many of you would have suffered heart failure if I sent you a 100mb file only 10 years ago. Keep that in mind.. Not everyone has their Internet coming off the end of an sfp. Sent from my Mobile Device. -------- Original message -------- From: Valdis.Kletnieks@vt.edu<mailto:Valdis.Kletnieks@vt.edu> Date: 09/19/2013 1:42 PM (GMT-08:00) To: Warren Bailey <wbailey@satelliteintelligencegroup.com<mailto:wbailey@satelliteintelligencegroup.com>> Cc: Fred Reimer <freimer@freimer.org<mailto:freimer@freimer.org>>,Mikael Abrahamsson <swmike@swm.pp.se<mailto:swmike@swm.pp.se>>,Paul Ferguson <fergdawgster@mykolab.com<mailto:fergdawgster@mykolab.com>>,NANOG <nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Re: iOS 7 update traffic On Thu, 19 Sep 2013 19:18:29 -0000, Warren Bailey said: Reversing a few paragraphs to make a point.
We strive to provide a great customer experience, and when "Hardware Maker X" decides to roll updates .. It can screw us. In this case, can means absolutely will happen.
I mean, would it be THAT hard to enable a bonjour update server on an apple router/computer/whatever and serve things up locally from there? I've had many replies to this email already, and people are talking about upgrading bandwidth and CDN's
So why didn't you?
Things are not created equal amongst internet providers, a transponder (90mbps ish) runs us close to 160k a month and that's not including gear costs, teleport, etc.
And you pay Apple *how* much to guarantee that they don't do things that upset the business model you consciously chose to use? Oh, you don't pay them? And your users pay *you* to ensure that when they hit 'Download', that magical things happen? And iOS downloads are user pulls, not Apple pushes? Sounds to me like you and your users need to have a chat about what they pay for and what their expectations should be.....
On 9/19/2013 5:29 PM, Warren Bailey wrote:
So you understand things aren't always metro e.. That's what I was trying to say. I still have a coupler.. ;)
-------- Original message -------- From: Fred Reimer <freimer@freimer.org>
Actually, I started out with a 300 baud acoustic modem. You know, the kind where you take the handset and jam it into two cups? But I digress…
Bah! That was a take-home convenience. How about the old ASR TeleType with the 110-baud link to get a hardcopy listing? Jeff
On 19/09/2013 9:29 PM, Jeff Kell wrote:
On 9/19/2013 5:29 PM, Warren Bailey wrote:
So you understand things aren't always metro e.. That's what I was trying to say. I still have a coupler.. ;)
-------- Original message -------- From: Fred Reimer <freimer@freimer.org>
Actually, I started out with a 300 baud acoustic modem. You know, the kind where you take the handset and jam it into two cups? But I digress…
Bah! That was a take-home convenience. How about the old ASR TeleType with the 110-baud link to get a hardcopy listing?
Jeff
Ah, distinctly I remember. The ASR shook the whole house when it started typing things out. I was using a government tie line from Ottawa to Toronto, and occasionally the operator would break in to investigate the long call with funny noises. Tom
On Sep 19, 2013, at 3:10 PM, Fred Reimer <freimer@freimer.org> wrote:
I was making the wrong assumption that people understood how the Internet works.
Absolutely! Most "people" understand that the internet works by use of a browser and are content with that knowledge. Much like most motor vehicle operators understand accelerator, brake, steering, and gas pump, with no knowledge of thermodynamics, much less physics or traffic laws. James R. Cutler james.cutler@consultant.com
I might agree if there were no warning, but this has happened a few times a year for many years. It's a predictable pattern and well known. It will last about a week and taper off. Jared Mauch
On Sep 19, 2013, at 2:52 PM, Warren Bailey <wbailey@satelliteintelligencegroup.com> wrote:
It's a dick move to globally enable updates on a single day and tell ISP's to deal with it.
Composed on a virtual keyboard, please forgive typos. On Sep 19, 2013, at 14:11, Warren Bailey <wbailey@satelliteintelligencegroup.com> wrote:
I don't see how operators could tolerate this, honestly. I can't think of a single provider who does not oversubscribe their access platform... Which leads me to this question :
Why does apple feel it is okay to send every mobile device an update on a single day?
That question makes no sense to me. Turn that around: Why would Apple think that is not OK?
Never mind the fact that we are we ones on the last mile responsible for getting it to their customers, 1gb per sub is pretty serious.. Why are they not caching at their head ends, dslams, etc?
Most providers are offered a cache for free (there is a minimum traffic volume, but it is not even as large as Netflix's requirements). Every provider, regardless of traffic, is offered peering for free. What was the problem again? -- TTFN, patrick
-------- Original message -------- From: Mikael Abrahamsson <swmike@swm.pp.se> Date: 09/19/2013 11:08 AM (GMT-08:00) To: Paul Ferguson <fergdawgster@mykolab.com> Cc: NANOG <nanog@nanog.org> Subject: Re: iOS 7 update traffic
On Thu, 19 Sep 2013, Paul Ferguson wrote:
Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this...
The IOS7 upgrade is ~750 megabyte download for the phones/pods, and ~950 megabytes for ipad. There are quite a few devices out there times these amounts to download...
-- Mikael Abrahamsson email: swmike@swm.pp.se
Patch Tuesday is not 1gb per patch. On 9/19/13 11:51 AM, "Valdis.Kletnieks@vt.edu" <Valdis.Kletnieks@vt.edu> wrote:
On Thu, 19 Sep 2013 18:11:11 -0000, Warren Bailey said:
Why does apple feel it is okay to send every mobile device an update on a single day?
How is Patch Tuesday any different?
(merging 2 replies) On Thu, 19 Sep 2013 19:11:21 -0000, Warren Bailey said:
Patch Tuesday is not 1gb per patch.
It is those months a service pack comes out. On Thu, 19 Sep 2013 18:22:50 -0000, Warren Bailey said:
These people (Apple) represent themselves as smart guys, but their actions reflect otherwise.
No - their actions *are* those of smart guys. They knew damned well that they didn't need to add capacity, because the chances that they'll lose an iOS customer to Android over slow update speeds is infinitesimal. In other words, they successfully made the bandwidth consumption an externality. (What, like you don't look at *your* network and wonder "How can I stick somebody else with the cost of XYZ?". :)
Microsoft Windows 8.1 is due out in October.. don't be so sure :) -- Stephen On 19/09/2013 3:11 PM, Warren Bailey wrote:
Patch Tuesday is not 1gb per patch.
On 9/19/13 11:51 AM, "Valdis.Kletnieks@vt.edu" <Valdis.Kletnieks@vt.edu> wrote:
On Thu, 19 Sep 2013 18:11:11 -0000, Warren Bailey said:
Why does apple feel it is okay to send every mobile device an update on a single day?
How is Patch Tuesday any different?
The attitude in this email I have encountered elsewhere. Apple pays for bandwidth, customers pay for access. Not sure why their release strategy is so highly critiqued. Microsoft and others have their own strategies for incremental downloads, caching, etc.. Apple has theirs. Seems like most consumers want the update and are actively fetching it vs having older software live forever and not be updated. Overall I see this as a win. Jared Mauch
On Sep 19, 2013, at 2:11 PM, Warren Bailey <wbailey@satelliteintelligencegroup.com> wrote:
I don't see how operators could tolerate this, honestly. I can't think of a single provider who does not oversubscribe their access platform... Which leads me to this question :
Why does apple feel it is okay to send every mobile device an update on a single day?
+1 If you do not/cannot have an Akamai cache, connect to an IX that does, and make sure you've got the capacity. My own rule of thumb is have 2x the capacity of your average *peak* traffic on an IX. When big events happen, whether it is news, sporting or a major software update, that extra capacity will be sorely needed. At TorIX, most peers traffic jumped by the same percentage that others have bandied about on this thread. One peer jumped almost 100%, but they had the right port speed and thus no issues (at least on the Exchange). Compared to transit in Canada, IX peering is dirt cheap, and pays dividends. -- Stephen On 19/09/2013 3:07 PM, Jared Mauch wrote:
The attitude in this email I have encountered elsewhere. Apple pays for bandwidth, customers pay for access. Not sure why their release strategy is so highly critiqued. Microsoft and others have their own strategies for incremental downloads, caching, etc.. Apple has theirs.
Seems like most consumers want the update and are actively fetching it vs having older software live forever and not be updated. Overall I see this as a win.
Jared Mauch
On Sep 19, 2013, at 2:11 PM, Warren Bailey <wbailey@satelliteintelligencegroup.com> wrote:
I don't see how operators could tolerate this, honestly. I can't think of a single provider who does not oversubscribe their access platform... Which leads me to this question :
Why does apple feel it is okay to send every mobile device an update on a single day?
In my experience just having a Akamai cache wasn't enough to handle this. Our local cache was doing 15 out of 20gbps usage and seemed pegged at that. One of our customers had a local Akamai cache on there end crash and we were mostly filling a 10gbps pipe to a datacenter with limelight cdn's at it. This was just the CDN spike and not the one's over our peering and commodity traffic. Yes apple might not of forced people to upgrade exactly at the time they released but they can at least do an enforced role out like google does(on much smaller updates of course). Our core handled it all with not much more then amazement from our staff but a majority of customers pipes were pegged at whatever bandwidth they were paying for(I'm talking between 200mbps-5gbps with a few as big as 10gbps customer connections) --Nick On 9/19/13 4:18 PM, "Stephen Fulton" <sf@lists.esoteric.ca> wrote:
+1
If you do not/cannot have an Akamai cache, connect to an IX that does, and make sure you've got the capacity. My own rule of thumb is have 2x the capacity of your average *peak* traffic on an IX. When big events happen, whether it is news, sporting or a major software update, that extra capacity will be sorely needed.
At TorIX, most peers traffic jumped by the same percentage that others have bandied about on this thread. One peer jumped almost 100%, but they had the right port speed and thus no issues (at least on the Exchange).
Compared to transit in Canada, IX peering is dirt cheap, and pays dividends.
-- Stephen
On 19/09/2013 3:07 PM, Jared Mauch wrote:
The attitude in this email I have encountered elsewhere. Apple pays for bandwidth, customers pay for access. Not sure why their release strategy is so highly critiqued. Microsoft and others have their own strategies for incremental downloads, caching, etc.. Apple has theirs.
Seems like most consumers want the update and are actively fetching it vs having older software live forever and not be updated. Overall I see this as a win.
Jared Mauch
On Sep 19, 2013, at 2:11 PM, Warren Bailey <wbailey@satelliteintelligencegroup.com> wrote:
I don't see how operators could tolerate this, honestly. I can't think of a single provider who does not oversubscribe their access platform... Which leads me to this question :
Why does apple feel it is okay to send every mobile device an update on a single day?
Hi Jared.
The attitude in this email I have encountered elsewhere. Apple pays for bandwidth, customers pay for access. Not sure why their release strategy is so highly critiqued.
Because it impacts other, non-Apple customers. Or, it costs the ISP more (passed through to all customers) to add capacity to handle an infrequent peak load. Question/suggestion: Could Apple perhaps shift their release to a Saturday morning? I would think that this would go a long way to diluting the peak. John John Souvestre - New Orleans LA - (504) 454-0899
On Sep 19, 2013, at 4:36 PM, "John Souvestre" <johns@sstar.com> wrote:
Hi Jared.
The attitude in this email I have encountered elsewhere. Apple pays for bandwidth, customers pay for access. Not sure why their release strategy is so highly critiqued.
Because it impacts other, non-Apple customers. Or, it costs the ISP more (passed through to all customers) to add capacity to handle an infrequent peak load.
Question/suggestion: Could Apple perhaps shift their release to a Saturday morning? I would think that this would go a long way to diluting the peak.
John
John Souvestre - New Orleans LA - (504) 454-0899
I think there's a lot that could be done when looking at how to shift this. I've seen one other carrier privately talk to me about the impact and possible impacts to their network. Most of these are folks (along with warren) who are worried about their RF budgets and these event traffic, or even just the nightly traffic peaks. I have advised some in the past to put up caches, but the content owners also make it difficult to do this. Apple sets very short expire values, and you end up with lots of "bad" settings. Apple devices don't honor DHCP option 252 either. This means you're stuck with a transparent proxy, (lets just say squid) putting itself in all tcp/80 traffic, or worse with lots of settings like: reload-into-ims override-expire etc.. This can solve some problems for those who have a 20-50Mb/s link to the internet and 50-100 customers each getting 1Mb/s+ on their CPE. The results I've always seen are you need to find the strategic location to deploy these caches, capabilities or expand your network bandwidth, etc.. Based on all the recent people asking for a fast link in "X" location recently, I'm hoping there will be some better match-making happening soon. - Jared
On 2013-09-19, at 18:08, Jared Mauch <jared@puck.nether.net> wrote:
I think there's a lot that could be done when looking at how to shift this.
But likely not before the first iOS 7 patch release. http://appleinsider.com/articles/13/09/19/apples-control-center-used-to-bypa... Joe
In message <64245AC1-BC00-4928-B2F7-F259E8632655@puck.nether.net>, Jared Mauch writes:
On Sep 19, 2013, at 4:36 PM, "John Souvestre" <johns@sstar.com> wrote:
Hi Jared.
The attitude in this email I have encountered elsewhere. Apple pays for bandwidth, customers pay for access. Not sure why their release strategy is so highly critiqued.
Because it impacts other, non-Apple customers. Or, it costs the ISP more (passed through to all customers) to add capacity to handle an infrequent peak load.
Question/suggestion: Could Apple perhaps shift their release to a Saturday morning? I would think that this would go a long way to diluting the peak.
John
John Souvestre - New Orleans LA - (504) 454-0899
I think there's a lot that could be done when looking at how to shift this.
I've seen one other carrier privately talk to me about the impact and possible impacts to their network. Most of these are folks (along with warren) who are worried about their RF budgets and these event traffic, or even just the nightly traffic peaks.
I have advised some in the past to put up caches, but the content owners also make it difficult to do this. Apple sets very short expire values, and you end up with lots of "bad" settings. Apple devices don't honor DHCP option 252 either.
Oh you mean that option that never made it past a internet-draft that expired 13 years ago[1] and is in the private range[2] to boot. If you want proxy discovery to work on all devices complete the process of getting a code point allocated then get the OS vendors to query for it. 252 is fine for experimenting / proof of concept but it really is the wrong value for long term use. Mark [1] http://tools.ietf.org/html/draft-ietf-wrec-wpad-01 [2] http://www.iana.org/assignments/bootp-dhcp-parameters/bootp-dhcp-parameters....
This means you're stuck with a transparent proxy, (lets just say squid) putting itself in all tcp/80 traffic, or worse with lots of settings like: reload-into-ims override-expire etc..
This can solve some problems for those who have a 20-50Mb/s link to the internet and 50-100 customers each getting 1Mb/s+ on their CPE.
The results I've always seen are you need to find the strategic location to deploy these caches, capabilities or expand your network bandwidth, etc..
Based on all the recent people asking for a fast link in "X" location recently, I'm hoping there will be some better match-making happening soon.
- Jared
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
On Sep 19, 2013, at 7:13 PM, Mark Andrews <marka@isc.org> wrote:
Oh you mean that option that never made it past a internet-draft that expired 13 years ago[1] and is in the private range[2] to boot.
If you want proxy discovery to work on all devices complete the process of getting a code point allocated then get the OS vendors to query for it. 252 is fine for experimenting / proof of concept but it really is the wrong value for long term use.
Mark
[1] http://tools.ietf.org/html/draft-ietf-wrec-wpad-01 [2] http://www.iana.org/assignments/bootp-dhcp-parameters/bootp-dhcp-parameters....
Sure! I've found that Microsoft devices honor this option, but others do not. I would be in support of something similar to provide this support, but the part of my original reply you missed is that the content is deliberately not-cachable on the part of either the CDN or the originator. Microsoft patches are also not easily cacheable as well because they only request about 100Kb per request, so you get an awful lot of HTTP/206. They also make it easier to run local caches for an enterprise. The apple process requires the full patch to come down in one-shot and doesn't like being interrupted. It might be easier for Warren to ship each customer a 16GB USB with the whole set of images for each device type. Then again, they would have to know how to use them.... I know what to do, but my other family members, not so much... - Jared
In message <F11FF3CF-D363-4AF0-A030-B72CD68DD988@puck.nether.net>, Jared Mauch writes:
On Sep 19, 2013, at 7:13 PM, Mark Andrews <marka@isc.org> wrote:
Oh you mean that option that never made it past a internet-draft that expired 13 years ago[1] and is in the private range[2] to boot.
If you want proxy discovery to work on all devices complete the process of getting a code point allocated then get the OS vendors to query for it. 252 is fine for experimenting / proof of concept but it really is the wrong value for long term use.
Mark
[1] http://tools.ietf.org/html/draft-ietf-wrec-wpad-01 [2] http://www.iana.org/assignments/bootp-dhcp-parameters/bootp-dhcp-parameter s.xhtml
Sure!
I've found that Microsoft devices honor this option, but others do not.
I would be in support of something similar to provide this support, but the part of my original reply you missed is that the content is deliberately not-cachable on the part of either the CDN or the originator. Microsoft patches are also not easily cacheable as well because they only request about 100Kb per request, so you get an awful lot of HTTP/206.
They also make it easier to run local caches for an enterprise.
The apple process requires the full patch to come down in one-shot and doesn't like being interrupted.
It might be easier for Warren to ship each customer a 16GB USB with the whole set of images for each device type. Then again, they would have to know how to use them.... I know what to do, but my other family members, not so much...
- Jared
So you fix one part at a time. Each part is independently fixable. Each part helps by itself. Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
On Sep 19, 2013, at 14:11, Warren Bailey <wbailey@satelliteintelligencegroup.com> wrote:
I don't see how operators could tolerate this, honestly. I can't think of a single provider who does not oversubscribe their access platform... Which leads me to this question :
Over-subscription is a business decision that every network has to make, it's a fact of life for any operator. When you are oversubscribing, you also need to take into consideration on how much you're getting yourself overleveraged in such a configuration for sudden events like this. Oversubscription is a routine process in capacity planning, which is not something you just set it and forget it, then get pissed off at Apple or any other content provider during a sudden upsurge of traffic.
Why does apple feel it is okay to send every mobile device an update on a single day?
Your customers/consumers have purchased a product (internet access) from you based on what you've offered. Whether you've oversubscribed or undersubscribed your network in delivering that service is your problem as a network operator, and your problem to deal with. Why should Apple care? Let's turn this question around: why should your customers think it's ok for network operators to irresponsibly increase their profit margins through egregious over-subscriptions and idiotic capacity planning? How is it Apple, content provider or your consumer's problem that your network can't deliver the bits at times when it needs to? As a network providing service, it's your problem to deal with capacity issues -- you've sold a product with specified speed levels to your subscribers, and you've made a bet on oversubscribing. Suddenly the subscriber wants to use what he previously paid for and now it's a problem?
Never mind the fact that we are we ones on the last mile responsible for getting it to their customers, 1gb per sub is pretty serious.. Why are they not caching at their head ends, dslams, etc?
Apple delivers their content through Akamai. As a last mile provider responsible for delivering content to your customers, you should check: http://www.akamai.com/html/partners/network_program.html james
Why does apple feel it is okay to send every mobile device an update on a single day?
(a) That's why god invented the concept of CDNs....to take the stress of the more contended parts of an operators network. ;-) (b) Its not just Apple but any vendor (e.g. Microsloth).... their updates are released to the world at the same time. (c) Your user is paying you to push packets. If that's causing you a problem, you either need to review your commercial structure (i.e. charge people more) or your technical network design. Face the facts, what with everyone jumping on the "cloud" bandwagon, the future is only going to see you pushing more packets, not less ! So if you can't stand the heat, get out of the kitchen (or the xSP industry). ;-) No need for you to bash Apple in this instance.
* Ben (ben+nanog@list-subs.com) wrote:
No need for you to bash Apple in this instance.
What this conversation badly needs is a sub-thread about whatever happened to the technical solutions which would address this issue (eg: mbone). Of course, I know what happened and what the issues are there, but it'd be fun to watch people complain about how Apple should try and do the "right thing" and make multicast work for their updates, 'cause it's the "right" technical solution. Thanks, Stephen
Haven't updated my iPad yet but the iPhone update size was 1.12GB On Sep 19, 2013, at 2:05 PM, Mikael Abrahamsson wrote:
On Thu, 19 Sep 2013, Paul Ferguson wrote:
Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this...
The IOS7 upgrade is ~750 megabyte download for the phones/pods, and ~950 megabytes for ipad. There are quite a few devices out there times these amounts to download...
-- Mikael Abrahamsson email: swmike@swm.pp.se
My iPhone4 was about 600MB IIRC. My iPad mini was about that. I have about 7 iDevices between everyone in my immediate family. FWIW not a single one has actually received the notification yet. I've only manually done my 2 devices. I'm waiting to see how long it takes before I get the 'official' notification of an update on the others. On Thu, Sep 19, 2013 at 11:12 AM, TR Shaw <tshaw@oitc.com> wrote:
Haven't updated my iPad yet but the iPhone update size was 1.12GB
On Sep 19, 2013, at 2:05 PM, Mikael Abrahamsson wrote:
On Thu, 19 Sep 2013, Paul Ferguson wrote:
Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this...
The IOS7 upgrade is ~750 megabyte download for the phones/pods, and ~950 megabytes for ipad. There are quite a few devices out there times these amounts to download...
-- Mikael Abrahamsson email: swmike@swm.pp.se
Okay, that makes sense. Just wanted to ensure it wasn't something more sinister. Thanks, - ferg On 9/19/2013 11:05 AM, Mikael Abrahamsson wrote:
On Thu, 19 Sep 2013, Paul Ferguson wrote:
Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this...
The IOS7 upgrade is ~750 megabyte download for the phones/pods, and ~950 megabytes for ipad. There are quite a few devices out there times these amounts to download...
-- Paul Ferguson Vice President, Threat Intelligence Internet Identity, Tacoma, Washington USA IID --> "Connect and Collaborate" --> www.internetidentity.com
----- Original Message -----
From: "Mikael Abrahamsson" <swmike@swm.pp.se> To: "Paul Ferguson" <fergdawgster@mykolab.com>
On Thu, 19 Sep 2013, Paul Ferguson wrote:
Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this...
The IOS7 upgrade is ~750 megabyte download for the phones/pods, and ~950 megabytes for ipad. There are quite a few devices out there times these amounts to download...
It wasn't sinister, Ferg, it was *stupid*. *They announced the release date and time*. Everyone's phone has a "check for release now" button, and yet they believed that only *push notifying* phones in waves would be enough to prevent what happened. See also: screwed the pooch. This went out, what, last Weds and Thu? I had half a dozen people from all different environments ask me "why's the Internet broke today?" on release day. There was a consumer-visible impact from this absolutely asinine release engineering project on Apple's part. You don't announce the exact release time, and you don't even *make the release visible to all devices at the same time* since Twitter. This is apparently their first rodeo. We should take pains to make it their last. As an anti-Apple guy, I resent having my stuff screwed up because of it. Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA #natog +1 727 647 1274
It was released Thanks Darren http://www.mellowd.co.uk/ccie
Date: Thu, 19 Sep 2013 10:58:24 -0700 From: fergdawgster@mykolab.com To: nanog@nanog.org Subject: Re: iOS 7 update traffic
Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this...
Thanks in adavnce,
- ferg
On 9/19/2013 10:23 AM, Nick Olsen wrote:
We also saw a huge spike in traffic. Still pretty high today as well. We saw a ~60% above average hit yesterday, And we're at ~20-30% above average today as well. Being an android user, It didn't dawn on me until some of the IOS users in the office started jumping up and down about IOS7 Nick Olsen Network Operations (855) FLSPEED x106
---------------------------------------- From: "Justin M. Streiner" <streiner@cluebyfour.org> Sent: Wednesday, September 18, 2013 6:19 PM To: "NANOG" <nanog@nanog.org> Subject: Re: iOS 7 update traffic
On Wed, 18 Sep 2013, Tassos Chatzithomaoglou wrote:
We also noticed an interesting spike (+ ~40%), mostly in akamai. The same happened on previous iOS too.
I see it here, too. At its peak, our traffic levels were roughly double what we would see on a normal weekday.
jms
Zachary McGibbon wrote on 18/9/2013 20:38:
So iOS 7 just came out, here's the spike in our graphs going to our ISP here at McGill, anyone else noticing a big spike?
[image: internet-sw1 - Traffic - Te0/7 - To Internet1-srp (IR Canet) - TenGigabitEthernet0/7]
Zachary McGibbon
-- Paul Ferguson Vice President, Threat Intelligence Internet Identity, Tacoma, Washington USA IID --> "Connect and Collaborate" --> www.internetidentity.com
On 2013-09-19, at 13:58, Paul Ferguson <fergdawgster@mykolab.com> wrote:
Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this...
I think the inference is that iOS 7 caused the extra traffic by being available for download. There are just a lot of Apple devices, and they tend to get upgraded more promptly than other platforms (e.g. on release day). We saw a similar phenomenon tracking downloads of the root zone DNSSEC trust anchor from data.iana.org -- we now see three million downloads per month, and pretty much all of those are iOS devices (or other devices impersonating them, which seems unlikely). Joe
Apple pushed out a new software upgrade for their user interface...a pretty big upgrade, all the iphone users are downloading it congesting the network. Garrison Carr This email message is intended for the use of the person to whom it has been sent, and may contain information that is confidential or legally protected. If you are not the intended recipient or have received this message in error, you are not authorized to copy, distribute, or otherwise use this message or its attachments. Please notify the sender immediately by return e-mail and permanently delete this message and any attachments. NTT America makes no warranty that this email is error or virus free. Thank you . -----Original Message----- From: Paul Ferguson [mailto:fergdawgster@mykolab.com] Sent: Thursday, September 19, 2013 10:58 AM To: NANOG Subject: Re: iOS 7 update traffic Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this... Thanks in adavnce, - ferg On 9/19/2013 10:23 AM, Nick Olsen wrote:
We also saw a huge spike in traffic. Still pretty high today as well. We saw a ~60% above average hit yesterday, And we're at ~20-30% above average today as well. Being an android user, It didn't dawn on me until some of the IOS users in the office started jumping up and down about IOS7 Nick Olsen Network Operations (855) FLSPEED x106
---------------------------------------- From: "Justin M. Streiner" <streiner@cluebyfour.org> Sent: Wednesday, September 18, 2013 6:19 PM To: "NANOG" <nanog@nanog.org> Subject: Re: iOS 7 update traffic
On Wed, 18 Sep 2013, Tassos Chatzithomaoglou wrote:
We also noticed an interesting spike (+ ~40%), mostly in akamai. The same happened on previous iOS too.
I see it here, too. At its peak, our traffic levels were roughly double what we would see on a normal weekday.
jms
Zachary McGibbon wrote on 18/9/2013 20:38:
So iOS 7 just came out, here's the spike in our graphs going to our ISP here at McGill, anyone else noticing a big spike?
[image: internet-sw1 - Traffic - Te0/7 - To Internet1-srp (IR Canet) - TenGigabitEthernet0/7]
Zachary McGibbon
-- Paul Ferguson Vice President, Threat Intelligence Internet Identity, Tacoma, Washington USA IID --> "Connect and Collaborate" --> www.internetidentity.com
Major update & provides many of 5S functionality to the 5, 4S, & 4 On Sep 19, 2013, at 1:58 PM, Paul Ferguson wrote:
Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this...
Thanks in adavnce,
- ferg
On 9/19/2013 10:23 AM, Nick Olsen wrote:
We also saw a huge spike in traffic. Still pretty high today as well. We saw a ~60% above average hit yesterday, And we're at ~20-30% above average today as well. Being an android user, It didn't dawn on me until some of the IOS users in the office started jumping up and down about IOS7 Nick Olsen Network Operations (855) FLSPEED x106
---------------------------------------- From: "Justin M. Streiner" <streiner@cluebyfour.org> Sent: Wednesday, September 18, 2013 6:19 PM To: "NANOG" <nanog@nanog.org> Subject: Re: iOS 7 update traffic
On Wed, 18 Sep 2013, Tassos Chatzithomaoglou wrote:
We also noticed an interesting spike (+ ~40%), mostly in akamai. The same happened on previous iOS too.
I see it here, too. At its peak, our traffic levels were roughly double what we would see on a normal weekday.
jms
Zachary McGibbon wrote on 18/9/2013 20:38:
So iOS 7 just came out, here's the spike in our graphs going to our ISP here at McGill, anyone else noticing a big spike?
[image: internet-sw1 - Traffic - Te0/7 - To Internet1-srp (IR Canet) - TenGigabitEthernet0/7]
Zachary McGibbon
-- Paul Ferguson Vice President, Threat Intelligence Internet Identity, Tacoma, Washington USA IID --> "Connect and Collaborate" --> www.internetidentity.com
Composed on a virtual keyboard, please forgive typos. On Sep 19, 2013, at 13:58, Paul Ferguson <fergdawgster@mykolab.com> wrote:
Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this...
BING for "ios adoption rate" (one estimate is 29% in 16 hours), multiply by # of iThings, multiply by size of iOS, divide by # of seconds in estimate. As for why so many users upgrade so fast, that's a harder question. It could be iThing users are more willing to believe the fruit company's advertising (hype) . Could be that the device tells them to upgrade so they do. It is also at least partially due to the fact all iThings are upgradable (within a certain age horizon). Hope that gives you something to chew on, even if it doesn't answer the question. -- TTFN, patrick
On 9/19/2013 10:23 AM, Nick Olsen wrote:
We also saw a huge spike in traffic. Still pretty high today as well. We saw a ~60% above average hit yesterday, And we're at ~20-30% above average today as well. Being an android user, It didn't dawn on me until some of the IOS users in the office started jumping up and down about IOS7 Nick Olsen Network Operations (855) FLSPEED x106
---------------------------------------- From: "Justin M. Streiner" <streiner@cluebyfour.org> Sent: Wednesday, September 18, 2013 6:19 PM To: "NANOG" <nanog@nanog.org> Subject: Re: iOS 7 update traffic
On Wed, 18 Sep 2013, Tassos Chatzithomaoglou wrote:
We also noticed an interesting spike (+ ~40%), mostly in akamai. The same happened on previous iOS too.
I see it here, too. At its peak, our traffic levels were roughly double what we would see on a normal weekday.
jms
Zachary McGibbon wrote on 18/9/2013 20:38:
So iOS 7 just came out, here's the spike in our graphs going to our ISP here at McGill, anyone else noticing a big spike?
[image: internet-sw1 - Traffic - Te0/7 - To Internet1-srp (IR Canet) - TenGigabitEthernet0/7]
Zachary McGibbon
-- Paul Ferguson Vice President, Threat Intelligence Internet Identity, Tacoma, Washington USA IID --> "Connect and Collaborate" --> www.internetidentity.com
On 9/19/13 10:58 AM, Paul Ferguson wrote:
Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this...
iOS 7 itself was implemented. ~Seth
Tens of millions of devices multiplied times a fairly large download = lots of bandwidth. It has an appreciable affect on the worldwide Internet. I would love to see some aggregate statistics. With most phones the carrier takes care of doing phone software updates and rollouts over a period of time since they all have customized versions of Android/Windows Phone/etc. Apple controls their phone software so they just hit the switch at a certain data/time and let as many people update as their servers can handle. Not to mention all the IPads which they chose to update at the same time. Last time around we saw a sustained increase in traffic for about a week after the release date. -Phil On 9/19/13 1:58 PM, "Paul Ferguson" <fergdawgster@mykolab.com> wrote:
Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this...
Thanks in adavnce,
- ferg
On 9/19/2013 10:23 AM, Nick Olsen wrote:
We also saw a huge spike in traffic. Still pretty high today as well. We saw a ~60% above average hit yesterday, And we're at ~20-30% above average today as well. Being an android user, It didn't dawn on me until some of the IOS users in the office started jumping up and down about IOS7 Nick Olsen Network Operations (855) FLSPEED x106
---------------------------------------- From: "Justin M. Streiner" <streiner@cluebyfour.org> Sent: Wednesday, September 18, 2013 6:19 PM To: "NANOG" <nanog@nanog.org> Subject: Re: iOS 7 update traffic
On Wed, 18 Sep 2013, Tassos Chatzithomaoglou wrote:
We also noticed an interesting spike (+ ~40%), mostly in akamai. The same happened on previous iOS too.
I see it here, too. At its peak, our traffic levels were roughly double what we would see on a normal weekday.
jms
Zachary McGibbon wrote on 18/9/2013 20:38:
So iOS 7 just came out, here's the spike in our graphs going to our ISP here at McGill, anyone else noticing a big spike?
[image: internet-sw1 - Traffic - Te0/7 - To Internet1-srp (IR Canet) - TenGigabitEthernet0/7]
Zachary McGibbon
-- Paul Ferguson Vice President, Threat Intelligence Internet Identity, Tacoma, Washington USA IID --> "Connect and Collaborate" --> www.internetidentity.com
Can someone please explain to a non-Apple person what the hell happened that started generating so much traffic? Perhaps I missed it in this thread, but I would be curious to know what iOS 7 implemented that caused this...
all the borders and highlights from the discarded skeuomorphisms cloged up the intertubes bigtime randy
participants (54)
-
Ben -
Blake Dunlap -
Brandon Galbraith -
Bryan Irvine -
Carsten Bormann -
Colin Alston -
Cutler James R -
Darren O'Connor -
Dorian Kim -
Doug McIntyre -
Fred Reimer -
Gabriel Blanchard -
Garrison Carr -
Glen Kent -
james＠towardex.com -
Jared Mauch -
Jay Ashworth -
Jeff Kell -
Jeroen Massar -
Jeroen van Aart -
Jethro R Binks -
Joe Abley -
Joe Greco -
joel jaeggli -
John Smith -
John Souvestre -
Jon Sands -
Justin M. Streiner -
Keith Medcalf -
Leo Bicknell -
Mark Andrews -
Masataka Ohta -
Matt Palmer -
Mikael Abrahamsson -
Mike A -
nanog＠namor.ca -
Neil Harris -
Nick Wolff -
Octavio Alvarez -
Patrick W. Gilmore -
Paul Ferguson -
Phil Bedard -
Ralph J.Mayer -
Randy Bush -
Ryan Brooks -
Ryan Harden -
Seth Mattinen -
Simon Leinen -
Stephen Frost -
Stephen Fulton -
Tom Taylor -
TR Shaw -
Valdis.Kletnieks＠vt.edu -
Warren Bailey