Netflix stuffing data on pipe
Has anyone else observed Netflix sessions attempting to come into customer CPE devices at well in excess of the customers throttled plan? I'm not talking error retries on the line. I'm talking like two to three times in excess of what the customers CPE device can handle. I'm observing massive buffer overruns in some of our switches that appear to be directly related to this. And I can't figure out what possible good purpose Netflix would have for attempting to do this. Curious if anyone else has seen it?
Adaptive bandwidth detection. On Dec 29, 2015 8:59 PM, "Matt Hoppes" <mhoppes@indigowireless.com> wrote:
Has anyone else observed Netflix sessions attempting to come into customer CPE devices at well in excess of the customers throttled plan?
I'm not talking error retries on the line. I'm talking like two to three times in excess of what the customers CPE device can handle.
I'm observing massive buffer overruns in some of our switches that appear to be directly related to this. And I can't figure out what possible good purpose Netflix would have for attempting to do this.
Curious if anyone else has seen it?
So they are trying to stuff every last bit as an end device modulates up and down? Or are you saying that's how they determine if they can scale up the resolution "because there is more throughout available now".
On Dec 29, 2015, at 22:10, Josh Reynolds <josh@kyneticwifi.com> wrote:
Adaptive bandwidth detection.
On Dec 29, 2015 8:59 PM, "Matt Hoppes" <mhoppes@indigowireless.com> wrote: Has anyone else observed Netflix sessions attempting to come into customer CPE devices at well in excess of the customers throttled plan?
I'm not talking error retries on the line. I'm talking like two to three times in excess of what the customers CPE device can handle.
I'm observing massive buffer overruns in some of our switches that appear to be directly related to this. And I can't figure out what possible good purpose Netflix would have for attempting to do this.
Curious if anyone else has seen it?
The second part. Fixed wireless is not even on their radar. On Dec 29, 2015 9:16 PM, "Matt Hoppes" <mhoppes@indigowireless.com> wrote:
So they are trying to stuff every last bit as an end device modulates up and down?
Or are you saying that's how they determine if they can scale up the resolution "because there is more throughout available now".
On Dec 29, 2015, at 22:10, Josh Reynolds <josh@kyneticwifi.com> wrote:
Adaptive bandwidth detection. On Dec 29, 2015 8:59 PM, "Matt Hoppes" <mhoppes@indigowireless.com> wrote:
Has anyone else observed Netflix sessions attempting to come into customer CPE devices at well in excess of the customers throttled plan?
I'm not talking error retries on the line. I'm talking like two to three times in excess of what the customers CPE device can handle.
I'm observing massive buffer overruns in some of our switches that appear to be directly related to this. And I can't figure out what possible good purpose Netflix would have for attempting to do this.
Curious if anyone else has seen it?
On Tue 2015-Dec-29 21:17:51 -0600, Josh Reynolds <josh@kyneticwifi.com> wrote:
The second part. Fixed wireless is not even on their radar. On Dec 29, 2015 9:16 PM, "Matt Hoppes" <mhoppes@indigowireless.com> wrote:
So they are trying to stuff every last bit as an end device modulates up and down?
Or are you saying that's how they determine if they can scale up the resolution "because there is more throughout available now".
On Dec 29, 2015, at 22:10, Josh Reynolds <josh@kyneticwifi.com> wrote:
Adaptive bandwidth detection. On Dec 29, 2015 8:59 PM, "Matt Hoppes" <mhoppes@indigowireless.com> wrote:
Has anyone else observed Netflix sessions attempting to come into customer CPE devices at well in excess of the customers throttled plan?
I'm not talking error retries on the line. I'm talking like two to three times in excess of what the customers CPE device can handle.
I'm observing massive buffer overruns in some of our switches that appear to be directly related to this. And I can't figure out what possible good purpose Netflix would have for attempting to do this.
Pardon my ignorance of WISP-specific bits here, but how are they supposed to know to back off on their bitrate ramp-up if you keep buffering rather than dropping packets when the TX rate exceeds the customer's service rate? Or what am I missing?
Curious if anyone else has seen it?
-- Hugo hugo@slabnet.com: email, xmpp/jabber PGP fingerprint (B178313E): CF18 15FA 9FE4 0CD1 2319 1D77 9AB1 0FFD B178 313E (also on Signal)
It's a long and ugly story... 1Gbps FD feeds -> switch -> 100Mbps FD radio port -> fluctuating PHY rate Half Duplex wireless link/CPE (shaped here). Netflix is microbusting, and its really nasty on his kind of network, especially with the shaping being toward the end of his network. On Dec 30, 2015 1:59 AM, "Hugo Slabbert" <hugo@slabnet.com> wrote:
On Tue 2015-Dec-29 21:17:51 -0600, Josh Reynolds <josh@kyneticwifi.com> wrote:
On Dec 29, 2015 9:16 PM, "Matt Hoppes" <mhoppes@indigowireless.com> wrote:
So they are trying to stuff every last bit as an end device modulates up
and down?
Or are you saying that's how they determine if they can scale up the resolution "because there is more throughout available now".
On Dec 29, 2015, at 22:10, Josh Reynolds <josh@kyneticwifi.com> wrote:
Adaptive bandwidth detection. On Dec 29, 2015 8:59 PM, "Matt Hoppes" <mhoppes@indigowireless.com> wrote:
Has anyone else observed Netflix sessions attempting to come into
customer CPE devices at well in excess of the customers throttled plan?
I'm not talking error retries on the line. I'm talking like two to three times in excess of what the customers CPE device can handle.
I'm observing massive buffer overruns in some of our switches that appear to be directly related to this. And I can't figure out what possible good purpose Netflix would have for attempting to do this.
Pardon my ignorance of WISP-specific bits here, but how are they supposed to know to back off on their bitrate ramp-up if you keep buffering rather
The second part. Fixed wireless is not even on their radar. than dropping packets when the TX rate exceeds the customer's service rate? Or what am I missing?
Curious if anyone else has seen it?
-- Hugo
hugo@slabnet.com: email, xmpp/jabber PGP fingerprint (B178313E): CF18 15FA 9FE4 0CD1 2319 1D77 9AB1 0FFD B178 313E
(also on Signal)
I'm not buffering. Switches have packet buffers. I'm seeing switch buffers getting overrun by what appears to be Netflix traffic coming in at rates faster than the subscribers throttled speeds.
On Dec 30, 2015, at 02:59, Hugo Slabbert <hugo@slabnet.com> wrote:
On Tue 2015-Dec-29 21:17:51 -0600, Josh Reynolds <josh@kyneticwifi.com> wrote:
The second part. Fixed wireless is not even on their radar.
On Dec 29, 2015 9:16 PM, "Matt Hoppes" <mhoppes@indigowireless.com> wrote:
So they are trying to stuff every last bit as an end device modulates up and down?
Or are you saying that's how they determine if they can scale up the resolution "because there is more throughout available now".
On Dec 29, 2015, at 22:10, Josh Reynolds <josh@kyneticwifi.com> wrote:
Adaptive bandwidth detection.
On Dec 29, 2015 8:59 PM, "Matt Hoppes" <mhoppes@indigowireless.com> wrote:
Has anyone else observed Netflix sessions attempting to come into customer CPE devices at well in excess of the customers throttled plan?
I'm not talking error retries on the line. I'm talking like two to three times in excess of what the customers CPE device can handle.
I'm observing massive buffer overruns in some of our switches that appear to be directly related to this. And I can't figure out what possible good purpose Netflix would have for attempting to do this.
Pardon my ignorance of WISP-specific bits here, but how are they supposed to know to back off on their bitrate ramp-up if you keep buffering rather than dropping packets when the TX rate exceeds the customer's service rate? Or what am I missing?
Curious if anyone else has seen it?
-- Hugo
hugo@slabnet.com: email, xmpp/jabber PGP fingerprint (B178313E): CF18 15FA 9FE4 0CD1 2319 1D77 9AB1 0FFD B178 313E
(also on Signal)
On 30 Dec 2015, at 19:42, Matt Hoppes wrote:
I'm seeing switch buffers getting overrun by what appears to be Netflix traffic coming in at rates faster than the subscribers throttled speeds.
By what mechanism is the throttling accomplished? QoS on routers, or some kind of middlebox, or . . . ? ----------------------------------- Roland Dobbins <rdobbins@arbor.net>
On Wed, 30 Dec 2015, Matt Hoppes wrote:
I'm not buffering. Switches have packet buffers. I'm seeing switch buffers getting overrun by what appears to be Netflix traffic coming in at rates faster than the subscribers throttled speeds.
How big are your buffers (preferrably answer would be in milliseconds)? What access speeds are you providing? It's standard behavior for network traffic to sometimes be at higher speeds than the customer access speeds, the sender only knows about congestion if there is an increase in RTT or if there is packet loss (or in case of ECN, EC=1 flag), and the only way to find out is to probe (=run faster than customer access speed). -- Mikael Abrahamsson email: swmike@swm.pp.se
I believe others have observed a similar situation with at least one other CDN and the situation continued solid for hours, not just occasional capacity detection. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "Josh Reynolds" <josh@kyneticwifi.com> To: "Matt Hoppes" <mhoppes@indigowireless.com> Cc: "NANOG" <nanog@nanog.org> Sent: Tuesday, December 29, 2015 9:10:51 PM Subject: Re: Netflix stuffing data on pipe Adaptive bandwidth detection. On Dec 29, 2015 8:59 PM, "Matt Hoppes" <mhoppes@indigowireless.com> wrote:
Has anyone else observed Netflix sessions attempting to come into customer CPE devices at well in excess of the customers throttled plan?
I'm not talking error retries on the line. I'm talking like two to three times in excess of what the customers CPE device can handle.
I'm observing massive buffer overruns in some of our switches that appear to be directly related to this. And I can't figure out what possible good purpose Netflix would have for attempting to do this.
Curious if anyone else has seen it?
On Tuesday, December 29, 2015, Josh Reynolds <josh@kyneticwifi.com> wrote:
Adaptive bandwidth detection.
Yes, ABR video attempts to fill the entire channel This has been problematic as peak edge speeds have increased and pushed the statistical multplexing logic / plans. There is also buffer bloat issues that exacerbate the problem by allowing elephant flows to be too greedy at the expsense of others on the access segment
On Dec 29, 2015 8:59 PM, "Matt Hoppes" <mhoppes@indigowireless.com <javascript:;>> wrote:
Has anyone else observed Netflix sessions attempting to come into customer CPE devices at well in excess of the customers throttled plan?
I'm not talking error retries on the line. I'm talking like two to three times in excess of what the customers CPE device can handle.
I'm observing massive buffer overruns in some of our switches that appear to be directly related to this. And I can't figure out what possible good purpose Netflix would have for attempting to do this.
Curious if anyone else has seen it?
It is actually buffer-based, as it picks the video rate as a function of the current buffer occupancy. See here http://yuba.stanford.edu/~nickm/papers/sigcomm2014-video.pdf -- evelio On Tue, Dec 29, 2015 at 6:56 PM, Matt Hoppes <mhoppes@indigowireless.com> wrote:
Has anyone else observed Netflix sessions attempting to come into customer CPE devices at well in excess of the customers throttled plan?
I'm not talking error retries on the line. I'm talking like two to three times in excess of what the customers CPE device can handle.
I'm observing massive buffer overruns in some of our switches that appear to be directly related to this. And I can't figure out what possible good purpose Netflix would have for attempting to do this.
Curious if anyone else has seen it?
Netflix is streaming video. It will try to do the best data rate it can. If the connection can handle 4 megs a second it is going to try and do 4 megs a second. If the network can’t handle it then Netflix will back off and adapt to try and fit. Keep in mind, at least last I knew, a full HD stream was somewhere around 5 megs a sec. If the customer has a 4 meg plan it will try and fill up that 4 megs unless the algorithm backs off and steps it down. ISPs who run into this on lower packages need to implement QOS at the customer level to deal with streaming. This can be done several ways. This is one reason an endpoint the ISP controls is a huge asset, especially if it does QOS. Justin Wilson j2sw@mtin.net --- http://www.mtin.net Owner/CEO xISP Solutions- Consulting – Data Centers - Bandwidth http://www.midwest-ix.com COO/Chairman
On Dec 31, 2015, at 1:39 PM, Evelio Vila <evelio@thousandeyes.com> wrote:
It is actually buffer-based, as it picks the video rate as a function of the current buffer occupancy.
See here http://yuba.stanford.edu/~nickm/papers/sigcomm2014-video.pdf
-- evelio
On Tue, Dec 29, 2015 at 6:56 PM, Matt Hoppes <mhoppes@indigowireless.com> wrote:
Has anyone else observed Netflix sessions attempting to come into customer CPE devices at well in excess of the customers throttled plan?
I'm not talking error retries on the line. I'm talking like two to three times in excess of what the customers CPE device can handle.
I'm observing massive buffer overruns in some of our switches that appear to be directly related to this. And I can't figure out what possible good purpose Netflix would have for attempting to do this.
Curious if anyone else has seen it?
As I understand it, the problem being discussed is an oscillation that is created when the reaction occurs faster than the feedback resulting in a series of dynamically increasing overcompensations. Owen
On Jan 3, 2016, at 21:26 , Justin Wilson <lists@mtin.net> wrote:
Netflix is streaming video. It will try to do the best data rate it can. If the connection can handle 4 megs a second it is going to try and do 4 megs a second. If the network can’t handle it then Netflix will back off and adapt to try and fit.
Keep in mind, at least last I knew, a full HD stream was somewhere around 5 megs a sec. If the customer has a 4 meg plan it will try and fill up that 4 megs unless the algorithm backs off and steps it down. ISPs who run into this on lower packages need to implement QOS at the customer level to deal with streaming. This can be done several ways. This is one reason an endpoint the ISP controls is a huge asset, especially if it does QOS.
Justin Wilson j2sw@mtin.net
--- http://www.mtin.net Owner/CEO xISP Solutions- Consulting – Data Centers - Bandwidth
http://www.midwest-ix.com COO/Chairman
On Dec 31, 2015, at 1:39 PM, Evelio Vila <evelio@thousandeyes.com> wrote:
It is actually buffer-based, as it picks the video rate as a function of the current buffer occupancy.
See here http://yuba.stanford.edu/~nickm/papers/sigcomm2014-video.pdf
-- evelio
On Tue, Dec 29, 2015 at 6:56 PM, Matt Hoppes <mhoppes@indigowireless.com> wrote:
Has anyone else observed Netflix sessions attempting to come into customer CPE devices at well in excess of the customers throttled plan?
I'm not talking error retries on the line. I'm talking like two to three times in excess of what the customers CPE device can handle.
I'm observing massive buffer overruns in some of our switches that appear to be directly related to this. And I can't figure out what possible good purpose Netflix would have for attempting to do this.
Curious if anyone else has seen it?
Very succiently put, Owen! I concur. Is anything the ISP could avoid to alleviate this occurrence, or is it entirely a 'server-side' issue to resolve? Pete
On 4/01/2016, at 8:42 pm, Owen DeLong <owen@delong.com> wrote:
As I understand it, the problem being discussed is an oscillation that is created when the reaction occurs faster than the feedback resulting in a series of dynamically increasing overcompensations.
Owen
I haven't done packet dumps to verify the behavior (too busy catching up on holiday email) but I can't help but wonder if IW10 (on by default in FreeBSD 10 which I believe might be what Netflix has underneath) is causing this problem, and that maybe a more gentle CWND ramp-up (or otherwise tweaking the slow start behavior) for prefixes that are known to be in networks with weak hardware might be a good choice. Of course this would be a change on Netflix's end... as for things the ISP could do to alleviate the problem the answer is always "sure, but it'll cost ya". -r
On Jan 4, 2016, at 3:11 AM, Pete Mundy <pete@fiberphone.co.nz> wrote:
Very succiently put, Owen!
I concur.
Is anything the ISP could avoid to alleviate this occurrence, or is it entirely a 'server-side' issue to resolve?
Pete
On 4/01/2016, at 8:42 pm, Owen DeLong <owen@delong.com> wrote:
As I understand it, the problem being discussed is an oscillation that is created when the reaction occurs faster than the feedback resulting in a series of dynamically increasing overcompensations.
Owen
The most obvious things would be to make feedback faster… Implement congestion controls further up stream with reduced buffering throughout the network, selective technologies like WRED, etc. As RS said, sure, but all come at a cost either in performance, equipment, support, or some combination thereof. Owen
On Jan 4, 2016, at 00:11 , Pete Mundy <pete@fiberphone.co.nz> wrote:
Very succiently put, Owen!
I concur.
Is anything the ISP could avoid to alleviate this occurrence, or is it entirely a 'server-side' issue to resolve?
Pete
On 4/01/2016, at 8:42 pm, Owen DeLong <owen@delong.com> wrote:
As I understand it, the problem being discussed is an oscillation that is created when the reaction occurs faster than the feedback resulting in a series of dynamically increasing overcompensations.
Owen
participants (12)
-
Ca By -
Evelio Vila -
Hugo Slabbert -
Josh Reynolds -
Justin Wilson -
Matt Hoppes -
Mikael Abrahamsson -
Mike Hammett -
Owen DeLong -
Pete Mundy -
Rob Seastrom -
Roland Dobbins