Bufferbloat related censorship at Virgin Media
I have put this on a blog post, and my g+ also, here, and submitted the story to slashdot and reddit. How I spend my sunday afternoons these days! The linky version: http://the-edge.blogspot.com/2015/03/virgin-media-fixing-epidemic-of.html or g+: https://plus.google.com/u/0/107942175615993706558/posts/E1yMgbWW81C --snip snip-- To whom it may concern at Virgin Media: My IP address is apparently now banned from accessing your site at all, for "advertising", on this thread: http://community.virginmedia.com/t5/Up-to-152Mb/Bufferbloat-High-Latency-amp... Believe me, I understand the degree to which advertising pollutes the internet. And certainly, given the brevity of my post, you could assume that I was just some random guy, selling snake oil. Nothing could be further from the truth. Admittedly, it was a short message, it was kind of late, and I was in a hurry, being that I have so many other networks to help fix. To clarify matters: I am the co-founder of the bufferbloat project, and I like to think, a world-wide acknowledged expert on the topic on this thread. In particular, I worked pretty hard on part of the DOCSIS 3.1 standard, which was ratified years ago, and has a specific section on it regarding technologies that can help fix *half* your bufferbloat problem. http://www.cablelabs.com/how-docsis-3-1-reduces-latency-with-active-queue-ma... I admit to some frustration as to how long it is taking DOCSIS 3.1 to roll out. The cablelabs study that led up to the AQM component in the 3.1 standard - in which I participated and am cited in, is here: http://www.cablelabs.com/wp-content/uploads/2013/11/Active_Queue_Management_... And while I continue to favor fq_codel as the best solution for low and medium bandwidths - I have no problem with you somehow, soon, getting DOCSIS-pie out the door. If you continue to exist in denial of what your own R&D department for your own industry is saying, ghu help you! After giving this talk at uknof, the premier conference for network operators in the UK: https://plus.google.com/u/0/107942175615993706558/posts/immF8Pkj19C *over two years ago*, I met with 6+ technical members of Virgin Media's staff, who all agreed they had a problem, understood what it was, and grokked the various means to fix it. Judging from the enthusiasm in the room, I figured you'd be rolling out fixes by now, but was wrong. A rather human readable explanation of what has gone into the pending 3.1 standard is in the IETF internet draft here: https://datatracker.ietf.org/doc/draft-white-aqm-docsis-pie/ Sadly, just DOCSIS-pie rolling out on the modems is not enough - you have to somehow, yourselves, fix the dramatic overbuffering on the CMTS side, as shown here: http://burntchrome.blogspot.com/2014/05/disabling-shaping-in-one-direction-w... These downlink problems have been discussed thoroughly on the bufferbloat.net bloat and the ietf aqm mailing lists, and rather than point at direct links I would encourage more people to join the discussions there, and browse the archives. As I have seen no visible progress on the CMTS front yet... The best way to fix bufferbloat for your suffering customers *now*, is to help them - and your customer service departments - recognise the problem when it occurs and propose sane ways to fix it with stuff available off the shelf which includes the free firmware upgrades distributed by openwrt, or nearly any linux derived product and by the products available downstream from those. I have no financial interest in *free firmware*. I'm just trying to fix bufferbloat on a billion+ devices and nearly every network in the world as fast as humanly possible. Furthermore, me and a whole bunch of Internet luminaries gave the theory and code away for *free* also, in the hope that by doing so that might more quickly get the megacorps of the world to adopt them and make the quality of experience of internet access for billions of users of the world vastly better. Fixing bufferbloat was a 50 year old network research problem, now solved, with great joy, thoroughly, by some of the best minds in the business, and the answers are now so simple as to fit into a few hundred lines of code, easy to configure for end-users and easily embeddible in your own devices and networks if only you would get on the stick about it. We have provided the code, are in the standardization process, and provided free tools to diagnose and fix your epidemic bufferbloat accurately on every kind of device you have. Two examples of fixing bufferbloat on cablemodems: http://burntchrome.blogspot.com/2014/05/fixing-bufferbloat-on-comcasts-blast... http://snapon.lab.bufferbloat.net/~cero2/jimreisert/results.html And the *free* tool designed not only to accurately measure bufferbloat, but one that you can setup internally to test your networks and devices for it privately and quietly and then fix them with, is here: https://github.com/tohojo/netperf-wrapper So, now, a rant: Now, if me pointing a customer of yours that has correctly identified the root cause of his own problems, at the solutions both available now, and pending, is considered "advertising", then there really is an orwellian mixup between the definition of that word, and the truth, on your side of the water. Please, unblock my dtaht account and unblock my IP, and allow in better information about how customers of yours can solve the incredibly serious, and incredibly epidemic problem of bufferbloat. ... A problem that is now easy to solve with cheap gear now all over the market so that all your customers suffering can fix it for themselves if they so choose. And: I would like a public apology for blocking me, and a clear statement from Virgin, as to how, when, and where, they will begin to roll out their own fixes to bufferbloat across their subscriber base. And perhaps, you could publish some guidelines - like accurate up/download settings to use - to help your customers fix your problems for themselves. Sincerely, Dave Taht Co-founder, bufferbloat.net -- Dave Täht Let's make wifi fast, less jittery and reliable again! https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb
Dave, I appreciate all your work on buffer bloat. It looks like you have done quite a lot of selfless contribution. However, I don't think you're effectively communicating with the people who can change things. After I read what you said, here is what I would have heard as a service provider: "I am the smartest person in the room. You better listen to me, because if you don't there will be trouble. But you probably won't because you're too stupid. Your customers suffer because you are idiots. Listen to me! This issue is too important for me to be polite, or even coherent. If you can't figure out what I'm saying, do some research and figure it out! Plus, apologize to me! I demand it!" Bees, honey, vinegar, etc. -mel beckman
On Mar 1, 2015, at 3:29 PM, "Dave Taht" <dave.taht@gmail.com> wrote:
I have put this on a blog post, and my g+ also, here, and submitted the story to slashdot and reddit. How I spend my sunday afternoons these days!
The linky version: http://the-edge.blogspot.com/2015/03/virgin-media-fixing-epidemic-of.html
or g+: https://plus.google.com/u/0/107942175615993706558/posts/E1yMgbWW81C
--snip snip--
To whom it may concern at Virgin Media:
My IP address is apparently now banned from accessing your site at all, for "advertising", on this thread:
[Unnecessary repetition removed]
On Sun, Mar 1, 2015 at 3:47 PM, Mel Beckman <mel@beckman.org> wrote:
Dave,
I appreciate all your work on buffer bloat. It looks like you have done quite a lot of selfless contribution. However, I don't think you're effectively communicating with the people who can change things.
After I read what you said, here is what I would have heard as a service provider:
"I am the smartest person in the room.
I am pretty close to being the smartest person in the room. That said, people like van jacobson, eric dumazet, tom herbert, jim gettys, eric raymond, vint cerf, dave reed, fred baker, and many, many others, smarter than me, have also been banging this drum, politely, and rationally, to not much effect, for 4+ years now. google for any of those names and the word "bufferbloat".
You better listen to me, because if you don't there will be trouble. But you probably won't because you're too stupid. Your customers suffer because you are idiots. Listen to me! This issue is too important for me to be polite, or even coherent. If you can't figure out what I'm saying, do some research and figure it out! Plus, apologize to me! I demand it!"
Oh, banning my ip for *3 links to sane benchmarks and fixes*, realllllly pushed me over the edge.
Bees, honey, vinegar, etc.
I have been polite, constructive, and helpful, for four+ years. I have worked both in the background and foreground with many companies, to start hopefully, getting bufferbloat fixed across the entire edge of the internet. It hasn't worked fast enough for my liking, and the last batch of new products that claimed to fix it, didn't, and the market is now rife with genuine lies as to whether they did or not. So, this morning, I tried this. Sorry for the noise on these lists. Honestly! I totally agree with your assessment of my tone, btw! but I would rather like the cable industry in particular, to come clean, with schedules for deployable fixes. I am off to go fix wifi next, and I do hope that 2+ billion people in the world - if not the isps, maybe - would like wifi to get better also, and indeed, I spent the weekend constructively starting to implement some of the fixes I outlined at the last 802.11 meeting I attended. That part, on fixing wifi bufferbloat - a much harder problem than edge bufferbloat - , is a lot of fun! For some info on what we plan to do there, see: http://snapon.lab.bufferbloat.net/~d/ieee802.11-sept-17-2014/11-14-1265-00-0... So I took a break from that, reared back, and got some stuff off my chest. -- Dave Täht Let's make wifi fast, less jittery and reliable again! https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb
Well, with luck probably it will just bounce off their corporate hull and drift into the Kuiper belt. Say hi to Sugar ;) -mel
On Mar 1, 2015, at 4:01 PM, "Dave Taht" <dave.taht@gmail.com> wrote:
On Sun, Mar 1, 2015 at 3:47 PM, Mel Beckman <mel@beckman.org> wrote: Dave,
I appreciate all your work on buffer bloat. It looks like you have done quite a lot of selfless contribution. However, I don't think you're effectively communicating with the people who can change things.
After I read what you said, here is what I would have heard as a service provider:
"I am the smartest person in the room.
I am pretty close to being the smartest person in the room. That said, people like van jacobson, eric dumazet, tom herbert, jim gettys, eric raymond, vint cerf, dave reed, fred baker, and many, many others, smarter than me, have also been banging this drum, politely, and rationally, to not much effect, for 4+ years now.
google for any of those names and the word "bufferbloat".
You better listen to me, because if you don't there will be trouble. But you probably won't because you're too stupid. Your customers suffer because you are idiots. Listen to me! This issue is too important for me to be polite, or even coherent. If you can't figure out what I'm saying, do some research and figure it out! Plus, apologize to me! I demand it!"
Oh, banning my ip for *3 links to sane benchmarks and fixes*, realllllly pushed me over the edge.
Bees, honey, vinegar, etc.
I have been polite, constructive, and helpful, for four+ years. I have worked both in the background and foreground with many companies, to start hopefully, getting bufferbloat fixed across the entire edge of the internet. It hasn't worked fast enough for my liking, and the last batch of new products that claimed to fix it, didn't, and the market is now rife with genuine lies as to whether they did or not.
So, this morning, I tried this. Sorry for the noise on these lists. Honestly! I totally agree with your assessment of my tone, btw! but I would rather like the cable industry in particular, to come clean, with schedules for deployable fixes.
I am off to go fix wifi next, and I do hope that 2+ billion people in the world - if not the isps, maybe - would like wifi to get better also, and indeed, I spent the weekend constructively starting to implement some of the fixes I outlined at the last 802.11 meeting I attended. That part, on fixing wifi bufferbloat - a much harder problem than edge bufferbloat - , is a lot of fun! For some info on what we plan to do there, see:
http://snapon.lab.bufferbloat.net/~d/ieee802.11-sept-17-2014/11-14-1265-00-0...
So I took a break from that, reared back, and got some stuff off my chest.
-- Dave Täht Let's make wifi fast, less jittery and reliable again!
https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb
On 3/1/2015 5:28 PM, Dave Taht wrote:
My IP address is apparently now banned from accessing your site at all, for "advertising", on this thread:
http://community.virginmedia.com/t5/Up-to-152Mb/Bufferbloat-High-Latency-amp...
I don't see how codel is related to the customer complaint from their perspective. The problem appears to be high latency on downstream with little to no upstream. I'd probably call it off-topic advertising. The only thing that seems to relate to codel is the user's use of bufferbloat in the topic. Nothing the user can do will fix the downstream to my knowledge. Not that I'm extremely knowledgeable on the subject. Jack
On Sun, Mar 1, 2015 at 4:04 PM, Jack Bates <jbates@paradoxnetworks.net> wrote:
On 3/1/2015 5:28 PM, Dave Taht wrote:
My IP address is apparently now banned from accessing your site at all, for "advertising", on this thread:
http://community.virginmedia.com/t5/Up-to-152Mb/Bufferbloat-High-Latency-amp...
I don't see how codel is related to the customer complaint from their perspective. The problem appears to be high latency on downstream with little to no upstream. I'd probably call it off-topic advertising. The only thing that seems to relate to codel is the user's use of bufferbloat in the topic. Nothing the user can do will fix the downstream to my knowledge. Not that I'm extremely knowledgeable on the subject.
It is 100% possible to fix excessive downstream buffering from some misconfigured device with a shaper on the download *on the CPE or home router*. I have been doing that for 15 years. So has everyone that uses nearly any of the shapers that are available for Linux, at least. http://burntchrome.blogspot.com/2014_05_01_archive.html doing it yourself, right, requires a good measurement, and you lose just a little bit of single-flow bandwidth - typically 5% - but you get it all back with faster tcp ramp up times, huge improvements in dns lookups, voip, gaming, and other traffic. it generally works way better than policers do.
Jack
-- Dave Täht Let's make wifi fast, less jittery and reliable again! https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb
On 3/1/2015 6:14 PM, Dave Taht wrote:
It is 100% possible to fix excessive downstream buffering from some misconfigured device with a shaper on the download *on the CPE or home router*.
From OP: "However I've recently noticed periods of 500-800ms latency to the CMTS gateway when only using 15-20 of the 60Mbps total (and little to none upstream utilisation)." I agree with you that it is better to run a shaper that insures your shaper hits saturation and handles queue policies before the upstream does. That is great if it is your pipe (and only its queue) that is saturating. I don't think this problem qualifies. I find it difficult to believe that he's hitting a buffer bloat issue on a single (not shared with others) queue using 1/3rd of the total bandwidth available to him at those speeds and with that latency value. His problem is more likely lower down (unable to obtain max speed resulting in saturation) or a shared queue where others are saturating it and him applying a shaper will not keep others from doing so. Jack
http://burntchrome.blogspot.com/2014/05/disabling-shaping-in-one-direction-w... On Sun, Mar 1, 2015 at 4:39 PM, Jack Bates <jbates@paradoxnetworks.net> wrote:
On 3/1/2015 6:14 PM, Dave Taht wrote:
It is 100% possible to fix excessive downstream buffering from some misconfigured device with a shaper on the download *on the CPE or home router*.
From OP: "However I've recently noticed periods of 500-800ms latency to the CMTS gateway when only using 15-20 of the 60Mbps total (and little to none upstream utilisation)."
Might be. Again, all I did on that thread was provide a few pointers to bufferbloat related resources, and pointed at the downlink being a real problem quite often with links to stuff like this http://burntchrome.blogspot.com/2014/05/disabling-shaping-in-one-direction-w... and they yanked me. Certainly I could have tried again, from another IP, but ya know, some sundays are more fun than others.
I agree with you that it is better to run a shaper that insures your shaper hits saturation and handles queue policies before the upstream does. That is great if it is your pipe (and only its queue) that is saturating. I don't think this problem qualifies.
Might not. That said, it was hardly an accurate measurement. It is also perfectly feasible for the upstream device or the downstream device to be measuring these problems and deal with them appropriately. It gets progressively easier cpu-wise, as the effective bandwidth goes down. It is unfortunately nearly impossible for the next device in line to do (although we have some tools measuring interpacket "smoothness" that can provide a hint now, they are not baked yet) It was my hope, in working on the DOCSIS 3.1 standard that all the possible downstream problems would be addressed. They weren't.
I find it difficult to believe that he's hitting a buffer bloat issue on a single (not shared with others) queue using 1/3rd of the total bandwidth available to him at those speeds and with that latency value. His problem is more likely lower down (unable to obtain max speed resulting in saturation) or a shared queue where others are saturating it and him applying a shaper will not keep others from doing so.
Jack
-- Dave Täht Let's make wifi fast, less jittery and reliable again! https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb
participants (3)
-
Dave Taht -
Jack Bates -
Mel Beckman