I've got a local peer with Cox for VPN users to co-lo. A VPN connection that otherwise shows no issues just had their file transfer rate during a large file transfer over the VPN go from 10Mbps to 43kbps, and stay there. This isn't transit, it's local peering. The local Cox guys don't see anything wrong. Other users can access the same server over their VPN with no problems, and the end user has no issues transferring large files from other sites. Looks to me like the P2P bandwidth clamping "solution" gone bad. Anyone have any insights into what Cox are up to? P.S. The Cox customer using the VPN has a Cox Business Services link, not a home cable modem.
On Jan 25, 2008 12:17 PM, Tomas L. Byrnes <tomb@byrneit.net> wrote:
I've got a local peer with Cox for VPN users to co-lo. A VPN connection that otherwise shows no issues just had their file transfer rate during a large file transfer over the VPN go from 10Mbps to 43kbps, and stay there. This isn't transit, it's local peering.
I see the *exact* same problem with Comcast at home. I get about 30 seconds of the 6.6Mbps provisioned rate then the drop kicks in and down to 43kbps it goes. In talking with Comcast engineers privately, I've learned that the "provisioned" rates should no longer be considered as sustainable, only initial. Now I don't normally need a sustained up/down rate, but it has come in handy in the past when up/down-loading backups or ISOs... but I guess those days are behind us as the large providers have started re-interpreting the definition of "provisioned", or to be more accurate they have implemented a TTL on it. That said, I do see their point of view wrt PTP, esp torrent traffic, and their desire to limit it's impact on their networks.... but it does boil my blood when *I* need to use "my" bandwidth for legitimate purposes only to find myself throttled. :-) Part of me wonders if this isn't an effort to push "business" class services. -Jim P.
The throttling I am talking about occurred on business class service which is rated at 16Mbps/2Mbps and is NOT cheap. I'd love to know what the throttling mechanism Comcast uses is, as Cox swears up and down that they have no such thing in place. That both got throttled to the EXACT SAME, non multiple of a DS0, is just a bit too coincidental for me. If it was some sort of trunk capacity or mux issue, I would expect the BW I was left with to be a multiple of 64Kbps or 1.544 Mbps, not some odd-ball number like 43Kbps.
-----Original Message----- From: Jim Popovitch [mailto:yahoo@jimpop.com] Sent: Friday, January 25, 2008 4:50 PM To: Tomas L. Byrnes Cc: nanog@nanog.org Subject: Re: Cox clamping VPN traffic?
On Jan 25, 2008 12:17 PM, Tomas L. Byrnes <tomb@byrneit.net> wrote:
I've got a local peer with Cox for VPN users to co-lo. A VPN connection that otherwise shows no issues just had their
file transfer
rate during a large file transfer over the VPN go from 10Mbps to 43kbps, and stay there. This isn't transit, it's local peering.
I see the *exact* same problem with Comcast at home. I get about 30 seconds of the 6.6Mbps provisioned rate then the drop kicks in and down to 43kbps it goes. In talking with Comcast engineers privately, I've learned that the "provisioned" rates should no longer be considered as sustainable, only initial. Now I don't normally need a sustained up/down rate, but it has come in handy in the past when up/down-loading backups or ISOs... but I guess those days are behind us as the large providers have started re-interpreting the definition of "provisioned", or to be more accurate they have implemented a TTL on it. That said, I do see their point of view wrt PTP, esp torrent traffic, and their desire to limit it's impact on their networks.... but it does boil my blood when *I* need to use "my" bandwidth for legitimate purposes only to find myself throttled. :-) Part of me wonders if this isn't an effort to push "business" class services.
-Jim P.
On Fri, 25 Jan 2008, Tomas L. Byrnes wrote:
The throttling I am talking about occurred on business class service which is rated at 16Mbps/2Mbps and is NOT cheap. I'd love to know what the throttling mechanism Comcast uses is, as Cox swears up and down that they have no such thing in place. That both got throttled to the EXACT SAME, non multiple of a DS0, is just a bit too coincidental for me.
If it was some sort of trunk capacity or mux issue, I would expect the BW I was left with to be a multiple of 64Kbps or 1.544 Mbps, not some odd-ball number like 43Kbps.
Could it be that the provider is silently inserting RSTs into live flows? This is not unheard of and would have the capability to send throughput into the toilet. Before walking down that road, however, are you sure you're not dealing with an MTU/TCP-MSS/DF bit issue? jms
Pretty sure it's not a fragmentation or Black-Hole routing (MTU/MSS) issue, since the traffic ran fine for nearly 6 hours before being clamped, and then, after being quiescent for about 6 hours, ran fine until completion. An MTU/Fragmentation issue would show up much faster than that. I've troubleshot those, and they show up pretty much any time you try to do a large transfer, as a lost connection. Post complaining and opening a trouble ticket, and after a sheepish call from Cox engineering that dids't protest too much that they did no such thing, all is well. We changed nothing on any of the endpoints. Methinks someone accidentally applied a policy to ALL connections on a given CMTSS/Router, as opposed to just the residential ones. That's just conjecture, but my experience correlates with enough others to note that carriers are, stealthily, pushing out traffic shaping and rate limiting, sometimes with unintended consequences for non P2P services. Keep your eyes open, and let's keep all these carriers honest!
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Justin M. Streiner Sent: Friday, January 25, 2008 10:32 PM To: nanog@nanog.org Subject: RE: Cox clamping VPN traffic?
On Fri, 25 Jan 2008, Tomas L. Byrnes wrote:
The throttling I am talking about occurred on business class service which is rated at 16Mbps/2Mbps and is NOT cheap. I'd love to know what the throttling mechanism Comcast uses is, as Cox swears up and down that they have no such thing in place. That both got throttled to the EXACT SAME, non multiple of a DS0, is just a bit too coincidental for me.
If it was some sort of trunk capacity or mux issue, I would expect the BW I was left with to be a multiple of 64Kbps or 1.544 Mbps, not some odd-ball number like 43Kbps.
Could it be that the provider is silently inserting RSTs into live flows? This is not unheard of and would have the capability to send throughput into the toilet.
Before walking down that road, however, are you sure you're not dealing with an MTU/TCP-MSS/DF bit issue?
jms
It's been happening for a long time, and in many places. That's how companies like Sandvine, CacheLogic, and Allot stay in business. Frank -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Tomas L. Byrnes Sent: Saturday, January 26, 2008 1:02 AM To: Justin M. Streiner; nanog@nanog.org Subject: RE: Cox clamping VPN traffic? <snip> "....but my experience correlates with enough others to note that carriers are, stealthily, pushing out traffic shaping and rate limiting, sometimes with unintended consequences for non P2P services."
Could it be that the provider is silently inserting RSTs into live flows?
He's talking about traffic inside a VPN also there doesn't seem to be a problem with the transfer stopping (which RSTs would cause) just slowing down.
Before walking down that road, however, are you sure you're not dealing with an MTU/TCP-MSS/DF bit issue? I'd suggest that this is a reasonable angle - I've seen various VPNs go slowly because of these kinds of issues.
MMC -- Matthew Moyle-Croft - Internode/Agile - Networks Level 3, 132 Grenfell Street, Adelaide, SA 5000 Australia Email: mmc@internode.com.au Web: http://www.on.net Direct: +61-8-8228-2909 Mobile: +61-419-900-366 Reception: +61-8-8228-2999 Fax: +61-8-8235-6909 "The difficulty lies, not in the new ideas, but in escaping from the old ones" - John Maynard Keynes
In article <70D072392E56884193E3D2DE09C097A9EE23@pascal.zaphodb.org>, Tomas L. Byrnes <tomb@byrneit.net> writes
some odd-ball number like 43Kbps.
There are slightly more Google hits for "44kbps throttling" than "43kbps throttling". On balance, I think your observations are a co-incidence, and whatever throttling mechanism it is that the networks aren't deploying, appears at fairly random numbers in the 35-50kbps range. -- Roland Perry
On Jan 25, 2008 7:49 PM, Jim Popovitch <yahoo@jimpop.com> wrote:
I see the *exact* same problem with Comcast at home. I get about 30 seconds of the 6.6Mbps provisioned rate then the drop kicks in and down to 43kbps it goes.
I suspect this is just bursting/clamping, as you suspect, but you may also want to investigate traffic shaping at your end. I've found I get much better *receive* throughput if I limit my *transmit* rate to less than nominal maximum. Presumably, this has to do with the fact that the feed is asymmetric; I can receive much faster than I can send, and so the send channel becomes congested and that impacts TCP ACK or other protocol control messages.
In talking with Comcast engineers privately, I've learned that the "provisioned" rates should no longer be considered as sustainable, only initial.
In all fairness, Comcast has made this very clear from day one. Heck, they've advertised it extensively on TV, under the brand name "SpeedBoost". They give you a higher initial rate on a given connection, and then reduce it after a few seconds. This makes anything that consists of irregular short requests a lot faster -- web browsing being the obvious intent. Whether you call it "bursting" (initial) or "clamping" (eventual) is semantics. :)
Part of me wonders if this isn't an effort to push "business" class services.
We've got Comcast Workplace at the office for our web browsing (cheap, disposable bandwidth) and the same "SpeedBoost" terms apply. They do offer a symmetric feed option, but it's only 1.5 megabit/sec and costs $300/month. -- Ben
On Fri, 25 Jan 2008 20:30:20 -0500 "Ben Scott" <mailvortex@gmail.com> wrote:
On Jan 25, 2008 7:49 PM, Jim Popovitch <yahoo@jimpop.com> wrote:
I see the *exact* same problem with Comcast at home. I get about 30 seconds of the 6.6Mbps provisioned rate then the drop kicks in and down to 43kbps it goes.
I suspect this is just bursting/clamping, as you suspect, but you may also want to investigate traffic shaping at your end. I've found I get much better *receive* throughput if I limit my *transmit* rate to less than nominal maximum. Presumably, this has to do with the fact that the feed is asymmetric; I can receive much faster than I can send, and so the send channel becomes congested and that impacts TCP ACK or other protocol control messages.
For more details, "RFC3449 - TCP Performance Implications of Network Path Asymmetry" http://tools.ietf.org/rfc/rfc3449.txt Regards, Mark. -- "Sheep are slow and tasty, and therefore must remain constantly alert." - Bruce Schneier, "Beyond Fear"
On Fri, 25 Jan 2008, Ben Scott wrote:
I suspect this is just bursting/clamping, as you suspect, but you may also want to investigate traffic shaping at your end. I've found I get much better *receive* throughput if I limit my *transmit* rate to less than nominal maximum. Presumably, this has to do with the fact that the feed is asymmetric; I can receive much faster than I can send, and so the send channel becomes congested and that impacts TCP ACK or other protocol control messages.
If you're the Linux router type, check out Wondershaper. It's a simple set of QoS tools, and it's literally a wonder. I can saturate my outbound on Cox and still run ssh or play an FPS with no real hassle. Swapping to something Linksys flavored the last time I took my firewall down for hardware changes was actually noticeable. - billn
participants (9)
-
Ben Scott -
Bill Nash -
Frank Bulk -
Jim Popovitch -
Justin M. Streiner -
Mark Smith -
Matthew Moyle-Croft -
Roland Perry -
Tomas L. Byrnes