I will probably be laughed at, but I'll ask just in case. We are having particularly bad luck trying to run VPN tunnels over Comcast cable in the Chicago area. The symptoms are basically complete loss of connectivity (lasting minutes to sometimes hours), or sometimes flapping for a period of time. More often than not, a reboot of the cable modem is required. The most interesting ones involve the following: a PIX or ASA configured as an EZvpn client, connecting to a 3000 concentrator, authentication over RADIUS. When I go to look at the RADIUS logs, I see connections from the same box with small intervals. Timeout is 8 hours, so theoretically I should see 3 connections in a 24-hr period. In some cases, I see dozens, in the most egregious cases, thousands over a 24-hour period. I am taking that as an indicator of a really unstable Comcast circuit. We have not had this problem with any other ISP, anywhere in the country. I am pretty much down to telling customers to find another provider... Any thoughts or ideas on the matter will be appreciated. PS. To be fair (?) to Comcast, this is not a ubiquitous problem. It affects about 25% of the installations I get to see. Sincerely, Michael Malitsky
On Apr 27, 2010, at 12:42 PM, Michael Malitsky wrote:
I will probably be laughed at, but I'll ask just in case.
We are having particularly bad luck trying to run VPN tunnels over Comcast cable in the Chicago area. The symptoms are basically complete loss of connectivity (lasting minutes to sometimes hours), or sometimes flapping for a period of time. More often than not, a reboot of the cable modem is required. The most interesting ones involve the following: a PIX or ASA configured as an EZvpn client, connecting to a 3000 concentrator, authentication over RADIUS. When I go to look at the RADIUS logs, I see connections from the same box with small intervals. Timeout is 8 hours, so theoretically I should see 3 connections in a 24-hr period. In some cases, I see dozens, in the most egregious cases, thousands over a 24-hour period. I am taking that as an indicator of a really unstable Comcast circuit. We have not had this problem with any other ISP, anywhere in the country. I am pretty much down to telling customers to find another provider...
Any thoughts or ideas on the matter will be appreciated.
PS. To be fair (?) to Comcast, this is not a ubiquitous problem. It affects about 25% of the installations I get to see.
Sincerely, Michael Malitsky
We experienced the same thing, and switching from UDP tunnels to TCP tunnels fixed it. There are two things at play here. 1) The SMC modem/router that they insist you use for their "Small Business" cable internet service seems to have trouble with very high rates of non-TCP traffic going through its NAT. 2) Comcast rate limits non-TCP traffic somewhere on their network. Tunneling TCP inside TCP is a bad idea, but actually made the VPNs useful for us. Using IPSEC or UDP tunnels left us with tunnels that were rate limited to about 1mbps each way, until either the modem crashed or their network throttled us down to near useless speeds. I don't know if they're trying to stop customers from DoS'ing people or... exactly what the goal of it is, and couldn't ever get them to explain anything.
On Apr 27, 2010, at 10:48 AM, Kevin Day wrote:
On Apr 27, 2010, at 12:42 PM, Michael Malitsky wrote:
I will probably be laughed at, but I'll ask just in case.
We are having particularly bad luck trying to run VPN tunnels over Comcast cable in the Chicago area. The symptoms are basically complete loss of connectivity (lasting minutes to sometimes hours), or sometimes flapping for a period of time. More often than not, a reboot of the cable modem is required. The most interesting ones involve the following: a PIX or ASA configured as an EZvpn client, connecting to a 3000 concentrator, authentication over RADIUS. When I go to look at the RADIUS logs, I see connections from the same box with small intervals. Timeout is 8 hours, so theoretically I should see 3 connections in a 24-hr period. In some cases, I see dozens, in the most egregious cases, thousands over a 24-hour period. I am taking that as an indicator of a really unstable Comcast circuit. We have not had this problem with any other ISP, anywhere in the country. I am pretty much down to telling customers to find another provider...
Any thoughts or ideas on the matter will be appreciated.
PS. To be fair (?) to Comcast, this is not a ubiquitous problem. It affects about 25% of the installations I get to see.
Sincerely, Michael Malitsky
We experienced the same thing, and switching from UDP tunnels to TCP tunnels fixed it. There are two things at play here.
1) The SMC modem/router that they insist you use for their "Small Business" cable internet service seems to have trouble with very high rates of non-TCP traffic going through its NAT.
If you have business class service, insist that they put the cablemodem in BRIDGE-ONLY mode. This will resolve this issue and eliminate the unnecessary NAT.
2) Comcast rate limits non-TCP traffic somewhere on their network.
Comcast rate limits traffic in general. TCP is not less rate limited than anything else in my experience. Owen
http://ckdake.com/content/2008/disable-gateway-smart-packet-detection.html showed a feature of "Gateway Smart Packet Detection" in some SMC cable modem. The current solution is to identify the affected Comcast modem, and ask Comcast engineer to turn that "IDS" feature off remotely. I spend several days to talk with comcast about our blackboard will not work sometimes in some shared business class residential building. Finally got hold of a Regional Engineer to confess this with my tcpdump proof. Local comcast engineer may not be aware of this feature. Schilling On Tue, Apr 27, 2010 at 2:36 PM, Owen DeLong <owen@delong.com> wrote:
On Apr 27, 2010, at 10:48 AM, Kevin Day wrote:
On Apr 27, 2010, at 12:42 PM, Michael Malitsky wrote:
I will probably be laughed at, but I'll ask just in case.
We are having particularly bad luck trying to run VPN tunnels over Comcast cable in the Chicago area. The symptoms are basically complete loss of connectivity (lasting minutes to sometimes hours), or sometimes flapping for a period of time. More often than not, a reboot of the cable modem is required. The most interesting ones involve the following: a PIX or ASA configured as an EZvpn client, connecting to a 3000 concentrator, authentication over RADIUS. When I go to look at the RADIUS logs, I see connections from the same box with small intervals. Timeout is 8 hours, so theoretically I should see 3 connections in a 24-hr period. In some cases, I see dozens, in the most egregious cases, thousands over a 24-hour period. I am taking that as an indicator of a really unstable Comcast circuit. We have not had this problem with any other ISP, anywhere in the country. I am pretty much down to telling customers to find another provider...
Any thoughts or ideas on the matter will be appreciated.
PS. To be fair (?) to Comcast, this is not a ubiquitous problem. It affects about 25% of the installations I get to see.
Sincerely, Michael Malitsky
We experienced the same thing, and switching from UDP tunnels to TCP tunnels fixed it. There are two things at play here.
1) The SMC modem/router that they insist you use for their "Small Business" cable internet service seems to have trouble with very high rates of non-TCP traffic going through its NAT.
If you have business class service, insist that they put the cablemodem in BRIDGE-ONLY mode. This will resolve this issue and eliminate the unnecessary NAT.
2) Comcast rate limits non-TCP traffic somewhere on their network.
Comcast rate limits traffic in general. TCP is not less rate limited than anything else in my experience.
Owen
You can get into the SMC device yourself by going to the http://10.1.10.1/login.asp link on the SMC. The username/password are well known as cusadmin/highspeed. I also recommend against using the integrated services in the device if at all possible. It's also mildly annoying that it does not respond to traceroute either when it's your gateway with a pool of static ips. I did have one case where it reverted to a mode where it ran dhcp/nat but that was shortlived and has not happened again. - Jared On Apr 27, 2010, at 2:56 PM, schilling wrote:
http://ckdake.com/content/2008/disable-gateway-smart-packet-detection.html showed a feature of "Gateway Smart Packet Detection" in some SMC cable modem.
The current solution is to identify the affected Comcast modem, and ask Comcast engineer to turn that "IDS" feature off remotely.
I spend several days to talk with comcast about our blackboard will not work sometimes in some shared business class residential building. Finally got hold of a Regional Engineer to confess this with my tcpdump proof. Local comcast engineer may not be aware of this feature.
Schilling
On Tue, Apr 27, 2010 at 2:36 PM, Owen DeLong <owen@delong.com> wrote:
On Apr 27, 2010, at 10:48 AM, Kevin Day wrote:
On Apr 27, 2010, at 12:42 PM, Michael Malitsky wrote:
I will probably be laughed at, but I'll ask just in case.
We are having particularly bad luck trying to run VPN tunnels over Comcast cable in the Chicago area. The symptoms are basically complete loss of connectivity (lasting minutes to sometimes hours), or sometimes flapping for a period of time. More often than not, a reboot of the cable modem is required. The most interesting ones involve the following: a PIX or ASA configured as an EZvpn client, connecting to a 3000 concentrator, authentication over RADIUS. When I go to look at the RADIUS logs, I see connections from the same box with small intervals. Timeout is 8 hours, so theoretically I should see 3 connections in a 24-hr period. In some cases, I see dozens, in the most egregious cases, thousands over a 24-hour period. I am taking that as an indicator of a really unstable Comcast circuit. We have not had this problem with any other ISP, anywhere in the country. I am pretty much down to telling customers to find another provider...
Any thoughts or ideas on the matter will be appreciated.
PS. To be fair (?) to Comcast, this is not a ubiquitous problem. It affects about 25% of the installations I get to see.
Sincerely, Michael Malitsky
We experienced the same thing, and switching from UDP tunnels to TCP tunnels fixed it. There are two things at play here.
1) The SMC modem/router that they insist you use for their "Small Business" cable internet service seems to have trouble with very high rates of non-TCP traffic going through its NAT.
If you have business class service, insist that they put the cablemodem in BRIDGE-ONLY mode. This will resolve this issue and eliminate the unnecessary NAT.
2) Comcast rate limits non-TCP traffic somewhere on their network.
Comcast rate limits traffic in general. TCP is not less rate limited than anything else in my experience.
Owen
On 2010-04-27 at 14:56:04 -0400, schilling wrote:
http://ckdake.com/content/2008/disable-gateway-smart-packet-detection.html showed a feature of "Gateway Smart Packet Detection" in some SMC cable modem.
On one of our cable modems I had this manifest itself by dropping every other packet. I spent several hours trying to figure that one out, resetting the modem, and talking with L1 support. Finally someone higher up said 'Turn off SPD'. -A
On 4/27/2010 1:42 PM, Michael Malitsky wrote:
I will probably be laughed at, but I'll ask just in case.
We are having particularly bad luck trying to run VPN tunnels over Comcast cable in the Chicago area. The symptoms are basically complete loss of connectivity (lasting minutes to sometimes hours), or sometimes flapping for a period of time. More often than not, a reboot of the cable modem is required. The most interesting ones involve the following: a PIX or ASA configured as an EZvpn client, connecting to a 3000 concentrator, authentication over RADIUS. When I go to look at the RADIUS logs, I see connections from the same box with small intervals. Timeout is 8 hours, so theoretically I should see 3 connections in a 24-hr period. In some cases, I see dozens, in the most egregious cases, thousands over a 24-hour period. I am taking that as an indicator of a really unstable Comcast circuit. We have not had this problem with any other ISP, anywhere in the country. I am pretty much down to telling customers to find another provider...
Any thoughts or ideas on the matter will be appreciated.
PS. To be fair (?) to Comcast, this is not a ubiquitous problem. It affects about 25% of the installations I get to see.
Sincerely, Michael Malitsky
I ran into issues in various Comcast serviced regions with SSL VPN over tcp-443. From testing we started getting drops or severe rate limits on the flow after 7-10 minutes. Best guess was it was anti-p2p systems throttling encrypted/unknown protocol traffic after a set timer. Disconnecting and reconnecting pushed performance back up to normal until the timer kicked in again. We ended up setting the SSL tunnel to re-key via new sessions every 5 minutes to keep the flow shorter then the observed timer intervals. Other then running into a Cisco AnyConnect client bug (the app would steal focus at the re-keys) worked around the issue on Comcast and even some FiOS end users. -- --- James M Keller
In June of last year, when Comcast did firmware updates on the business gateways in the MSP area, we lost all (3) of our sites with Netgear gateways, but not the sites SMC gateways (the management interface is almost identical, but the brand is marked on the modem). Business support was apparently aware of a Cisco VPN problem through the Netgear, and simply replaced the Netgear units with SMC, and we haven't had issues since. This is IOS to ASA site-to-site VPN. Mark Mayfield City of Roseville Network Systems Engineer 2660 Civic Center Drive Roseville, MN 55113 -----Original Message----- From: Michael Malitsky [mailto:malitsky@netabn.com] Sent: Tuesday, April 27, 2010 12:43 PM To: nanog@nanog.org Subject: VPN over Comcast I will probably be laughed at, but I'll ask just in case. We are having particularly bad luck trying to run VPN tunnels over Comcast cable in the Chicago area. The symptoms are basically complete loss of connectivity (lasting minutes to sometimes hours), or sometimes flapping for a period of time. More often than not, a reboot of the cable modem is required. The most interesting ones involve the following: a PIX or ASA configured as an EZvpn client, connecting to a 3000 concentrator, authentication over RADIUS. When I go to look at the RADIUS logs, I see connections from the same box with small intervals. Timeout is 8 hours, so theoretically I should see 3 connections in a 24-hr period. In some cases, I see dozens, in the most egregious cases, thousands over a 24-hour period. I am taking that as an indicator of a really unstable Comcast circuit. We have not had this problem with any other ISP, anywhere in the country. I am pretty much down to telling customers to find another provider... Any thoughts or ideas on the matter will be appreciated. PS. To be fair (?) to Comcast, this is not a ubiquitous problem. It affects about 25% of the installations I get to see. Sincerely, Michael Malitsky Confidentiality Statement: The documents accompanying this transmission contain confidential information that is legally privileged. This information is intended only for the use of the individuals or entities listed above. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or action taken in reliance on the contents of these documents is strictly prohibited. If you have received this information in error, please notify the sender immediately and arrange for the return or destruction of these documents.
participants (8)
-
Aaron C. de Bruyn -
James M Keller -
Jared Mauch -
Kevin Day -
Mark Mayfield -
Michael Malitsky -
Owen DeLong -
schilling