VPN tunnels between US and China dropping/slow
At my current place of business, we have several manufacturing plants in China as well as the United States. All of the plants have an OVPN tunnel to a datacenter here in Indianapolis which connect all of the plants. Our China plants pay for the basic 3mbit/3mbit fiber internet connections. I've had a hell of a time keeping their tunnels up. They're running on port 443 over TCP now, but every month or so the tunnel degrades so badly I have to switch the port. I've recently tried tunneling OVPN (UDP) over a GRE tunnel and that has worked for a few months..but even now is degrading. The interesting thing is that ONLY the tunnel traffic gets degraded. I've replaced all of the equipment on both ends of all of the VPN tunnels, which changed nothing. Currently, we're talking to Time Warner and some of our customers who have plants in China to see what solutions they're using to get around this kind of issue. One thing we are hearing quite often is that they're using a MPLS based connection to Hong Kong, then going to the USA from there. We're happy to try this, but due to cost issues we're (management mostly) considering this a last resort option. Are there any other options maybe some of you have to fixing this issue? Thanks Thomas York
On Tue, 10 May 2011 10:12:57 -0400 "Thomas York" <straterra@fuhell.com> wrote:
At my current place of business, we have several manufacturing plants in China as well as the United States. All of the plants have an OVPN tunnel to a datacenter here in Indianapolis which connect all of the plants. Our China plants pay for the basic 3mbit/3mbit fiber internet connections. I've had a hell of a time keeping their tunnels up. They're running on port 443 over TCP now, but every month or so the tunnel degrades so badly I have to switch the port. I've recently tried tunneling OVPN (UDP) over a GRE tunnel and that has worked for a few months..but even now is degrading. The interesting thing is that ONLY the tunnel traffic gets degraded. I've replaced all of the equipment on both ends of all of the VPN tunnels, which changed nothing.
This is actually caused by the Chinese firewall trying to reset the VPN connection. The reason why they are doing this is because people are buying VPN services to get around the firewall. As of late, they have become a lot more clever about VPN blocking.
Currently, we're talking to Time Warner and some of our customers who have plants in China to see what solutions they're using to get around this kind of issue. One thing we are hearing quite often is that they're using a MPLS based connection to Hong Kong, then going to the USA from there. We're happy to try this, but due to cost issues we're (management mostly) considering this a last resort option. Are there any other options maybe some of you have to fixing this issue? Thanks
The only option is to get transport to an endpoint outside China, e.g. in Hong Kong. William
On Tue, May 10, 2011 at 10:35 AM, William Pitcock <nenolod@systeminplace.net> wrote:
Currently, we're talking to Time Warner and some of our customers who have plants in China to see what solutions they're using to get around this kind of issue. One thing we are hearing quite often is that they're using a MPLS based connection to Hong Kong, then going to the USA from there. We're happy to try this, but due to cost issues we're (management mostly) considering this a last resort option. Are there any other options maybe some of you have to fixing this issue? Thanks
The only option is to get transport to an endpoint outside China, e.g. in Hong Kong.
or just tunnel without a protocol... or spread-spectrum across more than one endpoint set
On 5/10/2011 10:12 AM, Thomas York wrote:
At my current place of business, we have several manufacturing plants in China as well as the United States. All of the plants have an OVPN tunnel to a datacenter here in Indianapolis which connect all of the plants. Our China plants pay for the basic 3mbit/3mbit fiber internet connections. I've had a hell of a time keeping their tunnels up. They're running on port 443 over TCP now, but every month or so the tunnel degrades so badly I have to switch the port. I've recently tried tunneling OVPN (UDP) over a GRE tunnel and
Perhaps a DPI issue ? We make use of OpenVPN a lot here. When the local ILEC started rolling out their DPI boxes, our VPN traffic was initially identified as bit torrent traffic and was being tampered with. Of course they said that was impossible... It took a good month before I was able to get to the right people to actually look at the pcaps that demonstrated the issue. I setup an openvpn tunnel between the two impacted sites (A,B)
From A, I would do a straight up icmp ping to B. It would get to the other side 100% clean.
At the same, time, I would do a ping inside the VPN tunnel. It would show dropped packets. I then used hping to generate UDP packets of the same size or bigger of the VPN packets, but with all FF as the payload, so it didnt look like anything to the DPI boxes. This too would get to the other side 100% of the time. But the VPN UDP packets would experience loss. The DPI vendor then made some patches and/or config changes to stop messing up our traffic and we have been ok since. Not sure what you can do on the China side to test things, but perhaps setup an OpenVPN instance in one of those free test instances in Amazon and see if you see the loss from there to China. ---Mike -- ------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/
Realize also that China Telecom is congested both internally and on certain peering interfaces. While DPI is a likely culprit, be sure to not overlook a good old-fashioned inability to manage capacity, combined with certain hashing algorithms... -a
On 5/10/11 10:10 AM, Adam Rothschild wrote:
Realize also that China Telecom is congested both internally and on certain peering interfaces.
While DPI is a likely culprit, be sure to not overlook a good old-fashioned inability to manage capacity, combined with certain hashing algorithms...
if you're measuring the end-to-end path you'll likely see evidenced of the latency climbing on a near daily cycle. my median rtt from the us east coast is 268ms sometimes it's north of 370 with essentially the same loss properties.
-a
participants (6)
-
Adam Rothschild
-
Christopher Morrow
-
Joel Jaeggli
-
Mike Tancsa
-
Thomas York
-
William Pitcock