We seem to have narrowed down the problem to our Cisco SCE packet shaper. It seems to be misclassifying about 15-20% of the DTLS traffic into encrypted bittorrent and since we have shaping rules in place to limit torrent traffic, this was causing the issue. To resolve the issue, we put the IP of our VPN ASA into a different package on the SCE and did not apply any shaping rules to it. We are still monitoring to be sure but we are quite confident this was the issue. So note to anyone out there using a shaper and has a DTLS VPN behind it, check your classifications or whitelist your VPN box! - Zachary On Tue, Dec 9, 2014 at 7:39 PM, Zachary McGibbon < zachary.mcgibbon+nanog@gmail.com> wrote:
Hi Roberto,
- We have disabled the DTLS compression feature, this has been verified on the client side that compression says 'None' - We are not using the VPN load balancing feature, the two boxes are running in an active/standby configuration - Yes we are tunnelling all traffic however local lan access is available if the user checks the checkbox in their client - We are inspecting the following: dns preset_dns_map, ftp, h323 h225, h323 ras, rsh, rtsp, esmtp, sqlnet, skinny, sunrpc, xdmcp, sip, netbios, tftp, ip-options, icmp - Jumbo frames are not configured - We are using the following encryption methods: AES128 and 2048 bit certificate - We are running ASA 9.2.2.8 on a 5545X - We are pushing the Anyconnect client version 3.1.05182
Also, I should mention what I mean when we see slow speeds. For example, my internet connection at home is a cable modem with 30mb down, 10mb up. I have done a path mtu discovery to my VPN at work and it is 1500. When I run an iperf to a server at the office without vpn I get about 28mb down, 9.5mb up. When I connect to vpn, the iperf to the same server is about 1.2mb down, and 900k up. This is way too slow!
- Zachary
On Tue, Dec 9, 2014 at 4:39 PM, Roberto <roberto@ipnetworks.it> wrote:
The big issue we are having is that many of our users are complaining of low speed when connected to the VPN. Please can you indicate more details ?
Is it enabled on the ASA the "compression" feature ? Is it enabled on the ASA the VPN Load Balancing feature ? Are you using the AnyConnect FULL TUNNEL mode ? Which are the inspection configured on the ASA for the "remote access" clients ? Have you configured the Jumbo MTU on the CISCO ASA interfaces ? Which encryption are configured on the ASA (are you using Suite B Algorithms) ? Which version of ASA are you using ? Which version of AnyConnect are you using ?
Note: protocols such as L2TP/IPSec are not hardware accelerated -- the IPSec portion of L2TP/IPSec is hardware-accelerated, but the L2TP portion is not. Likewise, the SSL portions of SVC and WebVPN use hardware acceleration, but the application layer protocols are done in software.
Best Regards,
_________________________________ Roberto Taccon
e-mail: roberto@ipnetworks.it mobile: +39 340 4751352 fax: +39 045 4850850 skype: roberto.taccon
-----Messaggio originale----- Da: NANOG [mailto:nanog-bounces@nanog.org] Per conto di Zachary McGibbon Inviato: martedì 9 dicembre 2014 21.18 A: Matthew Huff Cc: NANOG Oggetto: Re: Cisco AnyConnect speed woes!
We are trying to use SSLVPN (udp 443) and results are really all over the place. Most of our complaints are users connecting on Teksavvy however we haven't been able to reach anyone in their network team to find out if they are doing any filtering or shaping on their side.
We don't have a lot of traffic coming through Cogent, most of the users are local here in Montreal on either Bell or Videotron and they traverse through the QIX (www.qix.ca)
On Tue, Dec 9, 2014 at 3:03 PM, Matthew Huff <mhuff@ox.com> wrote:
Are you using SSLVpn or IPSEC with anyconnect? I have had more luck with performance with IPSEC than SSLVpn.
Also, just because your ISP is saying that they aren't shaping/filtering, doesn't mean they aren't.
We had major issues with users using AnyConnect when it was transversing Cogent. We were getting 5-10% packet loss (although the Cisco stats didn't show it), and it was choking on it.
---- Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff | Fax: 914-694-5669
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Zachary McGibbon Sent: Tuesday, December 9, 2014 2:42 PM To: NANOG Subject: Cisco AnyConnect speed woes!
I'm looking for some input on a situation that has been plaguing our new AnyConnect VPN setup. Any input would be valuable, we are at a loss for what the problem is.
We recently upgraded our VPN from our old Cisco 3000 VPN concentrators running PPTP and we are now running a pair of Cisco 5545x ASAs in an HA active/standby pair.
The big issue we are having is that many of our users are complaining of low speed when connected to the VPN. We have done tons of troubleshooting with Cisco TAC and we still haven't found the root of our problem.
Some tests we have done:
- We have tested changing MTU values - We have tried all combinations of encryption methods (SSL, TLS, IPSec, L2TP) with similar results - We have switched our active/standby boxes - We have tested on our spare 5545x box - We connected our spare box directly to our ISP with another IP address - We have whitelisted our VPN IP on our shaper (Cisco SCE8000) and our IPS (HP Tipping Point) - We have bypassed our Shaper and our IPS - We made sure that traffic from the routers talking to our ASAs is synchronous, OSPF was configured to load balance but this has been changed by changing the costs on the links to the ASAs - We have verified with our two ISPs that they are not doing any kind of filtering or shaping - We have noticed that in some instances that if a user is on a low speed connection that their VPN speed gets cut by about 1/3. This doesn't seem normal that the VPN would use this much overhead - We do not have the issue when connecting to VPN directly on our own network, only connections from the Internet
If you have any ideas on what we could try net, please let me know!
- Zachary