I'm looking for some input on a situation that has been plaguing our new AnyConnect VPN setup. Any input would be valuable, we are at a loss for what the problem is. We recently upgraded our VPN from our old Cisco 3000 VPN concentrators running PPTP and we are now running a pair of Cisco 5545x ASAs in an HA active/standby pair. The big issue we are having is that many of our users are complaining of low speed when connected to the VPN. We have done tons of troubleshooting with Cisco TAC and we still haven't found the root of our problem. Some tests we have done: - We have tested changing MTU values - We have tried all combinations of encryption methods (SSL, TLS, IPSec, L2TP) with similar results - We have switched our active/standby boxes - We have tested on our spare 5545x box - We connected our spare box directly to our ISP with another IP address - We have whitelisted our VPN IP on our shaper (Cisco SCE8000) and our IPS (HP Tipping Point) - We have bypassed our Shaper and our IPS - We made sure that traffic from the routers talking to our ASAs is synchronous, OSPF was configured to load balance but this has been changed by changing the costs on the links to the ASAs - We have verified with our two ISPs that they are not doing any kind of filtering or shaping - We have noticed that in some instances that if a user is on a low speed connection that their VPN speed gets cut by about 1/3. This doesn't seem normal that the VPN would use this much overhead - We do not have the issue when connecting to VPN directly on our own network, only connections from the Internet If you have any ideas on what we could try net, please let me know! - Zachary
MTU should be automatically managed by the AnyConnect client. With that said, have you done PMTUd (e.g. nmap --script path-mtu <dest-ip> from one endpoint to the next)? I'd do a network map, working with your upstream provider, to identify and isolate variables. E.g. to find media changes (wrt MTU changes/mismatches). --start with icmp traceroute --next do a udp traceroute --next do a tcp traceroute --each traceroute will give you a slightly different picture, some hops will respond to one but not another --try a vpn connection from Upstream1 first, to see if it happens there. --try a vpn connection from Upstream2 next, to see if it happens there. --try a vpn connection in reverse from Upstream2, then Upstream1, to see if the speed in one direction, via one or another portal, is faster. --continue to isolate networks, network devices, until you can find the point (e.g. advertisement injector) or process (e.g. MTU LCD or asymmetric routing) which is causing this. --p -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Zachary McGibbon Sent: Tuesday, December 09, 2014 1:42 PM To: NANOG Subject: [EXTERNAL]Cisco AnyConnect speed woes! I'm looking for some input on a situation that has been plaguing our new AnyConnect VPN setup. Any input would be valuable, we are at a loss for what the problem is. We recently upgraded our VPN from our old Cisco 3000 VPN concentrators running PPTP and we are now running a pair of Cisco 5545x ASAs in an HA active/standby pair. The big issue we are having is that many of our users are complaining of low speed when connected to the VPN. We have done tons of troubleshooting with Cisco TAC and we still haven't found the root of our problem. Some tests we have done: - We have tested changing MTU values - We have tried all combinations of encryption methods (SSL, TLS, IPSec, L2TP) with similar results - We have switched our active/standby boxes - We have tested on our spare 5545x box - We connected our spare box directly to our ISP with another IP address - We have whitelisted our VPN IP on our shaper (Cisco SCE8000) and our IPS (HP Tipping Point) - We have bypassed our Shaper and our IPS - We made sure that traffic from the routers talking to our ASAs is synchronous, OSPF was configured to load balance but this has been changed by changing the costs on the links to the ASAs - We have verified with our two ISPs that they are not doing any kind of filtering or shaping - We have noticed that in some instances that if a user is on a low speed connection that their VPN speed gets cut by about 1/3. This doesn't seem normal that the VPN would use this much overhead - We do not have the issue when connecting to VPN directly on our own network, only connections from the Internet If you have any ideas on what we could try net, please let me know! - Zachary
Have you considered user protocol issues, higher up the stack where your NOC investigation can't see them? If TCP is not tuned, and detects TCP packets are dropping due to congestion, it drops (halves?) its transmit rate until all is well again. At a network operator level, you may have the L1 bandwidth ready and willing to tranport all the bits in sight, but just one poor TCP stack (old FTP? old SMB?) in the TCP roundtrip will throttle bits presented way down. I have on my desk here a badly configured example where poor TCP buffering drops throughput to 5% of expected. Well known issue, for IT folks in enterprises. Wireshark etc will easily let you see how fast user traffic is arriving. Just a thought. Roy *Roy Hirst* | 425-556-5773 | 425-324-0941 cell XKL LLC | 12020 113th Ave NE, Suite 100 | Kirkland, WA 98034 | USA On 12/9/2014 12:02 PM, Darden, Patrick wrote:
MTU should be automatically managed by the AnyConnect client. With that said, have you done PMTUd (e.g. nmap --script path-mtu <dest-ip> from one endpoint to the next)?
I'd do a network map, working with your upstream provider, to identify and isolate variables. E.g. to find media changes (wrt MTU changes/mismatches). --start with icmp traceroute --next do a udp traceroute --next do a tcp traceroute --each traceroute will give you a slightly different picture, some hops will respond to one but not another --try a vpn connection from Upstream1 first, to see if it happens there. --try a vpn connection from Upstream2 next, to see if it happens there. --try a vpn connection in reverse from Upstream2, then Upstream1, to see if the speed in one direction, via one or another portal, is faster. --continue to isolate networks, network devices, until you can find the point (e.g. advertisement injector) or process (e.g. MTU LCD or asymmetric routing) which is causing this.
--p
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Zachary McGibbon Sent: Tuesday, December 09, 2014 1:42 PM To: NANOG Subject: [EXTERNAL]Cisco AnyConnect speed woes!
I'm looking for some input on a situation that has been plaguing our new AnyConnect VPN setup. Any input would be valuable, we are at a loss for what the problem is.
We recently upgraded our VPN from our old Cisco 3000 VPN concentrators running PPTP and we are now running a pair of Cisco 5545x ASAs in an HA active/standby pair.
The big issue we are having is that many of our users are complaining of low speed when connected to the VPN. We have done tons of troubleshooting with Cisco TAC and we still haven't found the root of our problem.
Some tests we have done:
- We have tested changing MTU values - We have tried all combinations of encryption methods (SSL, TLS, IPSec, L2TP) with similar results - We have switched our active/standby boxes - We have tested on our spare 5545x box - We connected our spare box directly to our ISP with another IP address - We have whitelisted our VPN IP on our shaper (Cisco SCE8000) and our IPS (HP Tipping Point) - We have bypassed our Shaper and our IPS - We made sure that traffic from the routers talking to our ASAs is synchronous, OSPF was configured to load balance but this has been changed by changing the costs on the links to the ASAs - We have verified with our two ISPs that they are not doing any kind of filtering or shaping - We have noticed that in some instances that if a user is on a low speed connection that their VPN speed gets cut by about 1/3. This doesn't seem normal that the VPN would use this much overhead - We do not have the issue when connecting to VPN directly on our own network, only connections from the Internet
If you have any ideas on what we could try net, please let me know!
- Zachary
The information contained in this e-mail message may be privileged, confidential and protected from disclosure. If you are not the intended recipient, any dissemination, distribution or copying is strictly prohibited. If you think that you have received this e-mail message in error, please e-mail the sender at the above e-mail address.
Are you using SSLVpn or IPSEC with anyconnect? I have had more luck with performance with IPSEC than SSLVpn. Also, just because your ISP is saying that they aren't shaping/filtering, doesn't mean they aren't. We had major issues with users using AnyConnect when it was transversing Cogent. We were getting 5-10% packet loss (although the Cisco stats didn't show it), and it was choking on it. ---- Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff | Fax: 914-694-5669 -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Zachary McGibbon Sent: Tuesday, December 9, 2014 2:42 PM To: NANOG Subject: Cisco AnyConnect speed woes! I'm looking for some input on a situation that has been plaguing our new AnyConnect VPN setup. Any input would be valuable, we are at a loss for what the problem is. We recently upgraded our VPN from our old Cisco 3000 VPN concentrators running PPTP and we are now running a pair of Cisco 5545x ASAs in an HA active/standby pair. The big issue we are having is that many of our users are complaining of low speed when connected to the VPN. We have done tons of troubleshooting with Cisco TAC and we still haven't found the root of our problem. Some tests we have done: - We have tested changing MTU values - We have tried all combinations of encryption methods (SSL, TLS, IPSec, L2TP) with similar results - We have switched our active/standby boxes - We have tested on our spare 5545x box - We connected our spare box directly to our ISP with another IP address - We have whitelisted our VPN IP on our shaper (Cisco SCE8000) and our IPS (HP Tipping Point) - We have bypassed our Shaper and our IPS - We made sure that traffic from the routers talking to our ASAs is synchronous, OSPF was configured to load balance but this has been changed by changing the costs on the links to the ASAs - We have verified with our two ISPs that they are not doing any kind of filtering or shaping - We have noticed that in some instances that if a user is on a low speed connection that their VPN speed gets cut by about 1/3. This doesn't seem normal that the VPN would use this much overhead - We do not have the issue when connecting to VPN directly on our own network, only connections from the Internet If you have any ideas on what we could try net, please let me know! - Zachary
We are trying to use SSLVPN (udp 443) and results are really all over the place. Most of our complaints are users connecting on Teksavvy however we haven't been able to reach anyone in their network team to find out if they are doing any filtering or shaping on their side. We don't have a lot of traffic coming through Cogent, most of the users are local here in Montreal on either Bell or Videotron and they traverse through the QIX (www.qix.ca) On Tue, Dec 9, 2014 at 3:03 PM, Matthew Huff <mhuff@ox.com> wrote:
Are you using SSLVpn or IPSEC with anyconnect? I have had more luck with performance with IPSEC than SSLVpn.
Also, just because your ISP is saying that they aren't shaping/filtering, doesn't mean they aren't.
We had major issues with users using AnyConnect when it was transversing Cogent. We were getting 5-10% packet loss (although the Cisco stats didn't show it), and it was choking on it.
---- Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff | Fax: 914-694-5669
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Zachary McGibbon Sent: Tuesday, December 9, 2014 2:42 PM To: NANOG Subject: Cisco AnyConnect speed woes!
I'm looking for some input on a situation that has been plaguing our new AnyConnect VPN setup. Any input would be valuable, we are at a loss for what the problem is.
We recently upgraded our VPN from our old Cisco 3000 VPN concentrators running PPTP and we are now running a pair of Cisco 5545x ASAs in an HA active/standby pair.
The big issue we are having is that many of our users are complaining of low speed when connected to the VPN. We have done tons of troubleshooting with Cisco TAC and we still haven't found the root of our problem.
Some tests we have done:
- We have tested changing MTU values - We have tried all combinations of encryption methods (SSL, TLS, IPSec, L2TP) with similar results - We have switched our active/standby boxes - We have tested on our spare 5545x box - We connected our spare box directly to our ISP with another IP address - We have whitelisted our VPN IP on our shaper (Cisco SCE8000) and our IPS (HP Tipping Point) - We have bypassed our Shaper and our IPS - We made sure that traffic from the routers talking to our ASAs is synchronous, OSPF was configured to load balance but this has been changed by changing the costs on the links to the ASAs - We have verified with our two ISPs that they are not doing any kind of filtering or shaping - We have noticed that in some instances that if a user is on a low speed connection that their VPN speed gets cut by about 1/3. This doesn't seem normal that the VPN would use this much overhead - We do not have the issue when connecting to VPN directly on our own network, only connections from the Internet
If you have any ideas on what we could try net, please let me know!
- Zachary
On 12/09/2014 02:42 PM, Zachary McGibbon wrote:
I'm looking for some input on a situation that has been plaguing our new AnyConnect VPN setup. Any input would be valuable, we are at a loss for what the problem is.
We recently upgraded our VPN from our old Cisco 3000 VPN concentrators running PPTP and we are now running a pair of Cisco 5545x ASAs in an HA active/standby pair.
The big issue we are having is that many of our users are complaining of low speed when connected to the VPN. We have done tons of troubleshooting with Cisco TAC and we still haven't found the root of our problem.
Some tests we have done:
- We have tested changing MTU values - We have tried all combinations of encryption methods (SSL, TLS, IPSec, L2TP) with similar results - We have switched our active/standby boxes - We have tested on our spare 5545x box - We connected our spare box directly to our ISP with another IP address - We have whitelisted our VPN IP on our shaper (Cisco SCE8000) and our IPS (HP Tipping Point) - We have bypassed our Shaper and our IPS - We made sure that traffic from the routers talking to our ASAs is synchronous, OSPF was configured to load balance but this has been changed by changing the costs on the links to the ASAs - We have verified with our two ISPs that they are not doing any kind of filtering or shaping - We have noticed that in some instances that if a user is on a low speed connection that their VPN speed gets cut by about 1/3. This doesn't seem normal that the VPN would use this much overhead - We do not have the issue when connecting to VPN directly on our own network, only connections from the Internet
If you have any ideas on what we could try net, please let me know!
- Zachary
What OS builds? At one point the code had an 8 packet hard coded window per tcp flow, which capped ssl over tcp window size to about 5mbps depending on RTT. Recent 8 branches raised this to something more reasonable that capped around 20 mbps. DTLS over udp and IPSEC tunnels did not have this issue. -- -James
Confidently based on no knowledge at all - *Roy Hirst* | 425-556-5773 | 425-324-0941 cell XKL LLC | 12020 113th Ave NE, Suite 100 | Kirkland, WA 98034 | USA
- We have noticed that in some instances that if a user is on a low speed connection that their VPN speed gets cut by about 1/3. This doesn't seem normal that the VPN would use this much overhead
No, sure, but are you sure that congestion is not dropping a packet somewhere in the end-to-end? If you offend TCP it will likely cut the sender's packet transmit rate, even if the "possible" VPN rate is much higher.
- We do not have the issue when connecting to VPN directly on our own network, only connections from the Internet
Internet would mean maybe a proxy or firewall then, with too-small buffers or an old-time TCP/IP stack? Just a thought.
If you have any ideas on what we could try net, please let me know!
- Zachary
What OS builds? At one point the code had an 8 packet hard coded window per tcp flow, which capped ssl over tcp window size to about 5mbps depending on RTT. Recent 8 branches raised this to something more reasonable that capped around 20 mbps. DTLS over udp and IPSEC tunnels did not have this issue. UDP traffic does not have this problem but TCP does? Hmmm...
The information contained in this e-mail message may be privileged, confidential and protected from disclosure. If you are not the intended recipient, any dissemination, distribution or copying is strictly prohibited. If you think that you have received this e-mail message in error, please e-mail the sender at the above e-mail address.
On 12/11/2014 04:18 PM, Roy Hirst wrote:
Confidently based on no knowledge at all -
*Roy Hirst* | 425-556-5773 | 425-324-0941 cell XKL LLC | 12020 113th Ave NE, Suite 100 | Kirkland, WA 98034 | USA
- We have noticed that in some instances that if a user is on a low speed connection that their VPN speed gets cut by about 1/3. This doesn't seem normal that the VPN would use this much overhead
No, sure, but are you sure that congestion is not dropping a packet somewhere in the end-to-end? If you offend TCP it will likely cut the sender's packet transmit rate, even if the "possible" VPN rate is much higher.
- We do not have the issue when connecting to VPN directly on our own network, only connections from the Internet
Internet would mean maybe a proxy or firewall then, with too-small buffers or an old-time TCP/IP stack? Just a thought.
If you have any ideas on what we could try net, please let me know!
- Zachary
What OS builds? At one point the code had an 8 packet hard coded window per tcp flow, which capped ssl over tcp window size to about 5mbps depending on RTT. Recent 8 branches raised this to something more reasonable that capped around 20 mbps. DTLS over udp and IPSEC tunnels did not have this issue. UDP traffic does not have this problem but TCP does? Hmmm...
UDP transport with DTLS or IPSEC in UDP Encapsulation doesn't need to deal with tcp window size scaling and the associated packet buffers. -James
participants (5)
-
Darden, Patrick -
James Michael Keller -
Matthew Huff -
Roy Hirst -
Zachary McGibbon