Hi All, We are an MSP in San Diego that also offers multi-datacenter Colo and Cloud hosting. We are multi-homed with Level 3 and Cogent. A physical server client reported newly slow FTP transfers, so we started a network investigation. Our data (see below) seem to show packet loss through Level 3 with associated slow TCP based data transfers. Is anyone else seeing packet loss and consequent slow TCP based transfers when going through Level 3 in (LosAngeles1) Southern California? Thanks, Greg Focaccio CARI.net<https://cari.net/> Testing Data: ********** Internal tests - OK ********** FTP transfers from client server to another server 3 hops away in our adjacent datacenter was normal ********** Level 3 data - packet loss ********** External tests via Level 3 - 12% Packet Loss - see data below UDP IPERF testing from our data center (through Level 3 and Microsoft - trace below) to an Azure server showed repeatable packet loss TCP based testing - such as FTP or SCP transfers - the rate was very slow about 4Mbps [root@raynor ~]# iperf3 -c 40.80.156.2 -b 10000m Connecting to host 40.80.156.2, port 5201 [ 4] local 71.6.220.101 port 55684 connected to 40.80.156.2 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 1001 KBytes 8.20 Mbits/sec 34 9.76 KBytes [ 4] 1.00-2.00 sec 503 KBytes 4.12 Mbits/sec 12 11.2 KBytes [ 4] 2.00-3.00 sec 502 KBytes 4.11 Mbits/sec 3 25.1 KBytes [ 4] 3.00-4.00 sec 502 KBytes 4.11 Mbits/sec 12 11.2 KBytes [ 4] 4.00-5.00 sec 377 KBytes 3.08 Mbits/sec 9 15.3 KBytes [ 4] 5.00-6.00 sec 502 KBytes 4.11 Mbits/sec 10 19.5 KBytes [ 4] 6.00-7.00 sec 377 KBytes 3.09 Mbits/sec 13 6.97 KBytes [ 4] 7.00-8.00 sec 251 KBytes 2.06 Mbits/sec 9 5.58 KBytes [ 4] 8.00-9.00 sec 251 KBytes 2.06 Mbits/sec 6 9.76 KBytes [ 4] 9.00-10.00 sec 251 KBytes 2.06 Mbits/sec 10 12.6 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 4.41 MBytes 3.70 Mbits/sec 118 sender [ 4] 0.00-10.00 sec 4.24 MBytes 3.56 Mbits/sec receiver iperf Done. [root@raynor ~]# iperf3 -u -c 40.80.156.2 -b 10000m Connecting to host 40.80.156.2, port 5201 [ 4] local 71.6.220.101 port 39221 connected to 40.80.156.2 port 5201 [ ID] Interval Transfer Bandwidth Total Datagrams [ 4] 0.00-1.00 sec 85.0 MBytes 712 Mbits/sec 62402 [ 4] 1.00-2.00 sec 94.0 MBytes 789 Mbits/sec 69043 [ 4] 2.00-3.00 sec 92.6 MBytes 777 Mbits/sec 68030 [ 4] 3.00-4.00 sec 76.5 MBytes 641 Mbits/sec 56153 [ 4] 4.00-5.00 sec 94.9 MBytes 796 Mbits/sec 69662 [ 4] 5.00-6.00 sec 97.7 MBytes 819 Mbits/sec 71713 [ 4] 6.00-7.00 sec 98.5 MBytes 826 Mbits/sec 72347 [ 4] 7.00-8.00 sec 92.7 MBytes 778 Mbits/sec 68085 [ 4] 8.00-9.00 sec 91.3 MBytes 765 Mbits/sec 67045 [ 4] 9.00-10.00 sec 59.3 MBytes 498 Mbits/sec 43551 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 4] 0.00-10.00 sec 883 MBytes 740 Mbits/sec 0.014 ms 75647/647955 (12%) [ 4] Sent 647955 datagrams iperf Done. [root@raynor ~]# traceroute 40.80.156.2 traceroute to 40.80.156.2 (40.80.156.2), 30 hops max, 60 byte packets 1 gateway (71.6.220.97) 6.170 ms 7.827 ms 10.718 ms 2 209.126.134.65 (209.126.134.65) 0.426 ms 0.420 ms 0.543 ms 3 216.98.153.21 (216.98.153.21) 159.936 ms 160.215 ms 160.209 ms 4 gi1-2.gw65-50.c5.sdcix.net (216.98.153.89) 170.977 ms 170.972 ms 170.964 ms 5 xe-8-3-3.bar1.SanDiego1.Level3.net (4.16.105.93) 0.547 ms 0.586 ms 0.533 ms 6 * * * 7 * * * 8 Microsoft-level3-20G.LosAngeles1.Level3.net (4.68.111.122) 17.296 ms 18.026 ms 20.200 ms 9 be-61-0.ibr01.lax03.ntwk.msn.net (104.44.8.104) 30.108 ms 32.300 ms 32.283 ms 10 be-4-0.ibr01.by2.ntwk.msn.net (104.44.4.3) 30.364 ms 30.961 ms 28.960 ms 11 104.44.7.198 (104.44.7.198) 31.870 ms 30.120 ms 29.636 ms 12 ae100-0.icr01.by21.ntwk.msn.net (104.44.11.194) 27.828 ms ae101-0.icr01.by4.ntwk.msn.net (104.44.11.193) 29.611 ms ae100-0.icr01.by21.ntwk.msn.net (104.44.11.194) 27.710 ms 13 * * * [root@raynor ~]# scp ubuntu-14.04.5-desktop-amd64.iso carinet@40.80.156.2:/home/carinet/ubuntunew11.iso<mailto:carinet@40.80.156.2:/home/carinet/ubuntunew11.iso> ubuntu-14.04.5-desktop-amd64.iso 100% 1053MB 730.8KB/s 24:35 LEVEL3 - summary CARIcloud to Azure 2 Mbits/s TCP iperf 800 Mbits/s UDP iperf 24m and 35s 1053 MB upload to Azure through SCP ********** Cogent data - OK ********** External tests via Cogent - OK - No significant loss - see IPERF and trace data below [root@raynor ~]# iperf3 -c 40.80.156.2 -b 10000m Connecting to host 40.80.156.2, port 5201 [ 4] local 71.6.220.101 port 45076 connected to 40.80.156.2 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 149 MBytes 1.25 Gbits/sec 0 4.58 MBytes [ 4] 1.00-2.00 sec 180 MBytes 1.51 Gbits/sec 0 4.58 MBytes [ 4] 2.00-3.00 sec 178 MBytes 1.50 Gbits/sec 0 4.58 MBytes [ 4] 3.00-4.00 sec 180 MBytes 1.51 Gbits/sec 0 4.58 MBytes [ 4] 4.00-5.00 sec 180 MBytes 1.51 Gbits/sec 0 4.58 MBytes [ 4] 5.00-6.00 sec 180 MBytes 1.51 Gbits/sec 0 4.58 MBytes [ 4] 6.00-7.00 sec 179 MBytes 1.50 Gbits/sec 0 4.58 MBytes [ 4] 7.00-8.00 sec 180 MBytes 1.51 Gbits/sec 0 4.58 MBytes [ 4] 8.00-9.00 sec 181 MBytes 1.52 Gbits/sec 0 4.58 MBytes [ 4] 9.00-10.00 sec 180 MBytes 1.51 Gbits/sec 0 4.58 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 1.73 GBytes 1.48 Gbits/sec 0 sender [ 4] 0.00-10.00 sec 1.73 GBytes 1.48 Gbits/sec receiver iperf Done. [root@raynor ~]# iperf3 -u -c 40.80.156.2 -b 10000m Connecting to host 40.80.156.2, port 5201 [ 4] local 71.6.220.101 port 39130 connected to 40.80.156.2 port 5201 [ ID] Interval Transfer Bandwidth Total Datagrams [ 4] 0.00-1.00 sec 101 MBytes 844 Mbits/sec 73869 [ 4] 1.00-2.00 sec 103 MBytes 861 Mbits/sec 75344 [ 4] 2.00-3.00 sec 99.0 MBytes 830 Mbits/sec 72700 [ 4] 3.00-4.00 sec 97.9 MBytes 821 Mbits/sec 71885 [ 4] 4.00-5.00 sec 98.0 MBytes 822 Mbits/sec 71979 [ 4] 5.00-6.00 sec 100 MBytes 841 Mbits/sec 73640 [ 4] 6.00-7.00 sec 69.8 MBytes 585 Mbits/sec 51237 [ 4] 7.00-8.00 sec 94.9 MBytes 796 Mbits/sec 69650 [ 4] 8.00-9.00 sec 102 MBytes 855 Mbits/sec 74804 [ 4] 9.00-10.00 sec 100 MBytes 839 Mbits/sec 73407 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 4] 0.00-10.00 sec 965 MBytes 809 Mbits/sec 0.004 ms 3194/708505 (0.45%) [ 4] Sent 708505 datagrams iperf Done. [root@raynor ~]# traceroute 40.80.156.2 traceroute to 40.80.156.2 (40.80.156.2), 30 hops max, 60 byte packets 1 gateway (71.6.220.97) 9.892 ms 10.883 ms 15.574 ms 2 209.126.134.65 (209.126.134.65) 1.541 ms 1.966 ms 1.959 ms 3 216.98.153.21 (216.98.153.21) 0.523 ms 0.517 ms 0.511 ms 4 216.98.153.114 (216.98.153.114) 0.589 ms 0.480 ms 0.687 ms 5 te0-0-1-0.nr11.b005949-0.san01.atlas.cogentco.com (38.104.122.61) 1.045 ms 1.039 ms 1.072 ms 6 te0-0-0-3.agr11.san01.atlas.cogentco.com (154.24.32.53) 1.023 ms 0.823 ms te0-0-0-3.agr12.san01.atlas.cogentco.com (154.24.32.65) 1.359 ms 7 te0-0-1-3.rcr11.san01.atlas.cogentco.com (154.24.31.21) 0.919 ms te0-0-1-3.rcr12.san01.atlas.cogentco.com (154.24.31.37) 0.834 ms 0.874 ms 8 be2936.rcr11.sna02.atlas.cogentco.com (154.54.45.166) 3.947 ms 3.726 ms be2937.rcr12.sna02.atlas.cogentco.com (154.54.45.173) 3.221 ms 9 be2463.agr22.lax01.atlas.cogentco.com (154.54.80.61) 4.095 ms 3.960 ms 4.087 ms 10 be2584.ccr41.lax01.atlas.cogentco.com (154.54.29.33) 4.364 ms 3.961 ms be2586.ccr41.lax01.atlas.cogentco.com (154.54.29.245) 4.290 ms 11 be3360.ccr41.lax04.atlas.cogentco.com (154.54.25.150) 4.283 ms be3271.ccr41.lax04.atlas.cogentco.com (154.54.42.102) 4.020 ms 4.003 ms 12 38.142.33.250 (38.142.33.250) 3.828 ms 3.818 ms 3.481 ms 13 be-61-0.ibr01.lax03.ntwk.msn.net (104.44.8.104) 15.565 ms 15.291 ms 15.173 ms 14 be-4-0.ibr01.by2.ntwk.msn.net (104.44.4.3) 14.290 ms 13.903 ms 16.168 ms 15 104.44.7.198 (104.44.7.198) 15.445 ms 14.098 ms 13.866 ms 16 ae102-0.icr02.by21.ntwk.msn.net (104.44.11.198) 13.752 ms ae100-0.icr01.by21.ntwk.msn.net (104.44.11.194) 14.378 ms ae101-0.icr01.by4.ntwk.msn.net (104.44.11.193) 13.813 ms [root@raynor ~]# scp ubuntu-14.04.5-desktop-amd64.iso carinet@40.80.156.2:/home/carinet/ubuntunew10.iso<mailto:carinet@40.80.156.2:/home/carinet/ubuntunew10.iso> ubuntu-14.04.5-desktop-amd64.iso 100% 1053MB 111.5MB/s 00:09 COGENT - summary CARIcloud to Azure 1.5 Gbits/s TCP iperf 850 Mbits/s UDP iperf 9s 1053 MB upload to Azure through SCP