December 2017 - Test - lists.nanog.org

Weekly Routing Table Report
by cscora＠apnic.net 01 Dec '17

01 Dec '17

This is an automated weekly mailing describing the state of the Internet Routing Table as seen from APNIC's router in Japan. The posting is sent to APOPS, NANOG, AfNOG, SANOG, PacNOG, SAFNOG, CaribNOG TZNOG, MENOG, BJNOG, SDNOG, CMNOG, LACNOG, IRNOG and the RIPE Routing WG. Daily listings are sent to bgp-stats(a)lists.apnic.net For historical data, please see http://thyme.rand.apnic.net. If you have any comments please contact Philip Smith <pfsinoz(a)gmail.com>. Routing Table Report 04:00 +10GMT Sat 02 Dec, 2017 Report Website: http://thyme.rand.apnic.net Detailed Analysis: http://thyme.rand.apnic.net/current/ Analysis Summary ---------------- BGP routing table entries examined: 672139 Prefixes after maximum aggregation (per Origin AS): 261947 Deaggregation factor: 2.57 Unique aggregates announced (without unneeded subnets): 325805 Total ASes present in the Internet Routing Table: 59187 Prefixes per ASN: 11.36 Origin-only ASes present in the Internet Routing Table: 51121 Origin ASes announcing only one prefix: 22517 Transit ASes present in the Internet Routing Table: 8066 Transit-only ASes present in the Internet Routing Table: 238 Average AS path length visible in the Internet Routing Table: 4.4 Max AS path length visible: 42 Max AS path prepend of ASN ( 23679) 26 Prefixes from unregistered ASNs in the Routing Table: 85 Number of instances of unregistered ASNs: 85 Number of 32-bit ASNs allocated by the RIRs: 20763 Number of 32-bit ASNs visible in the Routing Table: 16600 Prefixes from 32-bit ASNs in the Routing Table: 68130 Number of bogon 32-bit ASNs visible in the Routing Table: 11 Special use prefixes present in the Routing Table: 0 Prefixes being announced from unallocated address space: 327 Number of addresses announced to Internet: 2861310370 Equivalent to 170 /8s, 140 /16s and 33 /24s Percentage of available address space announced: 77.3 Percentage of allocated address space announced: 77.3 Percentage of available address space allocated: 100.0 Percentage of address space in use by end-sites: 98.8 Total number of prefixes smaller than registry allocations: 222321 APNIC Region Analysis Summary ----------------------------- Prefixes being announced by APNIC Region ASes: 184628 Total APNIC prefixes after maximum aggregation: 53090 APNIC Deaggregation factor: 3.48 Prefixes being announced from the APNIC address blocks: 183767 Unique aggregates announced from the APNIC address blocks: 77055 APNIC Region origin ASes present in the Internet Routing Table: 8496 APNIC Prefixes per ASN: 21.63 APNIC Region origin ASes announcing only one prefix: 2393 APNIC Region transit ASes present in the Internet Routing Table: 1231 Average APNIC Region AS path length visible: 4.5 Max APNIC Region AS path length visible: 42 Number of APNIC region 32-bit ASNs visible in the Routing Table: 3373 Number of APNIC addresses announced to Internet: 767131170 Equivalent to 45 /8s, 185 /16s and 126 /24s APNIC AS Blocks 4608-4864, 7467-7722, 9216-10239, 17408-18431 (pre-ERX allocations) 23552-24575, 37888-38911, 45056-46079, 55296-56319, 58368-59391, 63488-64098, 64297-64395, 131072-137529 APNIC Address Blocks 1/8, 14/8, 27/8, 36/8, 39/8, 42/8, 43/8, 49/8, 58/8, 59/8, 60/8, 61/8, 101/8, 103/8, 106/8, 110/8, 111/8, 112/8, 113/8, 114/8, 115/8, 116/8, 117/8, 118/8, 119/8, 120/8, 121/8, 122/8, 123/8, 124/8, 125/8, 126/8, 133/8, 150/8, 153/8, 163/8, 171/8, 175/8, 180/8, 182/8, 183/8, 202/8, 203/8, 210/8, 211/8, 218/8, 219/8, 220/8, 221/8, 222/8, 223/8, ARIN Region Analysis Summary ---------------------------- Prefixes being announced by ARIN Region ASes: 201468 Total ARIN prefixes after maximum aggregation: 97048 ARIN Deaggregation factor: 2.08 Prefixes being announced from the ARIN address blocks: 202861 Unique aggregates announced from the ARIN address blocks: 94992 ARIN Region origin ASes present in the Internet Routing Table: 18034 ARIN Prefixes per ASN: 11.25 ARIN Region origin ASes announcing only one prefix: 6676 ARIN Region transit ASes present in the Internet Routing Table: 1784 Average ARIN Region AS path length visible: 3.8 Max ARIN Region AS path length visible: 23 Number of ARIN region 32-bit ASNs visible in the Routing Table: 2263 Number of ARIN addresses announced to Internet: 1112764192 Equivalent to 66 /8s, 83 /16s and 111 /24s ARIN AS Blocks 1-1876, 1902-2042, 2044-2046, 2048-2106 (pre-ERX allocations) 2138-2584, 2615-2772, 2823-2829, 2880-3153 3354-4607, 4865-5119, 5632-6655, 6912-7466 7723-8191, 10240-12287, 13312-15359, 16384-17407 18432-20479, 21504-23551, 25600-26591, 26624-27647, 29696-30719, 31744-33791 35840-36863, 39936-40959, 46080-47103 53248-55295, 62464-63487, 64198-64296, 393216-397212 ARIN Address Blocks 3/8, 4/8, 6/8, 7/8, 8/8, 9/8, 11/8, 12/8, 13/8, 15/8, 16/8, 17/8, 18/8, 19/8, 20/8, 21/8, 22/8, 23/8, 24/8, 26/8, 28/8, 29/8, 30/8, 32/8, 33/8, 34/8, 35/8, 38/8, 40/8, 44/8, 45/8, 47/8, 48/8, 50/8, 52/8, 53/8, 54/8, 55/8, 56/8, 57/8, 63/8, 64/8, 65/8, 66/8, 67/8, 68/8, 69/8, 70/8, 71/8, 72/8, 73/8, 74/8, 75/8, 76/8, 96/8, 97/8, 98/8, 99/8, 100/8, 104/8, 107/8, 108/8, 128/8, 129/8, 130/8, 131/8, 132/8, 134/8, 135/8, 136/8, 137/8, 138/8, 139/8, 140/8, 142/8, 143/8, 144/8, 146/8, 147/8, 148/8, 149/8, 152/8, 155/8, 156/8, 157/8, 158/8, 159/8, 160/8, 161/8, 162/8, 164/8, 165/8, 166/8, 167/8, 168/8, 169/8, 170/8, 172/8, 173/8, 174/8, 184/8, 192/8, 198/8, 199/8, 204/8, 205/8, 206/8, 207/8, 208/8, 209/8, 214/8, 215/8, 216/8, RIPE Region Analysis Summary ---------------------------- Prefixes being announced by RIPE Region ASes: 181131 Total RIPE prefixes after maximum aggregation: 88587 RIPE Deaggregation factor: 2.04 Prefixes being announced from the RIPE address blocks: 177042 Unique aggregates announced from the RIPE address blocks: 106948 RIPE Region origin ASes present in the Internet Routing Table: 24674 RIPE Prefixes per ASN: 7.18 RIPE Region origin ASes announcing only one prefix: 11273 RIPE Region transit ASes present in the Internet Routing Table: 3496 Average RIPE Region AS path length visible: 4.5 Max RIPE Region AS path length visible: 30 Number of RIPE region 32-bit ASNs visible in the Routing Table: 6443 Number of RIPE addresses announced to Internet: 713373312 Equivalent to 42 /8s, 133 /16s and 54 /24s RIPE AS Blocks 1877-1901, 2043, 2047, 2107-2136, 2585-2614 (pre-ERX allocations) 2773-2822, 2830-2879, 3154-3353, 5377-5631 6656-6911, 8192-9215, 12288-13311, 15360-16383 20480-21503, 24576-25599, 28672-29695 30720-31743, 33792-35839, 38912-39935 40960-45055, 47104-52223, 56320-58367 59392-61439, 61952-62463, 64396-64495 196608-207259 RIPE Address Blocks 2/8, 5/8, 25/8, 31/8, 37/8, 46/8, 51/8, 62/8, 77/8, 78/8, 79/8, 80/8, 81/8, 82/8, 83/8, 84/8, 85/8, 86/8, 87/8, 88/8, 89/8, 90/8, 91/8, 92/8, 93/8, 94/8, 95/8, 109/8, 141/8, 145/8, 151/8, 176/8, 178/8, 185/8, 188/8, 193/8, 194/8, 195/8, 212/8, 213/8, 217/8, LACNIC Region Analysis Summary ------------------------------ Prefixes being announced by LACNIC Region ASes: 86955 Total LACNIC prefixes after maximum aggregation: 19251 LACNIC Deaggregation factor: 4.52 Prefixes being announced from the LACNIC address blocks: 88230 Unique aggregates announced from the LACNIC address blocks: 39399 LACNIC Region origin ASes present in the Internet Routing Table: 6620 LACNIC Prefixes per ASN: 13.33 LACNIC Region origin ASes announcing only one prefix: 1819 LACNIC Region transit ASes present in the Internet Routing Table: 1217 Average LACNIC Region AS path length visible: 5.2 Max LACNIC Region AS path length visible: 29 Number of LACNIC region 32-bit ASNs visible in the Routing Table: 4146 Number of LACNIC addresses announced to Internet: 172636640 Equivalent to 10 /8s, 74 /16s and 57 /24s LACNIC AS Blocks 26592-26623, 27648-28671, 52224-53247, 61440-61951, 64099-64197, 262144-268700 + ERX transfers LACNIC Address Blocks 177/8, 179/8, 181/8, 186/8, 187/8, 189/8, 190/8, 191/8, 200/8, 201/8, AfriNIC Region Analysis Summary ------------------------------- Prefixes being announced by AfriNIC Region ASes: 17870 Total AfriNIC prefixes after maximum aggregation: 3909 AfriNIC Deaggregation factor: 4.57 Prefixes being announced from the AfriNIC address blocks: 19912 Unique aggregates announced from the AfriNIC address blocks: 7142 AfriNIC Region origin ASes present in the Internet Routing Table: 1098 AfriNIC Prefixes per ASN: 18.13 AfriNIC Region origin ASes announcing only one prefix: 355 AfriNIC Region transit ASes present in the Internet Routing Table: 218 Average AfriNIC Region AS path length visible: 4.7 Max AfriNIC Region AS path length visible: 21 Number of AfriNIC region 32-bit ASNs visible in the Routing Table: 375 Number of AfriNIC addresses announced to Internet: 95019008 Equivalent to 5 /8s, 169 /16s and 224 /24s AfriNIC AS Blocks 36864-37887, 327680-328703 & ERX transfers AfriNIC Address Blocks 41/8, 102/8, 105/8, 154/8, 196/8, 197/8, APNIC Region per AS prefix count summary ---------------------------------------- ASN No of nets /20 equiv MaxAgg Description 4538 5397 4192 75 ERX-CERNET-BKB China Education and Rese 7545 4067 410 408 TPG-INTERNET-AP TPG Telecom Limited, AU 4766 2786 11128 757 KIXS-AS-KR Korea Telecom, KR 17974 2778 916 76 TELKOMNET-AS2-AP PT Telekomunikasi Indo 9829 2763 1499 540 BSNL-NIB National Internet Backbone, IN 45899 2415 1521 159 VNPT-AS-VN VNPT Corp, VN 7552 2356 1158 55 VIETEL-AS-AP Viettel Group, VN 9394 2168 4931 26 CTTNET China TieTong Telecommunications 4755 2083 421 211 TATACOMM-AS TATA Communications formerl 9808 2068 8699 32 CMNET-GD Guangdong Mobile Communication Complete listing at http://thyme.rand.apnic.net/current/data-ASnet-APNIC ARIN Region per AS prefix count summary --------------------------------------- ASN No of nets /20 equiv MaxAgg Description 6327 3340 1323 87 SHAW - Shaw Communications Inc., CA 22773 3303 2951 147 ASN-CXA-ALL-CCI-22773-RDC - Cox Communi 18566 2171 405 107 MEGAPATH5-US - MegaPath Corporation, US 20115 2057 2290 455 CHARTER-NET-HKY-NC - Charter Communicat 6389 2012 3671 39 BELLSOUTH-NET-BLK - BellSouth.net Inc., 16509 1944 4076 586 AMAZON-02 - Amazon.com, Inc., US 209 1929 5062 646 CENTURYLINK-US-LEGACY-QWEST - Qwest Com 30036 1858 331 250 MEDIACOM-ENTERPRISE-BUSINESS - Mediacom 11492 1686 234 570 CABLEONE - CABLE ONE, INC., US 701 1547 10589 625 UUNET - MCI Communications Services, In Complete listing at http://thyme.rand.apnic.net/current/data-ASnet-ARIN RIPE Region per AS prefix count summary --------------------------------------- ASN No of nets /20 equiv MaxAgg Description 39891 3134 170 23 ALJAWWALSTC-AS, SA 20940 2856 852 2061 AKAMAI-ASN1, US 8551 2493 378 42 BEZEQ-INTERNATIONAL-AS Bezeqint Interne 12389 2062 1795 695 ROSTELECOM-AS, RU 34984 2059 330 433 TELLCOM-AS, TR 12479 1939 1069 124 UNI2-AS, ES 9121 1875 1691 18 TTNET, TR 13188 1605 100 46 TRIOLAN, UA 6849 1227 355 21 UKRTELNET, UA 8402 1221 544 14 CORBINA-AS OJSC "Vimpelcom", RU Complete listing at http://thyme.rand.apnic.net/current/data-ASnet-RIPE LACNIC Region per AS prefix count summary ----------------------------------------- ASN No of nets /20 equiv MaxAgg Description 8151 3645 3465 599 Uninet S.A. de C.V., MX 10620 3557 539 276 Telmex Colombia S.A., CO 11830 2647 370 66 Instituto Costarricense de Electricidad 6503 1625 437 53 Axtel, S.A.B. de C.V., MX 7303 1494 1021 195 Telecom Argentina S.A., AR 6147 1222 377 27 Telefonica del Peru S.A.A., PE 3816 1034 512 103 COLOMBIA TELECOMUNICACIONES S.A. ESP, C 28573 1013 2248 191 CLARO S.A., BR 11172 914 126 85 Alestra, S. de R.L. de C.V., MX 18881 904 1065 27 TELEFÔNICA BRASIL S.A, BR Complete listing at http://thyme.rand.apnic.net/current/data-ASnet-LACNIC AfriNIC Region per AS prefix count summary ------------------------------------------ ASN No of nets /20 equiv MaxAgg Description 24863 1212 398 48 LINKdotNET-AS, EG 37611 806 91 8 Afrihost, ZA 36903 737 370 135 MT-MPLS, MA 36992 679 1383 31 ETISALAT-MISR, EG 8452 534 1730 17 TE-AS TE-AS, EG 24835 466 850 15 RAYA-AS, EG 29571 435 36 10 CITelecom-AS, CI 37492 425 367 77 ORANGE-, TN 37342 356 32 1 MOVITEL, MZ 15399 349 35 7 WANANCHI-, KE Complete listing at http://thyme.rand.apnic.net/current/data-ASnet-AFRINIC Global Per AS prefix count summary ---------------------------------- ASN No of nets /20 equiv MaxAgg Description 4538 5397 4192 75 ERX-CERNET-BKB China Education and Rese 7545 4067 410 408 TPG-INTERNET-AP TPG Telecom Limited, AU 8151 3645 3465 599 Uninet S.A. de C.V., MX 10620 3557 539 276 Telmex Colombia S.A., CO 6327 3340 1323 87 SHAW - Shaw Communications Inc., CA 22773 3303 2951 147 ASN-CXA-ALL-CCI-22773-RDC - Cox Communi 39891 3134 170 23 ALJAWWALSTC-AS, SA 20940 2856 852 2061 AKAMAI-ASN1, US 4766 2786 11128 757 KIXS-AS-KR Korea Telecom, KR 17974 2778 916 76 TELKOMNET-AS2-AP PT Telekomunikasi Indo Complete listing at http://thyme.rand.apnic.net/current/data-ASnet Global Per AS Maximum Aggr summary ---------------------------------- ASN No of nets Net Savings Description 7545 4067 3659 TPG-INTERNET-AP TPG Telecom Limited, AU 10620 3557 3281 Telmex Colombia S.A., CO 6327 3340 3253 SHAW - Shaw Communications Inc., CA 22773 3303 3156 ASN-CXA-ALL-CCI-22773-RDC - Cox Communications 39891 3134 3111 ALJAWWALSTC-AS, SA 8151 3645 3046 Uninet S.A. de C.V., MX 17974 2778 2702 TELKOMNET-AS2-AP PT Telekomunikasi Indonesia, I 11830 2647 2581 Instituto Costarricense de Electricidad y Telec 8551 2493 2451 BEZEQ-INTERNATIONAL-AS Bezeqint Internet Backbo 7552 2356 2301 VIETEL-AS-AP Viettel Group, VN Complete listing at http://thyme.rand.apnic.net/current/data-CIDRnet List of Unregistered Origin ASNs (Global) ----------------------------------------- Bad AS Designation Network Transit AS Description 65602 UNALLOCATED 8.225.192.0/24 30372 SUNESYS2 - Sunesys, LLC, US 64710 PRIVATE 39.134.26.0/23 56042 CMNET-SHANXI-AP China Mobile c 64710 PRIVATE 39.134.28.0/23 56042 CMNET-SHANXI-AP China Mobile c 64710 PRIVATE 39.134.30.0/24 56042 CMNET-SHANXI-AP China Mobile c 64710 PRIVATE 39.134.31.0/24 56042 CMNET-SHANXI-AP China Mobile c 64710 PRIVATE 39.137.20.0/23 56042 CMNET-SHANXI-AP China Mobile c 64710 PRIVATE 39.137.22.0/24 56042 CMNET-SHANXI-AP China Mobile c 65001 PRIVATE 46.237.40.0/21 13118 ASN-YARTELECOM PJSC Rostelecom 64513 PRIVATE 62.150.101.0/24 9155 QNET Kuwait, KW 419333 UNALLOCATED 84.17.91.0/24 41933 IPRAGAZ-AS Istanbul/ TURKEY, T Complete listing at http://thyme.rand.apnic.net/current/data-badAS Advertised Unallocated Addresses -------------------------------- Network Origin AS Description 14.128.14.0/23 10247 NETLINE, ZA 14.192.50.0/23 10247 NETLINE, ZA 14.192.58.0/23 10247 NETLINE, ZA 27.100.7.0/24 56096 UNKNOWN 36.255.250.0/23 10247 NETLINE, ZA 41.78.180.0/23 37265 -Reserved AS-, ZZ 43.228.144.0/22 9829 BSNL-NIB National Internet Backbone, IN 43.251.20.0/22 9381 WTT-AS-AP WTT HK Limited, HK 43.251.84.0/22 133676 PNPL-AS Precious netcom pvt ltd, IN 43.251.84.0/23 133676 PNPL-AS Precious netcom pvt ltd, IN Complete listing at http://thyme.rand.apnic.net/current/data-add-IANA Number of prefixes announced per prefix length (Global) ------------------------------------------------------- /1:0 /2:0 /3:0 /4:0 /5:0 /6:0 /7:0 /8:15 /9:13 /10:36 /11:106 /12:285 /13:550 /14:1063 /15:1858 /16:13342 /17:7757 /18:13622 /19:25209 /20:38934 /21:43959 /22:81721 /23:66497 /24:374918 /25:838 /26:621 /27:657 /28:36 /29:21 /30:17 /31:4 /32:60 Advertised prefixes smaller than registry allocations ----------------------------------------------------- ASN No of nets Total ann. Description 6327 3132 3340 SHAW - Shaw Communications Inc., CA 39891 2562 3134 ALJAWWALSTC-AS, SA 22773 2546 3303 ASN-CXA-ALL-CCI-22773-RDC - Cox Communications 18566 2066 2171 MEGAPATH5-US - MegaPath Corporation, US 11830 1991 2647 Instituto Costarricense de Electricidad y Telec 8551 1957 2493 BEZEQ-INTERNATIONAL-AS Bezeqint Internet Backbo 30036 1625 1858 MEDIACOM-ENTERPRISE-BUSINESS - Mediacom Communi 10620 1550 3557 Telmex Colombia S.A., CO 11492 1499 1686 CABLEONE - CABLE ONE, INC., US 34984 1367 2059 TELLCOM-AS, TR Complete listing at http://thyme.rand.apnic.net/current/data-sXXas-nos Number of /24s announced per /8 block (Global) ---------------------------------------------- 1:1569 2:818 4:18 5:2605 6:38 8:1078 9:1 12:1849 13:181 14:1741 15:29 16:3 17:185 18:76 20:53 21:1 23:2454 24:1926 25:2 27:2349 31:1961 32:87 35:21 36:449 37:2442 38:1440 39:88 40:100 41:3031 42:524 43:1949 44:83 45:3361 46:2853 47:186 49:1369 50:1040 51:34 52:903 54:354 55:6 56:4 57:42 58:1573 59:996 60:373 61:1997 62:1782 63:1838 64:4605 65:2241 66:4503 67:2293 68:1193 69:3173 70:1134 71:588 72:2096 74:2676 75:390 76:424 77:1545 78:1560 79:1887 80:1442 81:1411 82:1067 83:754 84:998 85:1934 86:484 87:1229 88:904 89:2300 90:174 91:6246 92:1032 93:2093 94:2411 95:2704 96:666 97:351 98:950 99:103 100:153 101:1477 102:1 103:16277 104:3191 105:209 106:517 107:1776 108:814 109:2935 110:1509 111:1730 112:1270 113:1372 114:1071 115:1605 116:1693 117:1625 118:2162 119:1650 120:732 121:1059 122:2401 123:1910 124:1477 125:1825 128:946 129:567 130:442 131:1551 132:599 133:194 134:957 135:228 136:445 137:467 138:1956 139:547 140:381 141:677 142:761 143:947 144:795 145:193 146:1126 147:695 148:1544 149:621 150:711 151:1005 152:731 153:303 154:957 155:740 156:740 157:754 158:593 159:1616 160:817 161:733 162:2541 163:531 164:939 165:1476 166:392 167:1338 168:3121 169:817 170:3553 171:401 172:1018 173:1893 174:836 175:766 176:1887 177:4014 178:2499 179:1148 180:2235 181:2109 182:2403 183:813 184:887 185:11618 186:3356 187:2339 188:2591 189:1975 190:8190 191:1339 192:9701 193:5866 194:4759 195:3972 196:2261 197:1425 198:5551 199:5887 200:7201 201:4854 202:10422 203:9977 204:4551 205:2866 206:3072 207:3094 208:3980 209:3874 210:3975 211:2076 212:2893 213:2599 214:868 215:65 216:5729 217:2098 218:889 219:628 220:1681 221:938 222:734 223:1204 End of report

1 0

Fibertech / Lightower Contact?
by Bryon Adams 01 Dec '17

01 Dec '17

Anyone know if they're having trouble? I can't seem to get through to their NOC using a pair of toll frees I have or any of the individual contacts we have numbers for. If someone from there can contact me off list, I just need some info for a new turnup (vlan, port on CPE, etc). -- Bryon Adams bryonadams(a)fastmail.com

1 0

CenturyLink (former MFON) OSP Contact
by Mike Hammett 01 Dec '17

01 Dec '17

I'm looking for a contact at CenturyLink knowledgeable on one of their routes that originally was built by Midwest Fiber Optic Networks (was a part of LightCore's network). I got conflicting stories from a sales engineer as to the status on the route. We may have better options elsewhere, but I wanted to make sure I explored all options thoroughly. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com

1 0

Contacting AS6589 - "Beneficial Technologies"
by Carlos M. Martinez 01 Dec '17

01 Dec '17

Hello all, I’m trying to reach anyone at AS 6589, “Beneficial Technologies”. They are announcing large chunk of LACNIC unallocated space, as can be seen here: https://bgp.he.net/AS6589 Although I usually give people the benefit of doubt, in this case we are talking about 5 /16 prefixes. Talk about fat fingers. Private email is ok. Thanks Carlos LACNIC CTO

3 2

Update: Packet Loss through Level 3 in Southern California?
by Gregorio Focaccio 01 Dec '17

01 Dec '17

Hi All, We have continued our investigation and data seem to show a more focused issue at the peering between Level 3 (CenturyLink) and MSN in LA. Can someone look at our data (new data below) and see if this seems like a reasonable conclusion? >From our network (San Diego) to an MSN Azure sever through the "Cogent | MSN Portal in LA" has no loss, but the same server has loss when going through the "Level 3 | MSN Portal in LA" Also, we have data showing no packet loss through the same Level 3 LA (LosAngeles1) nodes to Cogent to Texas, so data seems to clear the Level 3 Los Angeles and Cogent peering. Thanks, Greg Testing data: IPERF UDP results looks good on path through Level 3 (CenturyLink) in LA (LosAngeles1) when going to a server in Texas (non-MSN Azure) gfxc0.localdomain (0.0.0.0) Thu Nov 30 16:16:02 2017 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. 216.75.40.1 0.0% 86 0.3 7.1 0.3 172.4 27.4 2. xe-8-3-3.bar1.SanDiego1.Level3.net 0.0% 86 3.3 3.6 3.3 25.4 2.4 3. ae-3-3.ebr1.LosAngeles1.Level3.net 90.6% 86 3.6 3.6 3.6 3.7 0.0 4. ae-1-51.ear2.LosAngeles1.Level3.net 95.3% 86 7225. 7157. 7134. 7225. 45.4 5. Cogent-level3-100G.LosAngeles1.Level3.net 0.0% 86 3.8 3.9 3.6 5.7 0.2 6. be3360.ccr42.lax01.atlas.cogentco.com 0.0% 86 3.6 3.7 3.5 4.0 0.0 7. be2932.ccr32.phx01.atlas.cogentco.com 0.0% 86 12.6 12.6 12.4 13.0 0.0 8. be2930.ccr21.elp01.atlas.cogentco.com 0.0% 85 20.7 20.9 20.6 22.5 0.2 9. be2928.ccr42.iah01.atlas.cogentco.com 0.0% 85 36.8 36.6 36.4 37.1 0.0 10. be2443.ccr32.dfw01.atlas.cogentco.com 0.0% 85 41.6 41.7 41.5 42.6 0.0 11. be2939.rcr21.dfw04.atlas.cogentco.com 0.0% 85 42.6 42.7 42.3 44.0 0.2 12. te0-0-1-1.nr12.b028597-0.dfw04.atlas.cogentco.com 0.0% 85 43.2 43.2 43.1 43.5 0.0 13. 38.122.200.202 0.0% 85 42.4 42.4 42.3 42.7 0.0 14. 138.128.243.167 0.0% 85 42.6 42.7 42.4 48.2 0.7 [root@gfxc0 ~]# iperf3 -uZVc 138.128.243.167 -b10m -t15 --get-server-output iperf 3.1.3 Linux gfxc0.localdomain 4.12.5-300.fc26.x86_64 #1 SMP Mon Aug 7 15:27:25 UTC 2017 x86_64 Time: Fri, 01 Dec 2017 00:17:19 GMT Connecting to host 138.128.243.167, port 5201 Cookie: gfxc0.localdomain.1512087439.341597. [ 4] local 216.75.40.2 port 42277 connected to 138.128.243.167 port 5201 Starting Test: protocol: UDP, 1 streams, 8192 byte blocks, omitting 0 seconds, 15 second test [ ID] Interval Transfer Bandwidth Total Datagrams [ 4] 0.00-1.00 sec 1.09 MBytes 9.11 Mbits/sec 139 [ 4] 1.00-2.00 sec 1.20 MBytes 10.0 Mbits/sec 153 [ 4] 2.00-3.00 sec 1.19 MBytes 9.96 Mbits/sec 152 [ 4] 3.00-4.00 sec 1.20 MBytes 10.0 Mbits/sec 153 [ 4] 4.00-5.00 sec 1.19 MBytes 9.96 Mbits/sec 152 [ 4] 5.00-6.00 sec 1.20 MBytes 10.0 Mbits/sec 153 [ 4] 6.00-7.00 sec 1.19 MBytes 9.96 Mbits/sec 152 [ 4] 7.00-8.00 sec 1.20 MBytes 10.0 Mbits/sec 153 [ 4] 8.00-9.00 sec 1.20 MBytes 10.0 Mbits/sec 153 [ 4] 9.00-10.00 sec 1.19 MBytes 9.96 Mbits/sec 152 [ 4] 10.00-11.00 sec 1.20 MBytes 10.0 Mbits/sec 153 [ 4] 11.00-12.00 sec 1.19 MBytes 9.96 Mbits/sec 152 [ 4] 12.00-13.00 sec 1.20 MBytes 10.0 Mbits/sec 153 [ 4] 13.00-14.00 sec 1.20 MBytes 10.0 Mbits/sec 153 [ 4] 14.00-15.00 sec 1.19 MBytes 9.96 Mbits/sec 152 - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 4] 0.00-15.00 sec 17.8 MBytes 9.94 Mbits/sec 0.057 ms 0/2274 (0%) [ 4] Sent 2274 datagrams CPU Utilization: local/sender 0.7% (0.1%u/0.6%s), remote/receiver 0.1% (0.1%u/0.0%s) Server output: Accepted connection from 216.75.40.2, port 43300 [ 5] local 138.128.243.167 port 5201 connected to 216.75.40.2 port 42277 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-1.00 sec 1.08 MBytes 9.04 Mbits/sec 0.068 ms 0/138 (0%) [ 5] 1.00-2.00 sec 1.20 MBytes 10.0 Mbits/sec 0.060 ms 0/153 (0%) [ 5] 2.00-3.00 sec 1.19 MBytes 9.96 Mbits/sec 0.058 ms 0/152 (0%) [ 5] 3.00-4.00 sec 1.20 MBytes 10.0 Mbits/sec 0.058 ms 0/153 (0%) [ 5] 4.00-5.00 sec 1.19 MBytes 9.96 Mbits/sec 0.061 ms 0/152 (0%) [ 5] 5.00-6.00 sec 1.20 MBytes 10.0 Mbits/sec 0.121 ms 0/153 (0%) [ 5] 6.00-7.00 sec 1.19 MBytes 9.96 Mbits/sec 0.057 ms 0/152 (0%) [ 5] 7.00-8.00 sec 1.20 MBytes 10.0 Mbits/sec 0.056 ms 0/153 (0%) [ 5] 8.00-9.00 sec 1.20 MBytes 10.0 Mbits/sec 0.106 ms 0/153 (0%) [ 5] 9.00-10.00 sec 1.19 MBytes 9.96 Mbits/sec 0.083 ms 0/152 (0%) [ 5] 10.00-11.00 sec 1.20 MBytes 9.99 Mbits/sec 0.069 ms 0/153 (0%) [ 5] 11.00-12.00 sec 1.19 MBytes 10.0 Mbits/sec 0.054 ms 0/152 (0%) [ 5] 12.00-13.00 sec 1.20 MBytes 10.0 Mbits/sec 0.061 ms 0/153 (0%) [ 5] 13.00-14.00 sec 1.20 MBytes 10.0 Mbits/sec 0.063 ms 0/153 (0%) [ 5] 14.00-15.00 sec 1.19 MBytes 9.96 Mbits/sec 0.057 ms 0/152 (0%) iperf Done. [root@gfxc0 ~]# Results still look NOT OK when going through same Level 3 (CenturyLink) LA (LosAngeles1) nodes ebr1 and ear2 to an MSN Azure server My traceroute [v0.87] gfxc0.localdomain (0.0.0.0) Thu Nov 30 16:24:54 2017 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. 216.75.40.1 0.0% 95 0.3 10.9 0.2 199.5 38.9 2. xe-8-3-3.bar1.SanDiego1.Level3.net 0.0% 95 3.3 3.7 3.2 20.6 1.9 3. ae-3-3.ebr1.LosAngeles1.Level3.net 98.9% 95 4.0 4.0 4.0 4.0 0.0 4. ae-1-51.ear2.LosAngeles1.Level3.net 96.8% 95 7101. 7046. 7004. 7101. 50.0 5. Microsoft-level3-20G.LosAngeles1.Level3.net 0.0% 95 15.2 17.7 9.3 20.8 2.4 6. be-63-0.ibr01.lax03.ntwk.msn.net 1.1% 95 29.9 30.1 22.7 34.0 2.6 7. be-4-0.ibr01.by2.ntwk.msn.net 1.1% 95 31.6 30.2 20.4 33.9 2.6 8. 104.44.7.198 0.0% 94 29.8 29.9 20.3 33.7 2.7 9. ae102-0.icr02.by21.ntwk.msn.net 0.0% 94 35.7 29.2 20.5 41.4 2.7 10. ??? [root@gfxc0 ~]# iperf3 -uZVc 13.91.55.110 -b10m -t15 --get-server-output iperf 3.1.3 Linux gfxc0.localdomain 4.12.5-300.fc26.x86_64 #1 SMP Mon Aug 7 15:27:25 UTC 2017 x86_64 Time: Fri, 01 Dec 2017 00:25:50 GMT Connecting to host 13.91.55.110, port 5201 Cookie: gfxc0.localdomain.1512087950.883346. [ 4] local 216.75.40.2 port 34611 connected to 13.91.55.110 port 5201 Starting Test: protocol: UDP, 1 streams, 8192 byte blocks, omitting 0 seconds, 15 second test [ ID] Interval Transfer Bandwidth Total Datagrams [ 4] 0.00-1.00 sec 1.09 MBytes 9.11 Mbits/sec 139 [ 4] 1.00-2.00 sec 1.20 MBytes 10.0 Mbits/sec 153 [ 4] 2.00-3.00 sec 1.19 MBytes 9.96 Mbits/sec 152 [ 4] 3.00-4.00 sec 1.20 MBytes 10.0 Mbits/sec 153 [ 4] 4.00-5.00 sec 1.19 MBytes 9.96 Mbits/sec 152 [ 4] 5.00-6.00 sec 1.20 MBytes 10.0 Mbits/sec 153 [ 4] 6.00-7.00 sec 1.19 MBytes 9.96 Mbits/sec 152 [ 4] 7.00-8.00 sec 1.20 MBytes 10.0 Mbits/sec 153 [ 4] 8.00-9.00 sec 1.20 MBytes 10.0 Mbits/sec 153 [ 4] 9.00-10.00 sec 1.19 MBytes 9.96 Mbits/sec 152 [ 4] 10.00-11.00 sec 1.20 MBytes 10.0 Mbits/sec 153 [ 4] 11.00-12.00 sec 1.19 MBytes 9.96 Mbits/sec 152 [ 4] 12.00-13.00 sec 1.20 MBytes 10.0 Mbits/sec 153 [ 4] 13.00-14.00 sec 1.20 MBytes 10.0 Mbits/sec 153 [ 4] 14.00-15.00 sec 1.19 MBytes 9.96 Mbits/sec 152 - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 4] 0.00-15.00 sec 17.8 MBytes 9.94 Mbits/sec 0.141 ms 102/2274 (4.5%) [ 4] Sent 2274 datagrams CPU Utilization: local/sender 0.7% (0.1%u/0.6%s), remote/receiver 0.1% (0.0%u/0.1%s) Server output: ----------------------------------------------------------- Accepted connection from 216.75.40.2, port 44374 [ 5] local 10.0.0.4 port 5201 connected to 216.75.40.2 port 34611 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-1.00 sec 1.08 MBytes 9.04 Mbits/sec 0.174 ms 0/138 (0%) [ 5] 1.00-2.00 sec 1.20 MBytes 10.0 Mbits/sec 0.203 ms 0/153 (0%) [ 5] 2.00-3.00 sec 1.17 MBytes 9.83 Mbits/sec 0.196 ms 2/152 (1.3%) [ 5] 3.00-4.00 sec 1.20 MBytes 10.0 Mbits/sec 0.110 ms 0/153 (0%) [ 5] 4.00-5.00 sec 1.16 MBytes 9.69 Mbits/sec 0.120 ms 4/152 (2.6%) [ 5] 5.00-6.00 sec 1.11 MBytes 9.31 Mbits/sec 0.106 ms 11/153 (7.2%) [ 5] 6.00-7.00 sec 1.13 MBytes 9.50 Mbits/sec 0.313 ms 7/152 (4.6%) [ 5] 7.00-8.00 sec 992 KBytes 8.13 Mbits/sec 0.163 ms 29/153 (19%) [ 5] 8.00-9.00 sec 1.06 MBytes 8.91 Mbits/sec 0.129 ms 17/153 (11%) [ 5] 9.00-10.00 sec 1.12 MBytes 9.37 Mbits/sec 0.131 ms 9/152 (5.9%) [ 5] 10.00-11.00 sec 1.19 MBytes 9.96 Mbits/sec 0.155 ms 1/153 (0.65%) [ 5] 11.00-12.00 sec 1.12 MBytes 9.44 Mbits/sec 0.336 ms 5/149 (3.4%) [ 5] 12.00-13.00 sec 1.09 MBytes 9.18 Mbits/sec 0.166 ms 16/156 (10%) [ 5] 13.00-14.00 sec 1.20 MBytes 10.0 Mbits/sec 0.109 ms 0/153 (0%) [ 5] 14.00-15.00 sec 1.18 MBytes 9.90 Mbits/sec 0.141 ms 1/152 (0.66%) iperf Done. [root@gfxc0 ~]# "tcp slowdown" [root@gfxc0 ~]# iperf3 -ZVc 13.91.55.110 -b20m -t15 --get-server-output iperf 3.1.3 Linux gfxc0.localdomain 4.12.5-300.fc26.x86_64 #1 SMP Mon Aug 7 15:27:25 UTC 2017 x86_64 Time: Fri, 01 Dec 2017 00:30:16 GMT Connecting to host 13.91.55.110, port 5201 Cookie: gfxc0.localdomain.1512088216.384940. TCP MSS: 1428 (default) [ 4] local 216.75.40.2 port 44388 connected to 13.91.55.110 port 5201 Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 15 second test [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 2.33 MBytes 19.6 Mbits/sec 13 411 KBytes [ 4] 1.00-2.00 sec 2.38 MBytes 19.9 Mbits/sec 13 201 KBytes [ 4] 2.00-3.00 sec 2.38 MBytes 19.9 Mbits/sec 3 105 KBytes [ 4] 3.00-4.00 sec 2.45 MBytes 20.6 Mbits/sec 3 64.1 KBytes [ 4] 4.00-5.00 sec 1.91 MBytes 16.1 Mbits/sec 10 44.6 KBytes [ 4] 5.00-6.00 sec 1.26 MBytes 10.5 Mbits/sec 8 36.3 KBytes [ 4] 6.00-7.00 sec 1.26 MBytes 10.6 Mbits/sec 6 20.9 KBytes [ 4] 7.00-8.00 sec 803 KBytes 6.58 Mbits/sec 3 22.3 KBytes [ 4] 8.00-9.00 sec 697 KBytes 5.71 Mbits/sec 3 25.1 KBytes [ 4] 9.00-10.00 sec 1.27 MBytes 10.6 Mbits/sec 0 48.8 KBytes [ 4] 10.00-11.00 sec 1.53 MBytes 12.9 Mbits/sec 4 37.7 KBytes [ 4] 11.00-12.00 sec 1.21 MBytes 10.1 Mbits/sec 5 33.5 KBytes [ 4] 12.00-13.00 sec 1.35 MBytes 11.3 Mbits/sec 1 40.4 KBytes [ 4] 13.00-14.00 sec 1.33 MBytes 11.1 Mbits/sec 1 51.6 KBytes [ 4] 14.00-15.00 sec 2.28 MBytes 19.1 Mbits/sec 2 53.0 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-15.00 sec 24.4 MBytes 13.6 Mbits/sec 75 sender [ 4] 0.00-15.00 sec 24.1 MBytes 13.5 Mbits/sec receiver CPU Utilization: local/sender 0.7% (0.1%u/0.6%s), remote/receiver 0.2% (0.0%u/0.2%s) Server output: ----------------------------------------------------------- Accepted connection from 216.75.40.2, port 44386 [ 5] local 10.0.0.4 port 5201 connected to 216.75.40.2 port 44388 [ ID] Interval Transfer Bandwidth [ 5] 0.00-1.00 sec 2.21 MBytes 18.5 Mbits/sec [ 5] 1.00-2.00 sec 2.38 MBytes 19.9 Mbits/sec [ 5] 2.00-3.00 sec 2.38 MBytes 19.9 Mbits/sec [ 5] 3.00-4.00 sec 2.23 MBytes 18.7 Mbits/sec [ 5] 4.00-5.00 sec 1.93 MBytes 16.2 Mbits/sec [ 5] 5.00-6.00 sec 1.26 MBytes 10.6 Mbits/sec [ 5] 6.00-7.00 sec 1.27 MBytes 10.6 Mbits/sec [ 5] 7.00-8.00 sec 800 KBytes 6.56 Mbits/sec [ 5] 8.00-9.00 sec 692 KBytes 5.67 Mbits/sec [ 5] 9.00-10.00 sec 1.27 MBytes 10.7 Mbits/sec [ 5] 10.00-11.00 sec 1.53 MBytes 12.9 Mbits/sec [ 5] 11.00-12.00 sec 1.21 MBytes 10.2 Mbits/sec [ 5] 12.00-13.00 sec 1.35 MBytes 11.3 Mbits/sec [ 5] 13.00-14.00 sec 1.32 MBytes 11.1 Mbits/sec [ 5] 14.00-15.00 sec 2.26 MBytes 18.9 Mbits/sec iperf Done. [root@gfxc0 ~]# ...but then results look OK to same Azure server when traffic forced to get to MSN via Cogent My traceroute [v0.87] gfxc0.localdomain (0.0.0.0) Thu Nov 30 16:20:50 2017 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. 216.75.40.1 0.0% 19 0.3 9.9 0.3 124.0 30.6 2. 216.98.153.90 0.0% 19 63.8 10.0 0.3 117.3 29.8 3. te0-0-2-2.agr11.san01.atlas.cogentco.com 0.0% 19 1.6 0.9 0.7 1.6 0.0 4. te0-0-0-3.rcr12.san01.atlas.cogentco.com 0.0% 19 1.0 1.0 0.8 1.2 0.0 5. be2937.rcr12.sna02.atlas.cogentco.com 0.0% 19 3.2 3.3 3.2 3.5 0.0 6. be2463.agr22.lax01.atlas.cogentco.com 0.0% 19 4.0 4.3 4.0 6.7 0.5 7. be2586.ccr41.lax01.atlas.cogentco.com 0.0% 19 4.1 4.1 4.0 4.4 0.0 8. be3271.ccr41.lax04.atlas.cogentco.com 0.0% 19 4.9 4.7 4.0 9.2 1.0 9. 38.142.33.250 0.0% 19 4.0 3.9 3.8 4.0 0.0 10. be-63-0.ibr01.lax03.ntwk.msn.net 0.0% 19 17.0 16.6 15.6 17.4 0.2 11. be-4-0.ibr01.by2.ntwk.msn.net 0.0% 19 17.2 16.9 15.9 17.9 0.0 12. 104.44.7.198 0.0% 19 16.1 17.0 16.1 17.8 0.2 13. ae103-0.icr02.by4.ntwk.msn.net 0.0% 19 15.2 15.3 15.1 17.3 0.4 14. ??? [root@gfxc0 ~]# iperf3 -uZVc 13.91.55.110 -b10m -t15 --get-server-output iperf 3.1.3 Linux gfxc0.localdomain 4.12.5-300.fc26.x86_64 #1 SMP Mon Aug 7 15:27:25 UTC 2017 x86_64 Time: Fri, 01 Dec 2017 00:21:51 GMT Connecting to host 13.91.55.110, port 5201 Cookie: gfxc0.localdomain.1512087711.332144. [ 4] local 216.75.40.2 port 43882 connected to 13.91.55.110 port 5201 Starting Test: protocol: UDP, 1 streams, 8192 byte blocks, omitting 0 seconds, 15 second test [ ID] Interval Transfer Bandwidth Total Datagrams [ 4] 0.00-1.00 sec 1.09 MBytes 9.11 Mbits/sec 139 [ 4] 1.00-2.00 sec 1.20 MBytes 10.0 Mbits/sec 153 [ 4] 2.00-3.00 sec 1.19 MBytes 9.96 Mbits/sec 152 [ 4] 3.00-4.00 sec 1.20 MBytes 10.0 Mbits/sec 153 [ 4] 4.00-5.00 sec 1.19 MBytes 9.96 Mbits/sec 152 [ 4] 5.00-6.00 sec 1.20 MBytes 10.0 Mbits/sec 153 [ 4] 6.00-7.00 sec 1.19 MBytes 9.96 Mbits/sec 152 [ 4] 7.00-8.00 sec 1.20 MBytes 10.0 Mbits/sec 153 [ 4] 8.00-9.00 sec 1.20 MBytes 10.0 Mbits/sec 153 [ 4] 9.00-10.00 sec 1.19 MBytes 9.96 Mbits/sec 152 [ 4] 10.00-11.00 sec 1.20 MBytes 10.0 Mbits/sec 153 [ 4] 11.00-12.00 sec 1.19 MBytes 9.96 Mbits/sec 152 [ 4] 12.00-13.00 sec 1.20 MBytes 10.0 Mbits/sec 153 [ 4] 13.00-14.00 sec 1.20 MBytes 10.0 Mbits/sec 153 [ 4] 14.00-15.00 sec 1.19 MBytes 9.96 Mbits/sec 152 - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 4] 0.00-15.00 sec 17.8 MBytes 9.94 Mbits/sec 0.046 ms 0/2274 (0%) [ 4] Sent 2274 datagrams CPU Utilization: local/sender 0.7% (0.1%u/0.7%s), remote/receiver 0.0% (0.0%u/0.0%s) Server output: ----------------------------------------------------------- Accepted connection from 216.75.40.2, port 44356 [ 5] local 10.0.0.4 port 5201 connected to 216.75.40.2 port 43882 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-1.00 sec 1.08 MBytes 9.04 Mbits/sec 0.036 ms 0/138 (0%) [ 5] 1.00-2.00 sec 1.20 MBytes 10.0 Mbits/sec 0.057 ms 0/153 (0%) [ 5] 2.00-3.00 sec 1.19 MBytes 9.96 Mbits/sec 0.047 ms 0/152 (0%) [ 5] 3.00-4.00 sec 1.20 MBytes 10.0 Mbits/sec 0.049 ms 0/153 (0%) [ 5] 4.00-5.00 sec 1.19 MBytes 9.96 Mbits/sec 0.064 ms 0/152 (0%) [ 5] 5.00-6.00 sec 1.20 MBytes 10.0 Mbits/sec 0.059 ms 0/153 (0%) [ 5] 6.00-7.00 sec 1.19 MBytes 9.96 Mbits/sec 0.056 ms 0/152 (0%) [ 5] 7.00-8.00 sec 1.20 MBytes 10.0 Mbits/sec 0.055 ms 0/153 (0%) [ 5] 8.00-9.00 sec 1.20 MBytes 10.0 Mbits/sec 0.061 ms 0/153 (0%) [ 5] 9.00-10.00 sec 1.19 MBytes 9.96 Mbits/sec 0.051 ms 0/152 (0%) [ 5] 10.00-11.00 sec 1.20 MBytes 10.0 Mbits/sec 0.057 ms 0/153 (0%) [ 5] 11.00-12.00 sec 1.19 MBytes 9.96 Mbits/sec 0.053 ms 0/152 (0%) [ 5] 12.00-13.00 sec 1.20 MBytes 10.0 Mbits/sec 0.055 ms 0/153 (0%) [ 5] 13.00-14.00 sec 1.20 MBytes 10.0 Mbits/sec 0.063 ms 0/153 (0%) [ 5] 14.00-15.00 sec 1.19 MBytes 9.96 Mbits/sec 0.046 ms 0/152 (0%) iperf Done. [root@gfxc0 ~]# From: Gregorio Focaccio Sent: Tuesday, November 28, 2017 2:05 PM To: 'nanog(a)nanog.org' <nanog(a)nanog.org> Subject: Packet Loss through Level 3 in Southern California? Hi All, We are a multi-datacenter MSP in San Diego that also offers Colo and Cloud hosting. We are multi-homed with Level 3 and Cogent. A client reported newly slow FTP transfers, so we started a network investigation. Our data (see below) seem to show packet loss through Level 3 with associated slow TCP based data transfers. Is anyone else seeing packet loss and consequent slow TCP based transfers when going through Level 3 in Southern California? Thanks, Greg Focaccio CARI.net<https://cari.net/> Testing Data: ********** Internal tests - OK ********** FTP transfers from client server to another server 3 hops away in our adjacent datacenter was normal ********** Level 3 data - packet loss ********** External tests via Level 3 - 12% Packet Loss - see data below UDP IPERF testing from our data center (through Level 3 and Microsoft - trace below) to an Azure server showed repeatable packet loss TCP based testing - such as FTP or SCP transfers - the rate was very slow about 4Mbps [root@raynor ~]# iperf3 -c 40.80.156.2 -b 10000m Connecting to host 40.80.156.2, port 5201 [ 4] local 71.6.220.101 port 55684 connected to 40.80.156.2 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 1001 KBytes 8.20 Mbits/sec 34 9.76 KBytes [ 4] 1.00-2.00 sec 503 KBytes 4.12 Mbits/sec 12 11.2 KBytes [ 4] 2.00-3.00 sec 502 KBytes 4.11 Mbits/sec 3 25.1 KBytes [ 4] 3.00-4.00 sec 502 KBytes 4.11 Mbits/sec 12 11.2 KBytes [ 4] 4.00-5.00 sec 377 KBytes 3.08 Mbits/sec 9 15.3 KBytes [ 4] 5.00-6.00 sec 502 KBytes 4.11 Mbits/sec 10 19.5 KBytes [ 4] 6.00-7.00 sec 377 KBytes 3.09 Mbits/sec 13 6.97 KBytes [ 4] 7.00-8.00 sec 251 KBytes 2.06 Mbits/sec 9 5.58 KBytes [ 4] 8.00-9.00 sec 251 KBytes 2.06 Mbits/sec 6 9.76 KBytes [ 4] 9.00-10.00 sec 251 KBytes 2.06 Mbits/sec 10 12.6 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 4.41 MBytes 3.70 Mbits/sec 118 sender [ 4] 0.00-10.00 sec 4.24 MBytes 3.56 Mbits/sec receiver iperf Done. [root@raynor ~]# iperf3 -u -c 40.80.156.2 -b 10000m Connecting to host 40.80.156.2, port 5201 [ 4] local 71.6.220.101 port 39221 connected to 40.80.156.2 port 5201 [ ID] Interval Transfer Bandwidth Total Datagrams [ 4] 0.00-1.00 sec 85.0 MBytes 712 Mbits/sec 62402 [ 4] 1.00-2.00 sec 94.0 MBytes 789 Mbits/sec 69043 [ 4] 2.00-3.00 sec 92.6 MBytes 777 Mbits/sec 68030 [ 4] 3.00-4.00 sec 76.5 MBytes 641 Mbits/sec 56153 [ 4] 4.00-5.00 sec 94.9 MBytes 796 Mbits/sec 69662 [ 4] 5.00-6.00 sec 97.7 MBytes 819 Mbits/sec 71713 [ 4] 6.00-7.00 sec 98.5 MBytes 826 Mbits/sec 72347 [ 4] 7.00-8.00 sec 92.7 MBytes 778 Mbits/sec 68085 [ 4] 8.00-9.00 sec 91.3 MBytes 765 Mbits/sec 67045 [ 4] 9.00-10.00 sec 59.3 MBytes 498 Mbits/sec 43551 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 4] 0.00-10.00 sec 883 MBytes 740 Mbits/sec 0.014 ms 75647/647955 (12%) [ 4] Sent 647955 datagrams iperf Done. [root@raynor ~]# traceroute 40.80.156.2 traceroute to 40.80.156.2 (40.80.156.2), 30 hops max, 60 byte packets 1 gateway (71.6.220.97) 6.170 ms 7.827 ms 10.718 ms 2 209.126.134.65 (209.126.134.65) 0.426 ms 0.420 ms 0.543 ms 3 216.98.153.21 (216.98.153.21) 159.936 ms 160.215 ms 160.209 ms 4 gi1-2.gw65-50.c5.sdcix.net (216.98.153.89) 170.977 ms 170.972 ms 170.964 ms 5 xe-8-3-3.bar1.SanDiego1.Level3.net (4.16.105.93) 0.547 ms 0.586 ms 0.533 ms 6 * * * 7 * * * 8 Microsoft-level3-20G.LosAngeles1.Level3.net (4.68.111.122) 17.296 ms 18.026 ms 20.200 ms 9 be-61-0.ibr01.lax03.ntwk.msn.net (104.44.8.104) 30.108 ms 32.300 ms 32.283 ms 10 be-4-0.ibr01.by2.ntwk.msn.net (104.44.4.3) 30.364 ms 30.961 ms 28.960 ms 11 104.44.7.198 (104.44.7.198) 31.870 ms 30.120 ms 29.636 ms 12 ae100-0.icr01.by21.ntwk.msn.net (104.44.11.194) 27.828 ms ae101-0.icr01.by4.ntwk.msn.net (104.44.11.193) 29.611 ms ae100-0.icr01.by21.ntwk.msn.net (104.44.11.194) 27.710 ms 13 * * * [root@raynor ~]# scp ubuntu-14.04.5-desktop-amd64.iso carinet@40.80.156.2:/home/carinet/ubuntunew11.iso<mailto:carinet@40.80.156.2:/home/carinet/ubuntunew11.iso> ubuntu-14.04.5-desktop-amd64.iso 100% 1053MB 730.8KB/s 24:35 LEVEL3 - summary CARIcloud to Azure 2 Mbits/s TCP iperf 800 Mbits/s UDP iperf 24m and 35s 1053 MB upload to Azure through SCP ********** Cogent data - OK ********** External tests via Cogent - OK - No significant loss - see IPERF and trace data below [root@raynor ~]# iperf3 -c 40.80.156.2 -b 10000m Connecting to host 40.80.156.2, port 5201 [ 4] local 71.6.220.101 port 45076 connected to 40.80.156.2 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 149 MBytes 1.25 Gbits/sec 0 4.58 MBytes [ 4] 1.00-2.00 sec 180 MBytes 1.51 Gbits/sec 0 4.58 MBytes [ 4] 2.00-3.00 sec 178 MBytes 1.50 Gbits/sec 0 4.58 MBytes [ 4] 3.00-4.00 sec 180 MBytes 1.51 Gbits/sec 0 4.58 MBytes [ 4] 4.00-5.00 sec 180 MBytes 1.51 Gbits/sec 0 4.58 MBytes [ 4] 5.00-6.00 sec 180 MBytes 1.51 Gbits/sec 0 4.58 MBytes [ 4] 6.00-7.00 sec 179 MBytes 1.50 Gbits/sec 0 4.58 MBytes [ 4] 7.00-8.00 sec 180 MBytes 1.51 Gbits/sec 0 4.58 MBytes [ 4] 8.00-9.00 sec 181 MBytes 1.52 Gbits/sec 0 4.58 MBytes [ 4] 9.00-10.00 sec 180 MBytes 1.51 Gbits/sec 0 4.58 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 1.73 GBytes 1.48 Gbits/sec 0 sender [ 4] 0.00-10.00 sec 1.73 GBytes 1.48 Gbits/sec receiver iperf Done. [root@raynor ~]# iperf3 -u -c 40.80.156.2 -b 10000m Connecting to host 40.80.156.2, port 5201 [ 4] local 71.6.220.101 port 39130 connected to 40.80.156.2 port 5201 [ ID] Interval Transfer Bandwidth Total Datagrams [ 4] 0.00-1.00 sec 101 MBytes 844 Mbits/sec 73869 [ 4] 1.00-2.00 sec 103 MBytes 861 Mbits/sec 75344 [ 4] 2.00-3.00 sec 99.0 MBytes 830 Mbits/sec 72700 [ 4] 3.00-4.00 sec 97.9 MBytes 821 Mbits/sec 71885 [ 4] 4.00-5.00 sec 98.0 MBytes 822 Mbits/sec 71979 [ 4] 5.00-6.00 sec 100 MBytes 841 Mbits/sec 73640 [ 4] 6.00-7.00 sec 69.8 MBytes 585 Mbits/sec 51237 [ 4] 7.00-8.00 sec 94.9 MBytes 796 Mbits/sec 69650 [ 4] 8.00-9.00 sec 102 MBytes 855 Mbits/sec 74804 [ 4] 9.00-10.00 sec 100 MBytes 839 Mbits/sec 73407 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 4] 0.00-10.00 sec 965 MBytes 809 Mbits/sec 0.004 ms 3194/708505 (0.45%) [ 4] Sent 708505 datagrams iperf Done. [root@raynor ~]# traceroute 40.80.156.2 traceroute to 40.80.156.2 (40.80.156.2), 30 hops max, 60 byte packets 1 gateway (71.6.220.97) 9.892 ms 10.883 ms 15.574 ms 2 209.126.134.65 (209.126.134.65) 1.541 ms 1.966 ms 1.959 ms 3 216.98.153.21 (216.98.153.21) 0.523 ms 0.517 ms 0.511 ms 4 216.98.153.114 (216.98.153.114) 0.589 ms 0.480 ms 0.687 ms 5 te0-0-1-0.nr11.b005949-0.san01.atlas.cogentco.com (38.104.122.61) 1.045 ms 1.039 ms 1.072 ms 6 te0-0-0-3.agr11.san01.atlas.cogentco.com (154.24.32.53) 1.023 ms 0.823 ms te0-0-0-3.agr12.san01.atlas.cogentco.com (154.24.32.65) 1.359 ms 7 te0-0-1-3.rcr11.san01.atlas.cogentco.com (154.24.31.21) 0.919 ms te0-0-1-3.rcr12.san01.atlas.cogentco.com (154.24.31.37) 0.834 ms 0.874 ms 8 be2936.rcr11.sna02.atlas.cogentco.com (154.54.45.166) 3.947 ms 3.726 ms be2937.rcr12.sna02.atlas.cogentco.com (154.54.45.173) 3.221 ms 9 be2463.agr22.lax01.atlas.cogentco.com (154.54.80.61) 4.095 ms 3.960 ms 4.087 ms 10 be2584.ccr41.lax01.atlas.cogentco.com (154.54.29.33) 4.364 ms 3.961 ms be2586.ccr41.lax01.atlas.cogentco.com (154.54.29.245) 4.290 ms 11 be3360.ccr41.lax04.atlas.cogentco.com (154.54.25.150) 4.283 ms be3271.ccr41.lax04.atlas.cogentco.com (154.54.42.102) 4.020 ms 4.003 ms 12 38.142.33.250 (38.142.33.250) 3.828 ms 3.818 ms 3.481 ms 13 be-61-0.ibr01.lax03.ntwk.msn.net (104.44.8.104) 15.565 ms 15.291 ms 15.173 ms 14 be-4-0.ibr01.by2.ntwk.msn.net (104.44.4.3) 14.290 ms 13.903 ms 16.168 ms 15 104.44.7.198 (104.44.7.198) 15.445 ms 14.098 ms 13.866 ms 16 ae102-0.icr02.by21.ntwk.msn.net (104.44.11.198) 13.752 ms ae100-0.icr01.by21.ntwk.msn.net (104.44.11.194) 14.378 ms ae101-0.icr01.by4.ntwk.msn.net (104.44.11.193) 13.813 ms [root@raynor ~]# scp ubuntu-14.04.5-desktop-amd64.iso carinet@40.80.156.2:/home/carinet/ubuntunew10.iso<mailto:carinet@40.80.156.2:/home/carinet/ubuntunew10.iso> ubuntu-14.04.5-desktop-amd64.iso 100% 1053MB 111.5MB/s 00:09 COGENT - summary CARIcloud to Azure 1.5 Gbits/s TCP iperf 850 Mbits/s UDP iperf 9s 1053 MB upload to Azure through SCP

1 0