Verizon: Extremely Strange CPE Routing in NYC/NJ Area
Can anyone from Verizon take a look at this behavior for us? We’re having multiple Verizon FiOS users in the NYC/NJ area appear to teleport from their FiOS router to our IP in the Pittsburgh region. Users are seeing extreme slowness with TCP traffic, but ping times seem reasonable. User 1: 1 fios_quantum_gateway (192.168.1.1) 1.575 ms 2.426 ms 3.193 ms 2 204.16.244.8 (204.16.244.8) 2.269 ms 3.055 ms 2.727 ms User 2: 1 fios_quantum_gateway (192.168.1.1) 1.565 ms 1.048 ms 0.947 ms 2 204.16.244.8 (204.16.244.8) 2.162 ms 3.588 ms 3.048 ms I can provide end-user NYC/NJ IPs off-list if desirable. Here's a normal looking trace from an FiOS line locally in the Pittsburgh region: IP: 108.39.229.34 Tracing route to four.libsyn.com [204.16.244.8] over a maximum of 30 hops: 1 <1 ms <1 ms <1 ms 192.168.1.1 2 5 ms 2 ms 7 ms lo0-100.PITBPA-VFTTP-301.verizon-gni.net <http://lo0-100.pitbpa-vfttp-301.verizon-gni.net/> [108.39.229.1] 3 5 ms 6 ms 6 ms B3301.PITBPA-LCR-22.verizon-gni.net <http://b3301.pitbpa-lcr-22.verizon-gni.net/> [100.41.223.244] 4 * * * Request timed out. 5 * * * Request timed out. 6 13 ms 12 ms 13 ms 0.et-7-1-5.BR1.IAD8.ALTER.NET <http://0.et-7-1-5.br1.iad8.alter.net/> [140.222.226.17] 7 10 ms 10 ms 10 ms verizon.com.customer.alter.net [152.179.50.110] 8 12 ms 12 ms 13 ms be3084.ccr42.dca01.atlas.cogentco.com [154.54.30.65] 9 22 ms 22 ms 22 ms be2820.rcr21.pit02.atlas.cogentco.com [154.54.83.54] 10 22 ms 22 ms 21 ms 38.104.120.90 11 26 ms 21 ms 19 ms 204.16.241.133 12 * * * Request timed out. 13 21 ms 21 ms 21 ms 204.16.244.8 Is this a possible traffic engineering blip? I can’t say we’ve ever seen trace routes return such sparse results and actually make it to the destination. -- Nick Zurku Systems Engineer TeraSwitch, Inc. Cell: 412-953-0481 Office: 412-945-7048 nzurku@teraswitch.com
On 11/29/18, Nick Zurku <nzurku@teraswitch.com> wrote:
Can anyone from Verizon take a look at this behavior for us?
We’re having multiple Verizon FiOS users in the NYC/NJ area appear to teleport from their FiOS router to our IP in the Pittsburgh region.
Verizon is doing something seriously weird to windows traceroute: C:\Users\Lee>tracert www.yahoo.com Tracing route to atsv2-fp-shed.wg1.b.yahoo.com [98.138.219.232] over a maximum of 30 hops: 1 <1 ms <1 ms <1 ms fw.home.net 2 1 ms <1 ms <1 ms vbz-router.home.net [192.168.1.1] 3 8 ms 3 ms 6 ms media-router-fp2.prod1.media.vip.ne1.yahoo.com [98.138.219.232] Trace complete. C:\Users\Lee>ping -i 12 www.yahoo.com. Pinging atsv2-fp-shed.wg1.b.yahoo.com [98.138.219.232] with 32 bytes of data: Reply from 98.138.0.87: TTL expired in transit. Reply from 98.138.0.87: TTL expired in transit. Reply from 98.138.0.87: TTL expired in transit. Reply from 98.138.0.87: TTL expired in transit. Ping statistics for 98.138.219.232: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), C:\Users\Lee>ping -i 13 www.yahoo.com. Pinging atsv2-fp-shed.wg1.b.yahoo.com [98.138.219.232] with 32 bytes of data: Reply from 98.138.219.232: bytes=32 time=32ms TTL=54 Reply from 98.138.219.232: bytes=32 time=31ms TTL=54 Reply from 98.138.219.232: bytes=32 time=33ms TTL=54 Reply from 98.138.219.232: bytes=32 time=33ms TTL=54 Ping statistics for 98.138.219.232: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 31ms, Maximum = 33ms, Average = 32ms C:\Users\Lee> traceroute from a linux box works as expected.. Lee
Users are seeing extreme slowness with TCP traffic, but ping times seem reasonable.
User 1: 1 fios_quantum_gateway (192.168.1.1) 1.575 ms 2.426 ms 3.193 ms 2 204.16.244.8 (204.16.244.8) 2.269 ms 3.055 ms 2.727 ms
User 2: 1 fios_quantum_gateway (192.168.1.1) 1.565 ms 1.048 ms 0.947 ms 2 204.16.244.8 (204.16.244.8) 2.162 ms 3.588 ms 3.048 ms
I can provide end-user NYC/NJ IPs off-list if desirable.
Here's a normal looking trace from an FiOS line locally in the Pittsburgh region:
IP: 108.39.229.34 Tracing route to four.libsyn.com [204.16.244.8] over a maximum of 30 hops:
1 <1 ms <1 ms <1 ms 192.168.1.1 2 5 ms 2 ms 7 ms lo0-100.PITBPA-VFTTP-301.verizon-gni.net <http://lo0-100.pitbpa-vfttp-301.verizon-gni.net/> [108.39.229.1] 3 5 ms 6 ms 6 ms B3301.PITBPA-LCR-22.verizon-gni.net <http://b3301.pitbpa-lcr-22.verizon-gni.net/> [100.41.223.244] 4 * * * Request timed out. 5 * * * Request timed out. 6 13 ms 12 ms 13 ms 0.et-7-1-5.BR1.IAD8.ALTER.NET <http://0.et-7-1-5.br1.iad8.alter.net/> [140.222.226.17] 7 10 ms 10 ms 10 ms verizon.com.customer.alter.net [152.179.50.110] 8 12 ms 12 ms 13 ms be3084.ccr42.dca01.atlas.cogentco.com [154.54.30.65] 9 22 ms 22 ms 22 ms be2820.rcr21.pit02.atlas.cogentco.com [154.54.83.54] 10 22 ms 22 ms 21 ms 38.104.120.90 11 26 ms 21 ms 19 ms 204.16.241.133 12 * * * Request timed out. 13 21 ms 21 ms 21 ms 204.16.244.8
Is this a possible traffic engineering blip? I can’t say we’ve ever seen trace routes return such sparse results and actually make it to the destination.
-- Nick Zurku Systems Engineer TeraSwitch, Inc. Cell: 412-953-0481 Office: 412-945-7048 nzurku@teraswitch.com
Windows traceroute uses ICMP by default, Unix based systems use UDP. For example from my Mac on Fios in Westchester, NY (and yes, they are doing something with ICMP traceroutes, possibly within their gateway box): acpro:Documents mhuff$ sudo traceroute www.yahoo.com traceroute: Warning: www.yahoo.com has multiple addresses; using 72.30.35.9 traceroute to atsv2-fp-shed.wg1.b.yahoo.com (72.30.35.9), 64 hops max, 52 byte packets 1 firewall (10.1.1.1) 0.916 ms 0.419 ms 0.394 ms 2 * * * 3 b3369.nycmny-lcr-22.verizon-gni.net (130.81.16.102) 3.939 ms 10.172 ms b3369.nycmny-lcr-21.verizon-gni.net (130.81.16.100) 5.192 ms 4 * * * 5 0.ae4.xl2.buf4.alter.net (140.222.228.143) 13.500 ms * 12.970 ms 6 0.xe-5-3-0.gw1.buf4.alter.net (152.63.26.62) 12.390 ms 12.122 ms 0.ae4.xl2.buf4.alter.net (140.222.228.143) 12.704 ms 7 0.xe-4-2-0.gw1.buf4.alter.net (152.63.26.46) 14.326 ms 13.069 ms verizon.com.customer.alter.net (204.148.47.162) 14.436 ms 8 et-19-0-0.pat1.bfz.yahoo.com (216.115.97.103) 13.990 ms unknown-72-30-223-x.yahoo.com (72.30.223.3) 14.220 ms et-19-0-0.pat1.bfz.yahoo.com (216.115.97.103) 13.711 ms 9 et-0-0-0.msr2.bf1.yahoo.com (74.6.227.137) 12.098 ms et-0-0-0.msr1.bf1.yahoo.com (74.6.227.129) 13.388 ms et-18-0-1.msr1.bf1.yahoo.com (74.6.227.131) 14.457 ms 10 et-19-0-0.clr2-a-gdc.bf1.yahoo.com (74.6.122.37) 14.554 ms et-0-0-0.msr1.bf1.yahoo.com (74.6.227.129) 14.810 ms et-1-1-1.msr1.bf1.yahoo.com (74.6.227.135) 13.502 ms 11 eth-18-3.bas2-1-flk.bf1.yahoo.com (98.139.128.75) 14.607 ms 16.608 ms eth-18-3-bas1-1-flk.bf1.yahoo.com (98.139.128.73) 15.161 ms 12 media-router-fp1.prod1.media.vip.bf1.yahoo.com (72.30.35.9) 13.470 ms eth-17-3.bas2-1-flk.bf1.yahoo.com (98.139.128.71) 13.582 ms media-router-fp1.prod1.media.vip.bf1.yahoo.com (72.30.35.9) 12.952 ms macpro:Documents mhuff$ sudo traceroute -I www.yahoo.com traceroute: Warning: www.yahoo.com has multiple addresses; using 72.30.35.10 traceroute to atsv2-fp-shed.wg1.b.yahoo.com (72.30.35.10), 64 hops max, 72 byte packets 1 firewall (10.1.1.1) 0.675 ms 0.347 ms 0.322 ms 2 media-router-fp2.prod1.media.vip.bf1.yahoo.com (72.30.35.10) 2.456 ms 21.139 ms 12.834 ms ---- Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Lee Sent: Thursday, November 29, 2018 5:42 PM To: Nick Zurku <nzurku@teraswitch.com> Cc: Brendan Mannella <bmannella@teraswitch.com>; nanog@nanog.org; Todd Kammerer <todd@libsyn.com> Subject: Re: Verizon: Extremely Strange CPE Routing in NYC/NJ Area On 11/29/18, Nick Zurku <nzurku@teraswitch.com> wrote:
Can anyone from Verizon take a look at this behavior for us?
We’re having multiple Verizon FiOS users in the NYC/NJ area appear to teleport from their FiOS router to our IP in the Pittsburgh region.
Verizon is doing something seriously weird to windows traceroute: C:\Users\Lee>tracert www.yahoo.com Tracing route to atsv2-fp-shed.wg1.b.yahoo.com [98.138.219.232] over a maximum of 30 hops: 1 <1 ms <1 ms <1 ms fw.home.net 2 1 ms <1 ms <1 ms vbz-router.home.net [192.168.1.1] 3 8 ms 3 ms 6 ms media-router-fp2.prod1.media.vip.ne1.yahoo.com [98.138.219.232] Trace complete. C:\Users\Lee>ping -i 12 www.yahoo.com. Pinging atsv2-fp-shed.wg1.b.yahoo.com [98.138.219.232] with 32 bytes of data: Reply from 98.138.0.87: TTL expired in transit. Reply from 98.138.0.87: TTL expired in transit. Reply from 98.138.0.87: TTL expired in transit. Reply from 98.138.0.87: TTL expired in transit. Ping statistics for 98.138.219.232: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), C:\Users\Lee>ping -i 13 www.yahoo.com. Pinging atsv2-fp-shed.wg1.b.yahoo.com [98.138.219.232] with 32 bytes of data: Reply from 98.138.219.232: bytes=32 time=32ms TTL=54 Reply from 98.138.219.232: bytes=32 time=31ms TTL=54 Reply from 98.138.219.232: bytes=32 time=33ms TTL=54 Reply from 98.138.219.232: bytes=32 time=33ms TTL=54 Ping statistics for 98.138.219.232: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 31ms, Maximum = 33ms, Average = 32ms C:\Users\Lee> traceroute from a linux box works as expected.. Lee
Users are seeing extreme slowness with TCP traffic, but ping times seem reasonable.
User 1: 1 fios_quantum_gateway (192.168.1.1) 1.575 ms 2.426 ms 3.193 ms 2 204.16.244.8 (204.16.244.8) 2.269 ms 3.055 ms 2.727 ms
User 2: 1 fios_quantum_gateway (192.168.1.1) 1.565 ms 1.048 ms 0.947 ms 2 204.16.244.8 (204.16.244.8) 2.162 ms 3.588 ms 3.048 ms
I can provide end-user NYC/NJ IPs off-list if desirable.
Here's a normal looking trace from an FiOS line locally in the Pittsburgh region:
IP: 108.39.229.34 Tracing route to four.libsyn.com [204.16.244.8] over a maximum of 30 hops:
1 <1 ms <1 ms <1 ms 192.168.1.1 2 5 ms 2 ms 7 ms lo0-100.PITBPA-VFTTP-301.verizon-gni.net <http://lo0-100.pitbpa-vfttp-301.verizon-gni.net/> [108.39.229.1] 3 5 ms 6 ms 6 ms B3301.PITBPA-LCR-22.verizon-gni.net <http://b3301.pitbpa-lcr-22.verizon-gni.net/> [100.41.223.244] 4 * * * Request timed out. 5 * * * Request timed out. 6 13 ms 12 ms 13 ms 0.et-7-1-5.BR1.IAD8.ALTER.NET <http://0.et-7-1-5.br1.iad8.alter.net/> [140.222.226.17] 7 10 ms 10 ms 10 ms verizon.com.customer.alter.net [152.179.50.110] 8 12 ms 12 ms 13 ms be3084.ccr42.dca01.atlas.cogentco.com [154.54.30.65] 9 22 ms 22 ms 22 ms be2820.rcr21.pit02.atlas.cogentco.com [154.54.83.54] 10 22 ms 22 ms 21 ms 38.104.120.90 11 26 ms 21 ms 19 ms 204.16.241.133 12 * * * Request timed out. 13 21 ms 21 ms 21 ms 204.16.244.8
Is this a possible traffic engineering blip? I can’t say we’ve ever seen trace routes return such sparse results and actually make it to the destination.
-- Nick Zurku Systems Engineer TeraSwitch, Inc. Cell: 412-953-0481 Office: 412-945-7048 nzurku@teraswitch.com
See this post for more info: http://www.dslreports.com/forum/r32136909-Has-Vz-disabled-TTL-propagation On Thu, Nov 29, 2018 at 3:09 PM Nick Zurku <nzurku@teraswitch.com> wrote:
Can anyone from Verizon take a look at this behavior for us?
We’re having multiple Verizon FiOS users in the NYC/NJ area appear to teleport from their FiOS router to our IP in the Pittsburgh region. Users are seeing extreme slowness with TCP traffic, but ping times seem reasonable.
User 1: 1 fios_quantum_gateway (192.168.1.1) 1.575 ms 2.426 ms 3.193 ms 2 204.16.244.8 (204.16.244.8) 2.269 ms 3.055 ms 2.727 ms
User 2: 1 fios_quantum_gateway (192.168.1.1) 1.565 ms 1.048 ms 0.947 ms 2 204.16.244.8 (204.16.244.8) 2.162 ms 3.588 ms 3.048 ms
I can provide end-user NYC/NJ IPs off-list if desirable.
Here's a normal looking trace from an FiOS line locally in the Pittsburgh region:
IP: 108.39.229.34 Tracing route to four.libsyn.com [204.16.244.8] over a maximum of 30 hops:
1 <1 ms <1 ms <1 ms 192.168.1.1 2 5 ms 2 ms 7 ms lo0-100.PITBPA-VFTTP-301.verizon-gni.net <http://lo0-100.pitbpa-vfttp-301.verizon-gni.net/> [108.39.229.1] 3 5 ms 6 ms 6 ms B3301.PITBPA-LCR-22.verizon-gni.net <http://b3301.pitbpa-lcr-22.verizon-gni.net/> [100.41.223.244] 4 * * * Request timed out. 5 * * * Request timed out. 6 13 ms 12 ms 13 ms 0.et-7-1-5.BR1.IAD8.ALTER.NET <http://0.et-7-1-5.br1.iad8.alter.net/> [140.222.226.17] 7 10 ms 10 ms 10 ms verizon.com.customer.alter.net [152.179.50.110] 8 12 ms 12 ms 13 ms be3084.ccr42.dca01.atlas.cogentco.com [154.54.30.65] 9 22 ms 22 ms 22 ms be2820.rcr21.pit02.atlas.cogentco.com [154.54.83.54] 10 22 ms 22 ms 21 ms 38.104.120.90 11 26 ms 21 ms 19 ms 204.16.241.133 12 * * * Request timed out. 13 21 ms 21 ms 21 ms 204.16.244.8
Is this a possible traffic engineering blip? I can’t say we’ve ever seen trace routes return such sparse results and actually make it to the destination.
-- Nick Zurku Systems Engineer TeraSwitch, Inc. Cell: 412-953-0481 Office: 412-945-7048 nzurku@teraswitch.com
participants (4)
-
Dave Temkin -
Lee -
Matthew Huff -
Nick Zurku