Short-circuited traceroutes on FIOS
Anyone have an idea why there are some destinations that on residential verizon fios here in NY area terminate right on first external hop? There seems to be a CDN common denominator here. On other networks with more typical BGP paths and traceroutes, users are reporting issues accessing these sites. C:\Users\Home>tracert www.usfoods.com Tracing route to statics.usfoods.com [205.132.109.90] over a maximum of 30 hops: 1 3 ms <1 ms <1 ms 172.18.24.1 2 4 ms 3 ms 3 ms 192.168.2.33 3 17 ms 6 ms 3 ms statics.usfoods.com [205.132.109.90] Trace complete. C:\Users\Home>tracert atworkhp.americanexpress.com Tracing route to atworkhp.americanexpress.com.akadns.net [139.71.19.87] over a maximum of 30 hops: 1 2 ms <1 ms <1 ms 172.18.24.1 2 3 ms 4 ms 23 ms 192.168.2.33 3 21 ms 11 ms 5 ms atworkhomepage2.americanexpress.com [139.71.19.87] Trace complete. C:\Users\Home>tracert portal.discover.com Tracing route to e14577.x.akamaiedge.net [23.51.172.254] over a maximum of 30 hops: 1 3 ms 1 ms 18 ms 172.18.24.1 2 21 ms 7 ms 6 ms 192.168.2.33 3 4 ms 2 ms 2 ms a23-51-172-254.deploy.static.akamaitechnologies.com [23.51.172.254] Trace complete.
Apparently Verizon FIOS is a red herring, terminating ICMP traceroutes right on their gateways. More internet breakage. Thanks for the information to all who responded. Random control test. C:\Users\Home>tracert -d 1.4.5.6 Tracing route to 1.4.5.6 over a maximum of 30 hops 1 15 ms 5 ms <1 ms 172.18.24.1 2 3 ms 23 ms 24 ms 192.168.2.33 3 3 ms 6 ms 3 ms 1.4.5.6 Trace complete. Joe Joe Maimon wrote:
Anyone have an idea why there are some destinations that on residential verizon fios here in NY area terminate right on first external hop?
There seems to be a CDN common denominator here. On other networks with more typical BGP paths and traceroutes, users are reporting issues accessing these sites.
C:\Users\Home>tracert www.usfoods.com
Tracing route to statics.usfoods.com [205.132.109.90] over a maximum of 30 hops:
1 3 ms <1 ms <1 ms 172.18.24.1 2 4 ms 3 ms 3 ms 192.168.2.33 3 17 ms 6 ms 3 ms statics.usfoods.com [205.132.109.90]
Trace complete.
C:\Users\Home>tracert atworkhp.americanexpress.com
Tracing route to atworkhp.americanexpress.com.akadns.net [139.71.19.87] over a maximum of 30 hops:
1 2 ms <1 ms <1 ms 172.18.24.1 2 3 ms 4 ms 23 ms 192.168.2.33 3 21 ms 11 ms 5 ms atworkhomepage2.americanexpress.com [139.71.19.87]
Trace complete.
C:\Users\Home>tracert portal.discover.com
Tracing route to e14577.x.akamaiedge.net [23.51.172.254] over a maximum of 30 hops:
1 3 ms 1 ms 18 ms 172.18.24.1 2 21 ms 7 ms 6 ms 192.168.2.33 3 4 ms 2 ms 2 ms a23-51-172-254.deploy.static.akamaitechnologies.com [23.51.172.254]
Trace complete.
Is that unique to the FiOS gateway device? I don't use their router and my traces go right out. On Tue, Dec 10, 2019 at 3:08 PM Joe Maimon <jmaimon@jmaimon.com> wrote:
Apparently Verizon FIOS is a red herring, terminating ICMP traceroutes right on their gateways.
More internet breakage. Thanks for the information to all who responded.
Random control test.
C:\Users\Home>tracert -d 1.4.5.6
Tracing route to 1.4.5.6 over a maximum of 30 hops
1 15 ms 5 ms <1 ms 172.18.24.1 2 3 ms 23 ms 24 ms 192.168.2.33 3 3 ms 6 ms 3 ms 1.4.5.6
Trace complete.
Joe
Joe Maimon wrote:
Anyone have an idea why there are some destinations that on residential verizon fios here in NY area terminate right on first external hop?
There seems to be a CDN common denominator here. On other networks with more typical BGP paths and traceroutes, users are reporting issues accessing these sites.
C:\Users\Home>tracert www.usfoods.com
Tracing route to statics.usfoods.com [205.132.109.90] over a maximum of 30 hops:
1 3 ms <1 ms <1 ms 172.18.24.1 2 4 ms 3 ms 3 ms 192.168.2.33 3 17 ms 6 ms 3 ms statics.usfoods.com [205.132.109.90]
Trace complete.
C:\Users\Home>tracert atworkhp.americanexpress.com
Tracing route to atworkhp.americanexpress.com.akadns.net [139.71.19.87] over a maximum of 30 hops:
1 2 ms <1 ms <1 ms 172.18.24.1 2 3 ms 4 ms 23 ms 192.168.2.33 3 21 ms 11 ms 5 ms atworkhomepage2.americanexpress.com [139.71.19.87]
Trace complete.
C:\Users\Home>tracert portal.discover.com
Tracing route to e14577.x.akamaiedge.net [23.51.172.254] over a maximum of 30 hops:
1 3 ms 1 ms 18 ms 172.18.24.1 2 21 ms 7 ms 6 ms 192.168.2.33 3 4 ms 2 ms 2 ms a23-51-172-254.deploy.static.akamaitechnologies.com [23.51.172.254]
Trace complete.
This is not from a verizon CPE. Its happening on their CO internet gateway customer facing routers. tcptraceroute looks more legit Joe Nimrod Levy wrote:
Is that unique to the FiOS gateway device? I don't use their router and my traces go right out.
On Tue, Dec 10, 2019 at 3:08 PM Joe Maimon <jmaimon@jmaimon.com <mailto:jmaimon@jmaimon.com>> wrote:
Apparently Verizon FIOS is a red herring, terminating ICMP traceroutes right on their gateways.
More internet breakage. Thanks for the information to all who responded.
Random control test.
C:\Users\Home>tracert -d 1.4.5.6
Tracing route to 1.4.5.6 over a maximum of 30 hops
1 15 ms 5 ms <1 ms 172.18.24.1 2 3 ms 23 ms 24 ms 192.168.2.33 3 3 ms 6 ms 3 ms 1.4.5.6
Trace complete.
Joe
Joe Maimon wrote: > Anyone have an idea why there are some destinations that on > residential verizon fios here in NY area terminate right on first > external hop? > > There seems to be a CDN common denominator here. On other networks > with more typical BGP paths and traceroutes, users are reporting > issues accessing these sites. > > C:\Users\Home>tracert www.usfoods.com <http://www.usfoods.com> > > Tracing route to statics.usfoods.com <http://statics.usfoods.com> [205.132.109.90] > over a maximum of 30 hops: > > 1 3 ms <1 ms <1 ms 172.18.24.1 > 2 4 ms 3 ms 3 ms 192.168.2.33 > 3 17 ms 6 ms 3 ms statics.usfoods.com <http://statics.usfoods.com> [205.132.109.90] > > Trace complete. > > C:\Users\Home>tracert atworkhp.americanexpress.com <http://atworkhp.americanexpress.com> > > Tracing route to atworkhp.americanexpress.com.akadns.net <http://atworkhp.americanexpress.com.akadns.net> [139.71.19.87] > over a maximum of 30 hops: > > 1 2 ms <1 ms <1 ms 172.18.24.1 > 2 3 ms 4 ms 23 ms 192.168.2.33 > 3 21 ms 11 ms 5 ms atworkhomepage2.americanexpress.com <http://atworkhomepage2.americanexpress.com> > [139.71.19.87] > > Trace complete. > > C:\Users\Home>tracert portal.discover.com <http://portal.discover.com> > > Tracing route to e14577.x.akamaiedge.net <http://e14577.x.akamaiedge.net> [23.51.172.254] > over a maximum of 30 hops: > > 1 3 ms 1 ms 18 ms 172.18.24.1 > 2 21 ms 7 ms 6 ms 192.168.2.33 > 3 4 ms 2 ms 2 ms > a23-51-172-254.deploy.static.akamaitechnologies.com <http://a23-51-172-254.deploy.static.akamaitechnologies.com> [23.51.172.254] > > Trace complete. > > >
mtr -u 4.2.2.2 --report-wide Start: 2019-12-10T21:26:20-0500 HOST: fedora-lenovo Loss% Snt Last Avg Best Wrst StDev 1.|-- _gateway 0.0% 10 1.3 1.4 1.1 2.3 0.3 2.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 3.|-- b3346.nwrknj-lcr-21.verizon-gni.net 0.0% 10 6.7 4.8 2.2 8.2 1.9 4.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 5.|-- 0.ae1.br1.ewr6.alter.net 0.0% 10 18.9 6.2 3.6 18.9 4.6 6.|-- lag-12.ear3.newark1.level3.net 20.0% 10 3.9 3.3 2.2 4.4 1.0 7.|-- ae-2-3602.ear2.newyork1.level3.net 90.0% 10 5.6 5.6 5.6 5.6 0.0 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 Verizon FIOS They are blocking ICMP and returning false responses right at their gateway at the CO, not the CPE (I'm using my own router) You have to do it using UDP to get real results of a traceroute. - Javier On Tue, Dec 10, 2019 at 3:56 PM Joe Maimon <jmaimon@jmaimon.com> wrote:
This is not from a verizon CPE. Its happening on their CO internet gateway customer facing routers.
tcptraceroute looks more legit
Joe
Nimrod Levy wrote:
Is that unique to the FiOS gateway device? I don't use their router and my traces go right out.
On Tue, Dec 10, 2019 at 3:08 PM Joe Maimon <jmaimon@jmaimon.com <mailto:jmaimon@jmaimon.com>> wrote:
Apparently Verizon FIOS is a red herring, terminating ICMP traceroutes right on their gateways.
More internet breakage. Thanks for the information to all who responded.
Random control test.
C:\Users\Home>tracert -d 1.4.5.6
Tracing route to 1.4.5.6 over a maximum of 30 hops
1 15 ms 5 ms <1 ms 172.18.24.1 2 3 ms 23 ms 24 ms 192.168.2.33 3 3 ms 6 ms 3 ms 1.4.5.6
Trace complete.
Joe
Joe Maimon wrote: > Anyone have an idea why there are some destinations that on > residential verizon fios here in NY area terminate right on first > external hop? > > There seems to be a CDN common denominator here. On other networks > with more typical BGP paths and traceroutes, users are reporting > issues accessing these sites. > > C:\Users\Home>tracert www.usfoods.com <http://www.usfoods.com> > > Tracing route to statics.usfoods.com <http://statics.usfoods.com> [205.132.109.90] > over a maximum of 30 hops: > > 1 3 ms <1 ms <1 ms 172.18.24.1 > 2 4 ms 3 ms 3 ms 192.168.2.33 > 3 17 ms 6 ms 3 ms statics.usfoods.com <http://statics.usfoods.com> [205.132.109.90] > > Trace complete. > > C:\Users\Home>tracert atworkhp.americanexpress.com <http://atworkhp.americanexpress.com> > > Tracing route to atworkhp.americanexpress.com.akadns.net <http://atworkhp.americanexpress.com.akadns.net> [139.71.19.87] > over a maximum of 30 hops: > > 1 2 ms <1 ms <1 ms 172.18.24.1 > 2 3 ms 4 ms 23 ms 192.168.2.33 > 3 21 ms 11 ms 5 ms atworkhomepage2.americanexpress.com <http://atworkhomepage2.americanexpress.com> > [139.71.19.87] > > Trace complete. > > C:\Users\Home>tracert portal.discover.com <http://portal.discover.com> > > Tracing route to e14577.x.akamaiedge.net <http://e14577.x.akamaiedge.net> [23.51.172.254] > over a maximum of 30 hops: > > 1 3 ms 1 ms 18 ms 172.18.24.1 > 2 21 ms 7 ms 6 ms 192.168.2.33 > 3 4 ms 2 ms 2 ms > a23-51-172-254.deploy.static.akamaitechnologies.com <http://a23-51-172-254.deploy.static.akamaitechnologies.com> [23.51.172.254] > > Trace complete. > > >
On Tue, Dec 10, 2019 at 5:36 PM Nimrod Levy <nimrodl@gmail.com> wrote:
Is that unique to the FiOS gateway device? I don't use their router and my traces go right out.
I also don't use their device and: $ traceroute 205.132.109.90 traceroute to 205.132.109.90 (205.132.109.90), 30 hops max, 60 byte packets 1 _gateway (192.168.100.1) 3.085 ms 2.990 ms 2.795 ms .... 15 * 12.250.138.90 (12.250.138.90) 65.970 ms *^C -chris (perhaps this is location specific? I'm in the ashburn-ish-area-ish)
On Tue, Dec 10, 2019 at 3:08 PM Joe Maimon <jmaimon@jmaimon.com> wrote:
Apparently Verizon FIOS is a red herring, terminating ICMP traceroutes right on their gateways.
More internet breakage. Thanks for the information to all who responded.
Random control test.
C:\Users\Home>tracert -d 1.4.5.6
Tracing route to 1.4.5.6 over a maximum of 30 hops
1 15 ms 5 ms <1 ms 172.18.24.1 2 3 ms 23 ms 24 ms 192.168.2.33 3 3 ms 6 ms 3 ms 1.4.5.6
Trace complete.
Joe
Joe Maimon wrote:
Anyone have an idea why there are some destinations that on residential verizon fios here in NY area terminate right on first external hop?
There seems to be a CDN common denominator here. On other networks with more typical BGP paths and traceroutes, users are reporting issues accessing these sites.
C:\Users\Home>tracert www.usfoods.com
Tracing route to statics.usfoods.com [205.132.109.90] over a maximum of 30 hops:
1 3 ms <1 ms <1 ms 172.18.24.1 2 4 ms 3 ms 3 ms 192.168.2.33 3 17 ms 6 ms 3 ms statics.usfoods.com [205.132.109.90]
Trace complete.
C:\Users\Home>tracert atworkhp.americanexpress.com
Tracing route to atworkhp.americanexpress.com.akadns.net [139.71.19.87] over a maximum of 30 hops:
1 2 ms <1 ms <1 ms 172.18.24.1 2 3 ms 4 ms 23 ms 192.168.2.33 3 21 ms 11 ms 5 ms atworkhomepage2.americanexpress.com [139.71.19.87]
Trace complete.
C:\Users\Home>tracert portal.discover.com
Tracing route to e14577.x.akamaiedge.net [23.51.172.254] over a maximum of 30 hops:
1 3 ms 1 ms 18 ms 172.18.24.1 2 21 ms 7 ms 6 ms 192.168.2.33 3 4 ms 2 ms 2 ms a23-51-172-254.deploy.static.akamaitechnologies.com [23.51.172.254]
Trace complete.
On 12/10/19, Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Tue, Dec 10, 2019 at 5:36 PM Nimrod Levy <nimrodl@gmail.com> wrote:
Is that unique to the FiOS gateway device? I don't use their router and my traces go right out.
I also don't use their device and: $ traceroute 205.132.109.90 traceroute to 205.132.109.90 (205.132.109.90), 30 hops max, 60 byte packets 1 _gateway (192.168.100.1) 3.085 ms 2.990 ms 2.795 ms .... 15 * 12.250.138.90 (12.250.138.90) 65.970 ms *^C
-chris (perhaps this is location specific? I'm in the ashburn-ish-area-ish)
It's protocol specific. Windows tracert uses icmp instead of udp. On a linux box try ping -t 2 205.132.109.90 You should get a time to live exceeded but the Verizon router gives you an echo reply instead. Regards, Lee
On Tue, Dec 10, 2019 at 3:08 PM Joe Maimon <jmaimon@jmaimon.com> wrote:
Apparently Verizon FIOS is a red herring, terminating ICMP traceroutes right on their gateways.
More internet breakage. Thanks for the information to all who responded.
Random control test.
C:\Users\Home>tracert -d 1.4.5.6
Tracing route to 1.4.5.6 over a maximum of 30 hops
1 15 ms 5 ms <1 ms 172.18.24.1 2 3 ms 23 ms 24 ms 192.168.2.33 3 3 ms 6 ms 3 ms 1.4.5.6
Trace complete.
Joe
Joe Maimon wrote:
Anyone have an idea why there are some destinations that on residential verizon fios here in NY area terminate right on first external hop?
There seems to be a CDN common denominator here. On other networks with more typical BGP paths and traceroutes, users are reporting issues accessing these sites.
C:\Users\Home>tracert www.usfoods.com
Tracing route to statics.usfoods.com [205.132.109.90] over a maximum of 30 hops:
1 3 ms <1 ms <1 ms 172.18.24.1 2 4 ms 3 ms 3 ms 192.168.2.33 3 17 ms 6 ms 3 ms statics.usfoods.com [205.132.109.90]
Trace complete.
C:\Users\Home>tracert atworkhp.americanexpress.com
Tracing route to atworkhp.americanexpress.com.akadns.net [139.71.19.87] over a maximum of 30 hops:
1 2 ms <1 ms <1 ms 172.18.24.1 2 3 ms 4 ms 23 ms 192.168.2.33 3 21 ms 11 ms 5 ms atworkhomepage2.americanexpress.com [139.71.19.87]
Trace complete.
C:\Users\Home>tracert portal.discover.com
Tracing route to e14577.x.akamaiedge.net [23.51.172.254] over a maximum of 30 hops:
1 3 ms 1 ms 18 ms 172.18.24.1 2 21 ms 7 ms 6 ms 192.168.2.33 3 4 ms 2 ms 2 ms a23-51-172-254.deploy.static.akamaitechnologies.com [23.51.172.254]
Trace complete.
On Tue, Dec 10, 2019 at 11:44 PM Lee <ler762@gmail.com> wrote:
It's protocol specific. Windows tracert uses icmp instead of udp. On a linux box try ping -t 2 205.132.109.90
You should get a time to live exceeded but the Verizon router gives you an echo reply instead.
that's hilariously bad :( I think this is the OLT really that's doing this... $ ping -t 3 205.132.109.90 PING 205.132.109.90 (205.132.109.90) 56(84) bytes of data.
From 130.81.32.236 icmp_seq=1 Time to live exceeded
$ ping -t 1 205.132.109.90 PING 205.132.109.90 (205.132.109.90) 56(84) bytes of data.
From 192.168.100.1 icmp_seq=1 Time to live exceeded
$ ping -t 2 205.132.109.90 PING 205.132.109.90 (205.132.109.90) 56(84) bytes of data. 64 bytes from 205.132.109.90: icmp_seq=1 ttl=254 time=3.38 ms An outbound traceroute has: 1 _gateway (192.168.100.1) 2.537 ms 2.587 ms 2.703 ms 2 * * * 3 B3320.WASHDC-LCR-21.verizon-gni.net (130.81.32.236) 6.638 ms B3320.WASHDC-LCR-22.verizon-gni.net (130.81.32.238) 6.223 ms 6.414 ms ... and inbound that hop 2 is: 6 HundredGigE2-4-0-3.WASHDC-LCR-22.verizon-gni.NET (140.222.238.55) 5.504 ms HundredGigE2-6-0-3.WASHDC-LCR-21.verizon-gni.NET (140.222.234.53) 9.261 ms 9.266 ms 7 ae203-0.WASHDC-VFTTP-320.verizon-gni.net () 7.955 ms 3.026 ms ae204-0.WASHDC-VFTTP-320.verizon-gni.net (130.81.32.239) 2.347 ms oh well, just wonky gpon again?
I'm in the same region as Chris and I still can't make it fail. I wonder if it's because I have static addressing? On Tue, Dec 10, 2019 at 11:59 PM Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Tue, Dec 10, 2019 at 11:44 PM Lee <ler762@gmail.com> wrote:
It's protocol specific. Windows tracert uses icmp instead of udp. On a linux box try ping -t 2 205.132.109.90
You should get a time to live exceeded but the Verizon router gives you an echo reply instead.
that's hilariously bad :( I think this is the OLT really that's doing this... $ ping -t 3 205.132.109.90 PING 205.132.109.90 (205.132.109.90) 56(84) bytes of data. From 130.81.32.236 icmp_seq=1 Time to live exceeded
$ ping -t 1 205.132.109.90 PING 205.132.109.90 (205.132.109.90) 56(84) bytes of data. From 192.168.100.1 icmp_seq=1 Time to live exceeded
$ ping -t 2 205.132.109.90 PING 205.132.109.90 (205.132.109.90) 56(84) bytes of data. 64 bytes from 205.132.109.90: icmp_seq=1 ttl=254 time=3.38 ms
An outbound traceroute has: 1 _gateway (192.168.100.1) 2.537 ms 2.587 ms 2.703 ms 2 * * * 3 B3320.WASHDC-LCR-21.verizon-gni.net (130.81.32.236) 6.638 ms B3320.WASHDC-LCR-22.verizon-gni.net (130.81.32.238) 6.223 ms 6.414 ms ...
and inbound that hop 2 is: 6 HundredGigE2-4-0-3.WASHDC-LCR-22.verizon-gni.NET (140.222.238.55) 5.504 ms HundredGigE2-6-0-3.WASHDC-LCR-21.verizon-gni.NET (140.222.234.53) 9.261 ms 9.266 ms 7 ae203-0.WASHDC-VFTTP-320.verizon-gni.net () 7.955 ms 3.026 ms ae204-0.WASHDC-VFTTP-320.verizon-gni.net (130.81.32.239) 2.347 ms
oh well, just wonky gpon again?
If you have static addressing (biz account) then possibly different from what I have. In North NJ, 3 different accounts I can verify have ICMP blocked as of sometime earlier this year or late last year so have to use udp to get a real traceroute. Could not be deployed in all areas the same way. - Javier On Wed, Dec 11, 2019 at 7:19 AM Nimrod Levy <nimrodl@gmail.com> wrote:
I'm in the same region as Chris and I still can't make it fail. I wonder if it's because I have static addressing?
On Tue, Dec 10, 2019 at 11:59 PM Christopher Morrow < morrowc.lists@gmail.com> wrote:
On Tue, Dec 10, 2019 at 11:44 PM Lee <ler762@gmail.com> wrote:
It's protocol specific. Windows tracert uses icmp instead of udp. On a linux box try ping -t 2 205.132.109.90
You should get a time to live exceeded but the Verizon router gives you an echo reply instead.
that's hilariously bad :( I think this is the OLT really that's doing this... $ ping -t 3 205.132.109.90 PING 205.132.109.90 (205.132.109.90) 56(84) bytes of data. From 130.81.32.236 icmp_seq=1 Time to live exceeded
$ ping -t 1 205.132.109.90 PING 205.132.109.90 (205.132.109.90) 56(84) bytes of data. From 192.168.100.1 icmp_seq=1 Time to live exceeded
$ ping -t 2 205.132.109.90 PING 205.132.109.90 (205.132.109.90) 56(84) bytes of data. 64 bytes from 205.132.109.90: icmp_seq=1 ttl=254 time=3.38 ms
An outbound traceroute has: 1 _gateway (192.168.100.1) 2.537 ms 2.587 ms 2.703 ms 2 * * * 3 B3320.WASHDC-LCR-21.verizon-gni.net (130.81.32.236) 6.638 ms B3320.WASHDC-LCR-22.verizon-gni.net (130.81.32.238) 6.223 ms 6.414 ms ...
and inbound that hop 2 is: 6 HundredGigE2-4-0-3.WASHDC-LCR-22.verizon-gni.NET (140.222.238.55) 5.504 ms HundredGigE2-6-0-3.WASHDC-LCR-21.verizon-gni.NET (140.222.234.53) 9.261 ms 9.266 ms 7 ae203-0.WASHDC-VFTTP-320.verizon-gni.net () 7.955 ms 3.026 ms ae204-0.WASHDC-VFTTP-320.verizon-gni.net (130.81.32.239) 2.347 ms
oh well, just wonky gpon again?
On Wed, 11 Dec 2019, Javier J wrote:
If you have static addressing (biz account) then possibly different from what I have.
In North NJ, 3 different accounts I can verify have ICMP blocked as of sometime earlier this year or late last year so have to use udp to get a real traceroute.
Could not be deployed in all areas the same way.
I noticed this about the same time I installed Ubiquiti gear at home, December 2018. Until this thread, I thought there was something wrong with my gateway router config. I could do UDP/TCP traceroutes, but ICMP kept dying. Glad to know it isn't my gateway, but frustrated as hell that Verizon decided that a few customers doing less-than-ideal things was enough to cut a standard network protocol off at the knees. --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman@angryox.com http://www.angryox.com/ ---------------------------------------------------------------------------
Is anyone from Verizon on this list? They probably are but not allowed to comment. I would love to know if there is an official comment on why they do this. It annoyed me when they first implemented and I was trying to diagnose an issue with a client. Regarding your edge device: Same here, I had ubiquity gear at my GW for a while and before that PFsense. When i saw 1ms responses to a ping one day I was confused. - J On Thu, Dec 12, 2019 at 12:51 PM Peter Beckman <beckman@angryox.com> wrote:
On Wed, 11 Dec 2019, Javier J wrote:
If you have static addressing (biz account) then possibly different from what I have.
In North NJ, 3 different accounts I can verify have ICMP blocked as of sometime earlier this year or late last year so have to use udp to get a real traceroute.
Could not be deployed in all areas the same way.
I noticed this about the same time I installed Ubiquiti gear at home, December 2018.
Until this thread, I thought there was something wrong with my gateway router config. I could do UDP/TCP traceroutes, but ICMP kept dying.
Glad to know it isn't my gateway, but frustrated as hell that Verizon decided that a few customers doing less-than-ideal things was enough to cut a standard network protocol off at the knees.
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman@angryox.com http://www.angryox.com/ ---------------------------------------------------------------------------
On Fri, Dec 13, 2019 at 4:16 PM Javier J <javier@advancedmachines.us> wrote:
Is anyone from Verizon on this list? They probably are but not allowed to comment. I would love to know if there is an official comment on why they do this.
It annoyed me when they first implemented and I was trying to diagnose an issue with a client.
Regarding your edge device: Same here, I had ubiquity gear at my GW for a while and before that PFsense.
When i saw 1ms responses to a ping one day I was confused.
Well, yes, but you *did* see a 1ms ping response. I'm sure no ISPs would intentionally configure their networks to artificially improve their latency measurements on certain automated tools, so I don't know why that would be a useful outcome... W
- J
On Thu, Dec 12, 2019 at 12:51 PM Peter Beckman <beckman@angryox.com> wrote:
On Wed, 11 Dec 2019, Javier J wrote:
If you have static addressing (biz account) then possibly different from what I have.
In North NJ, 3 different accounts I can verify have ICMP blocked as of sometime earlier this year or late last year so have to use udp to get a real traceroute.
Could not be deployed in all areas the same way.
I noticed this about the same time I installed Ubiquiti gear at home, December 2018.
Until this thread, I thought there was something wrong with my gateway router config. I could do UDP/TCP traceroutes, but ICMP kept dying.
Glad to know it isn't my gateway, but frustrated as hell that Verizon decided that a few customers doing less-than-ideal things was enough to cut a standard network protocol off at the knees.
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman@angryox.com http://www.angryox.com/ ---------------------------------------------------------------------------
-- I don't think the execution is relevant when it was obviously a bad idea in the first place. This is like putting rabid weasels in your pants, and later expressing regret at having chosen those particular rabid weasels and that pair of pants. ---maf
wasn't vz pursuing some 'get the a cdn in the central office' for a time? :) perhaps this is the manifestation of that? :) or perhaps jared arranged to get links back from each CO to his network gear in akamai-land? I love conspiracies! On Tue, Dec 10, 2019 at 2:48 PM Joe Maimon <jmaimon@jmaimon.com> wrote:
Anyone have an idea why there are some destinations that on residential verizon fios here in NY area terminate right on first external hop?
There seems to be a CDN common denominator here. On other networks with more typical BGP paths and traceroutes, users are reporting issues accessing these sites.
C:\Users\Home>tracert www.usfoods.com
Tracing route to statics.usfoods.com [205.132.109.90] over a maximum of 30 hops:
1 3 ms <1 ms <1 ms 172.18.24.1 2 4 ms 3 ms 3 ms 192.168.2.33 3 17 ms 6 ms 3 ms statics.usfoods.com [205.132.109.90]
Trace complete.
C:\Users\Home>tracert atworkhp.americanexpress.com
Tracing route to atworkhp.americanexpress.com.akadns.net [139.71.19.87] over a maximum of 30 hops:
1 2 ms <1 ms <1 ms 172.18.24.1 2 3 ms 4 ms 23 ms 192.168.2.33 3 21 ms 11 ms 5 ms atworkhomepage2.americanexpress.com [139.71.19.87]
Trace complete.
C:\Users\Home>tracert portal.discover.com
Tracing route to e14577.x.akamaiedge.net [23.51.172.254] over a maximum of 30 hops:
1 3 ms 1 ms 18 ms 172.18.24.1 2 21 ms 7 ms 6 ms 192.168.2.33 3 4 ms 2 ms 2 ms a23-51-172-254.deploy.static.akamaitechnologies.com [23.51.172.254]
Trace complete.
I had this issue while looking at Ripe Atlas measurements. Turns out these Verizon boxes spoof ICMP with TTL = 3 (or 2, I don't recall). Try doing a UDP or TCP based traceroute instead. Maybe you're seeing the same problem. Kind Regards, Filip On 12/10/19 8:47 PM, Joe Maimon wrote:
Anyone have an idea why there are some destinations that on residential verizon fios here in NY area terminate right on first external hop?
There seems to be a CDN common denominator here. On other networks with more typical BGP paths and traceroutes, users are reporting issues accessing these sites.
C:\Users\Home>tracert www.usfoods.com
Tracing route to statics.usfoods.com [205.132.109.90] over a maximum of 30 hops:
1 3 ms <1 ms <1 ms 172.18.24.1 2 4 ms 3 ms 3 ms 192.168.2.33 3 17 ms 6 ms 3 ms statics.usfoods.com [205.132.109.90]
Trace complete.
C:\Users\Home>tracert atworkhp.americanexpress.com
Tracing route to atworkhp.americanexpress.com.akadns.net [139.71.19.87] over a maximum of 30 hops:
1 2 ms <1 ms <1 ms 172.18.24.1 2 3 ms 4 ms 23 ms 192.168.2.33 3 21 ms 11 ms 5 ms atworkhomepage2.americanexpress.com [139.71.19.87]
Trace complete.
C:\Users\Home>tracert portal.discover.com
Tracing route to e14577.x.akamaiedge.net [23.51.172.254] over a maximum of 30 hops:
1 3 ms 1 ms 18 ms 172.18.24.1 2 21 ms 7 ms 6 ms 192.168.2.33 3 4 ms 2 ms 2 ms a23-51-172-254.deploy.static.akamaitechnologies.com [23.51.172.254]
Trace complete.
-- Filip Hruska Linux System Administrator
On Tue, 10 Dec 2019, Joe Maimon wrote:
Anyone have an idea why there are some destinations that on residential verizon fios here in NY area terminate right on first external hop?
They're returning fake ICMP echo replies from their BNGs for echo packets with TTL=1, thus any ICMP traceroute (Windows and mtr by default, etc.) seems to terminate at their layer 3 edge. UDP/TCP traceroute are unaffected, ICMP works fine if you set the initial TTL to n+1 where n is the hop that's lying. Support claims that it was a mistake, but it's also been 15+ months and it's pretty deliberate behavior. Draw your own conclusions... -Rob
On Wed, 11 Dec 2019 at 19:14, Rob Foehl <rwf@loonybin.net> wrote:
Support claims that it was a mistake, but it's also been 15+ months and it's pretty deliberate behavior. Draw your own conclusions...
TTL decrement issues are fairly common across multiple vendors and hw, can be sw can be hw limit. Common issues for example is if MPLS egress PE receives explicit null labeled packet, it may not be able to decrement TTL. I may lack in imagination, but I struggle to envision a situation where people decided to do this and then decided to be sneaky peaky about it. -- ++ytti
On Wed, 11 Dec 2019 19:26:09 +0200, Saku Ytti said:
On Wed, 11 Dec 2019 at 19:14, Rob Foehl <rwf@loonybin.net> wrote:
Support claims that it was a mistake, but it's also been 15+ months and it's pretty deliberate behavior. Draw your own conclusions...
TTL decrement issues are fairly common across multiple vendors and hw, can be sw can be hw limit
Yes, but you need to screw up gloriously on the decrement if you think that "I decremented and it's zero now" means "therefor it must have been addressed to me, so I'll send an ECHO REPLY instead of TTL EXCEEDED".
Traceroute is becoming more and more an expert's tool because interpretation of its results isn't straightforward. I had written a paper last year and mentioned its misuse in academia in the context of estimating the number of energy-consuming devices between a source and a destination. Traceroute was being used to count the number of physical router devices from the hop count, notwithstanding the use of MPLS in domain cores. To an external observer, this results in significant underestimation of the energy consumption in the path from source to destination. On Thu, Dec 12, 2019 at 12:51 AM Valdis Klētnieks <valdis.kletnieks@vt.edu> wrote:
On Wed, 11 Dec 2019 19:26:09 +0200, Saku Ytti said:
On Wed, 11 Dec 2019 at 19:14, Rob Foehl <rwf@loonybin.net> wrote:
Support claims that it was a mistake, but it's also been 15+ months and it's pretty deliberate behavior. Draw your own conclusions...
TTL decrement issues are fairly common across multiple vendors and hw, can be sw can be hw limit
Yes, but you need to screw up gloriously on the decrement if you think that "I decremented and it's zero now" means "therefor it must have been addressed to me, so I'll send an ECHO REPLY instead of TTL EXCEEDED".
-- Ing. Etienne-Victor Depasquale Assistant Lecturer Department of Communications & Computer Engineering Faculty of Information & Communication Technology University of Malta Web. https://www.um.edu.mt/profile/etiennedepasquale
Yeah, and what do you do with a traceroute that looks like this…. (ip address intentionally changed) C:\>tracert -d -w 1 1.2.3.4 Tracing route to 1.2.3.4 over a maximum of 30 hops 1 8 ms 5 ms 5 ms 96.8.191.129 2 * * * Request timed out. 3 * * * Request timed out. 4 * * * Request timed out. 5 * * * Request timed out. 6 * * * Request timed out. 7 * * * Request timed out. 8 * * * Request timed out. 9 * * * Request timed out. 10 * * * Request timed out. 11 * * * Request timed out. 12 * * * Request timed out. 13 * * * Request timed out. 14 * * * Request timed out. 15 * * * Request timed out. 16 * * * Request timed out. 17 267 ms 202 ms * 1.2.3.4 18 205 ms 175 ms * 1.2.3.4 19 160 ms 233 ms * 1.2.3.4 20 199 ms 201 ms * 1.2.3.4 21 213 ms 206 ms * 1.2.3.4 22 165 ms 158 ms * 1.2.3.4 23 237 ms 158 ms * 1.2.3.4 24 158 ms 290 ms * 1.2.3.4 25 158 ms 160 ms 158 ms 1.2.3.4 Trace complete. C:\> From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Etienne-Victor Depasquale Sent: Thursday, December 12, 2019 1:18 AM To: Valdis Klētnieks Cc: nanog@nanog.org Subject: Re: Short-circuited traceroutes on FIOS Traceroute is becoming more and more an expert's tool because interpretation of its results isn't straightforward. I had written a paper last year and mentioned its misuse in academia in the context of estimating the number of energy-consuming devices between a source and a destination. Traceroute was being used to count the number of physical router devices from the hop count, notwithstanding the use of MPLS in domain cores. To an external observer, this results in significant underestimation of the energy consumption in the path from source to destination. On Thu, Dec 12, 2019 at 12:51 AM Valdis Klētnieks <valdis.kletnieks@vt.edu> wrote: On Wed, 11 Dec 2019 19:26:09 +0200, Saku Ytti said:
On Wed, 11 Dec 2019 at 19:14, Rob Foehl <rwf@loonybin.net> wrote:
Support claims that it was a mistake, but it's also been 15+ months and it's pretty deliberate behavior. Draw your own conclusions...
TTL decrement issues are fairly common across multiple vendors and hw, can be sw can be hw limit
Yes, but you need to screw up gloriously on the decrement if you think that "I decremented and it's zero now" means "therefor it must have been addressed to me, so I'll send an ECHO REPLY instead of TTL EXCEEDED". -- Ing. Etienne-Victor Depasquale Assistant Lecturer Department of Communications & Computer Engineering Faculty of Information & Communication Technology University of Malta Web. https://www.um.edu.mt/profile/etiennedepasquale
what do you do with a traceroute that looks like this
Tell you to not change IP addresses so that I can do a proper analysis on it? Recommend you use something other than windows? Give you a stock tip? The possibilities are endless. (I'm being sarcastic) It is shitty and I have no clue why ISPs play these games. Rumor is that with FiOS, that so many gamers or game software was sending out ICMP requests that it was enough traffic for them to say screw this and block it. I don't buy that but whatever. Just annoying. Oh yeah, and why do I still to this day have to use a HE ipv6 tunnel? On Thu, Dec 12, 2019 at 8:55 AM Aaron Gould <aaron1@gvtc.com> wrote:
Yeah, and what do you do with a traceroute that looks like this…. (ip address intentionally changed)
C:\>tracert -d -w 1 1.2.3.4
Tracing route to 1.2.3.4 over a maximum of 30 hops
1 8 ms 5 ms 5 ms 96.8.191.129
2 * * * Request timed out.
3 * * * Request timed out.
4 * * * Request timed out.
5 * * * Request timed out.
6 * * * Request timed out.
7 * * * Request timed out.
8 * * * Request timed out.
9 * * * Request timed out.
10 * * * Request timed out.
11 * * * Request timed out.
12 * * * Request timed out.
13 * * * Request timed out.
14 * * * Request timed out.
15 * * * Request timed out.
16 * * * Request timed out.
17 267 ms 202 ms * 1.2.3.4
18 205 ms 175 ms * 1.2.3.4
19 160 ms 233 ms * 1.2.3.4
20 199 ms 201 ms * 1.2.3.4
21 213 ms 206 ms * 1.2.3.4
22 165 ms 158 ms * 1.2.3.4
23 237 ms 158 ms * 1.2.3.4
24 158 ms 290 ms * 1.2.3.4
25 158 ms 160 ms 158 ms 1.2.3.4
Trace complete.
C:\>
*From:* NANOG [mailto:nanog-bounces@nanog.org] *On Behalf Of *Etienne-Victor Depasquale *Sent:* Thursday, December 12, 2019 1:18 AM *To:* Valdis Klētnieks *Cc:* nanog@nanog.org *Subject:* Re: Short-circuited traceroutes on FIOS
Traceroute is becoming more and more an expert's tool because interpretation of its results isn't straightforward.
I had written a paper last year and mentioned its misuse in academia in the context of estimating the number of energy-consuming devices between a source and a destination.
Traceroute was being used to count the number of physical router devices from the hop count, notwithstanding the use of MPLS in domain cores.
To an external observer, this results in significant underestimation of the energy consumption in the path from source to destination.
On Thu, Dec 12, 2019 at 12:51 AM Valdis Klētnieks <valdis.kletnieks@vt.edu> wrote:
On Wed, 11 Dec 2019 19:26:09 +0200, Saku Ytti said:
On Wed, 11 Dec 2019 at 19:14, Rob Foehl <rwf@loonybin.net> wrote:
Support claims that it was a mistake, but it's also been 15+ months and it's pretty deliberate behavior. Draw your own conclusions...
TTL decrement issues are fairly common across multiple vendors and hw, can be sw can be hw limit
Yes, but you need to screw up gloriously on the decrement if you think that "I decremented and it's zero now" means "therefor it must have been addressed to me, so I'll send an ECHO REPLY instead of TTL EXCEEDED".
--
Ing. Etienne-Victor Depasquale Assistant Lecturer Department of Communications & Computer Engineering Faculty of Information & Communication Technology University of Malta
On Dec 11, 2019, at 09:26 , Saku Ytti <saku@ytti.fi> wrote:
On Wed, 11 Dec 2019 at 19:14, Rob Foehl <rwf@loonybin.net> wrote:
Support claims that it was a mistake, but it's also been 15+ months and it's pretty deliberate behavior. Draw your own conclusions...
TTL decrement issues are fairly common across multiple vendors and hw, can be sw can be hw limit. Common issues for example is if MPLS egress PE receives explicit null labeled packet, it may not be able to decrement TTL. I may lack in imagination, but I struggle to envision a situation where people decided to do this and then decided to be sneaky peaky about it.
All of those would still result in either a dropped packet or some form of erroneous ICMP error message. Responding to an ICMP ECHO REQUEST with a TTL of 1 and a destination address that isn’t local using an ICMP ECHO REPLY spoofing the destination address (the observed behavior) doesn’t fit any of those scenarios. It would require some pretty strong creativity and custom code to implement. Owen
How have we gone this long of a conversation without someone from FIOS stepping in and setting the record straight? ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Joe Maimon" <jmaimon@jmaimon.com> To: "North American Networking and Offtopic Gripes List" <nanog@nanog.org> Sent: Tuesday, December 10, 2019 1:47:17 PM Subject: Short-circuited traceroutes on FIOS Anyone have an idea why there are some destinations that on residential verizon fios here in NY area terminate right on first external hop? There seems to be a CDN common denominator here. On other networks with more typical BGP paths and traceroutes, users are reporting issues accessing these sites. C:\Users\Home>tracert www.usfoods.com Tracing route to statics.usfoods.com [205.132.109.90] over a maximum of 30 hops: 1 3 ms <1 ms <1 ms 172.18.24.1 2 4 ms 3 ms 3 ms 192.168.2.33 3 17 ms 6 ms 3 ms statics.usfoods.com [205.132.109.90] Trace complete. C:\Users\Home>tracert atworkhp.americanexpress.com Tracing route to atworkhp.americanexpress.com.akadns.net [139.71.19.87] over a maximum of 30 hops: 1 2 ms <1 ms <1 ms 172.18.24.1 2 3 ms 4 ms 23 ms 192.168.2.33 3 21 ms 11 ms 5 ms atworkhomepage2.americanexpress.com [139.71.19.87] Trace complete. C:\Users\Home>tracert portal.discover.com Tracing route to e14577.x.akamaiedge.net [23.51.172.254] over a maximum of 30 hops: 1 3 ms 1 ms 18 ms 172.18.24.1 2 21 ms 7 ms 6 ms 192.168.2.33 3 4 ms 2 ms 2 ms a23-51-172-254.deploy.static.akamaitechnologies.com [23.51.172.254] Trace complete.
Greetings, * Mike Hammett (nanog@ics-il.net) wrote:
How have we gone this long of a conversation without someone from FIOS stepping in and setting the record straight?
How have we gone this long without ipv6 on FIOS ...? (though I've heard that it may have shown up in some places, I'm certainly still not seeing it...) Thanks, Stephen
participants (16)
-
Aaron Gould -
Christopher Morrow -
Etienne-Victor Depasquale -
Filip Hruska -
Javier J -
Joe Maimon -
Lee -
Mike Hammett -
Nimrod Levy -
Owen DeLong -
Peter Beckman -
Rob Foehl -
Saku Ytti -
Stephen Frost -
Valdis Klētnieks -
Warren Kumari