Any Verizon folks here? I've been having some rather weird network issues lately - just reading email via IMAP, from home. Over a 1gig FIOS connection to a machine in a nearby Tierpoint data center that has LOTS of good connectivity. I just tried some traceroutes, and got some interesting results: These originate on a machine connected to a 1gig FIOS feed, and end at a machine, located in a Tierpoint datacenter, about 10 miles from here. traceroute to ntcorp.com (207.154.13.58), 64 hops max, 52 byte packets 1 * fios_quantum_gateway (192.168.1.1) 3.530 ms 2.822 ms 2 * * * 3 100.41.27.110 (100.41.27.110) 14.970 ms 5.323 ms 6.306 ms 4 0.csi1.bstnmafr-mse01-bb-su1.alter.net (140.222.10.32) 11.069 ms 8.477 ms 17.097 ms 5 * * * 6 0.ae1.br1.bos30.alter.net (140.222.236.253) 17.121 ms 19.027 ms 0.ae2.br1.bos30.alter.net (140.222.236.255) 19.795 ms 7 * * * 8 colo4-dalla.bear1.boston1.level3.net (4.53.61.86) 2205.648 ms 8.331 ms 13.161 ms 9 static-33-65-203-66.axsne.net (66.203.65.33) 16.951 ms 13.791 ms static-145-65-203-66.axsne.net (66.203.65.145) 21.503 ms 10 server1.ntcorp.com (207.154.13.58) 17.872 ms 15.902 ms 14.415 ms Several things jump out: 1. alter.net is not a common path between here & there - usually a lower grade connection, when other backbones aren't working right 2. origin - alter.net - level.3 - endpoint is just bizarre, one would think that the regional FIOS network has a direct connection to level.3 (it also seems kind of odd that the packets are flowing from Acton MA, to Boston, and back out to Marlboro MA - there's an awful lot of fiber running along Rt. 495, and the networks are fairly dense around here) 3. The intermittent, high delays (factor of 10) jump out (also, when running ping tests, there seem to be intermittent periods of long sequences of timeouts) All in all it's really mucking with both streaming services, and simply posting emails (SMTP timeouts). All of which leads me to wonder if there's something mucked up with Verizon's routing tables (or a particular network interface). Any insights (or fixes) to be had? Thanks, Miles Fidelman -- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra Theory is when you know everything but nothing works. Practice is when everything works but no one knows why. In our lab, theory and practice are combined: nothing works and no one knows why. ... unknown
On Sat, Oct 9, 2021, 13:45 Miles Fidelman <mfidelman@meetinghouse.net> wrote:
Any Verizon folks here?
I've been having some rather weird network issues lately - just reading email via IMAP, from home. Over a 1gig FIOS connection to a machine in a nearby Tierpoint data center that has LOTS of good connectivity.
I just tried some traceroutes, and got some interesting results:
These originate on a machine connected to a 1gig FIOS feed, and end at a machine, located in a Tierpoint datacenter, about 10 miles from here.
traceroute to ntcorp.com (207.154.13.58), 64 hops max, 52 byte packets 1 * fios_quantum_gateway (192.168.1.1) 3.530 ms 2.822 ms 2 * * * 3 100.41.27.110 (100.41.27.110) 14.970 ms 5.323 ms 6.306 ms 4 0.csi1.bstnmafr-mse01-bb-su1.alter.net (140.222.10.32) 11.069 ms 8.477 ms 17.097 ms 5 * * * 6 0.ae1.br1.bos30.alter.net (140.222.236.253) 17.121 ms 19.027 ms 0.ae2.br1.bos30.alter.net (140.222.236.255) 19.795 ms 7 * * * 8 colo4-dalla.bear1.boston1.level3.net (4.53.61.86) 2205.648 ms 8.331 ms 13.161 ms 9 static-33-65-203-66.axsne.net (66.203.65.33) 16.951 ms 13.791 ms static-145-65-203-66.axsne.net (66.203.65.145) 21.503 ms 10 server1.ntcorp.com (207.154.13.58) 17.872 ms 15.902 ms 14.415 ms
Several things jump out:
1. alter.net is not a common path between here & there - usually a lower grade connection, when other backbones aren't working right
Alternet is the domain used by legacy uunet equipment/ips, or was that domain "forever".
2. origin - alter.net - level.3 - endpoint is just bizarre, one would
Why? "Br" is the role name used for sfp peer interconnect devices on uunet's network.
think that the regional FIOS network has a direct connection to level.3 (it also seems kind of odd that the packets are flowing from Acton MA, to Boston, and back out to Marlboro MA - there's an awful lot of fiber running along Rt. 495, and the networks are fairly dense around here)
3. The intermittent, high delays (factor of 10) jump out (also, when running ping tests, there seem to be intermittent periods of long sequences of timeouts)
All in all it's really mucking with both streaming services, and simply posting emails (SMTP timeouts).
All of which leads me to wonder if there's something mucked up with Verizon's routing tables (or a particular network interface).
Any insights (or fixes) to be had?
Thanks,
Miles Fidelman
-- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra
Theory is when you know everything but nothing works. Practice is when everything works but no one knows why. In our lab, theory and practice are combined: nothing works and no one knows why. ... unknown
On Sat, Oct 9, 2021, 13:45 Miles Fidelman <mfidelman@meetinghouse.net>
2. origin - alter.net - level.3 - endpoint is just bizarre, one would think that the regional FIOS network has a direct connection to level.3
No. Former verizon-gni backbone (where FiOS sits) takes transit solely from VZB (now UUNET), this has been the case for many many years, but most people get confused when interpretting traceroutes due to mpls no-ttl-propagate on VZ networks.
(it also seems kind of odd that the packets are flowing from Acton MA, to Boston, and back out to Marlboro MA - there's an awful lot of fiber running along Rt. 495, and the networks are fairly dense around here)
Level3-VZB-px is at 300 Bent Street, Cambridge, MA, where both parties have peering routers installed (bear1.Boston1.Level3.net and BR1.BOS30.ALTER.NET). It makes perfect sense for interconnection to occur in Cambridge/Boston, rather than maintaining peering routers out in the suburb within the same metro. None of these are contributory factors to the issues you're describing. James
alter.net is just the legacy RDNS for things in AS701 (uunet). Nothing weird there. https://en.wikipedia.org/wiki/UUNET On Sat, Oct 9, 2021 at 1:46 PM Miles Fidelman <mfidelman@meetinghouse.net> wrote:
Any Verizon folks here?
I've been having some rather weird network issues lately - just reading email via IMAP, from home. Over a 1gig FIOS connection to a machine in a nearby Tierpoint data center that has LOTS of good connectivity.
I just tried some traceroutes, and got some interesting results:
These originate on a machine connected to a 1gig FIOS feed, and end at a machine, located in a Tierpoint datacenter, about 10 miles from here.
traceroute to ntcorp.com (207.154.13.58), 64 hops max, 52 byte packets 1 * fios_quantum_gateway (192.168.1.1) 3.530 ms 2.822 ms 2 * * * 3 100.41.27.110 (100.41.27.110) 14.970 ms 5.323 ms 6.306 ms 4 0.csi1.bstnmafr-mse01-bb-su1.alter.net (140.222.10.32) 11.069 ms 8.477 ms 17.097 ms 5 * * * 6 0.ae1.br1.bos30.alter.net (140.222.236.253) 17.121 ms 19.027 ms 0.ae2.br1.bos30.alter.net (140.222.236.255) 19.795 ms 7 * * * 8 colo4-dalla.bear1.boston1.level3.net (4.53.61.86) 2205.648 ms 8.331 ms 13.161 ms 9 static-33-65-203-66.axsne.net (66.203.65.33) 16.951 ms 13.791 ms static-145-65-203-66.axsne.net (66.203.65.145) 21.503 ms 10 server1.ntcorp.com (207.154.13.58) 17.872 ms 15.902 ms 14.415 ms
Several things jump out:
1. alter.net is not a common path between here & there - usually a lower grade connection, when other backbones aren't working right
2. origin - alter.net - level.3 - endpoint is just bizarre, one would think that the regional FIOS network has a direct connection to level.3 (it also seems kind of odd that the packets are flowing from Acton MA, to Boston, and back out to Marlboro MA - there's an awful lot of fiber running along Rt. 495, and the networks are fairly dense around here)
3. The intermittent, high delays (factor of 10) jump out (also, when running ping tests, there seem to be intermittent periods of long sequences of timeouts)
All in all it's really mucking with both streaming services, and simply posting emails (SMTP timeouts).
All of which leads me to wonder if there's something mucked up with Verizon's routing tables (or a particular network interface).
Any insights (or fixes) to be had?
Thanks,
Miles Fidelman
-- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra
Theory is when you know everything but nothing works. Practice is when everything works but no one knows why. In our lab, theory and practice are combined: nothing works and no one knows why. ... unknown
3. The intermittent, high delays (factor of 10) jump out (also, when running ping tests, there seem to be intermittent periods of long sequences of timeouts) or, that, for about 4 years now, gamers seem to be reporting really poor
Ok folks, Thanks for the info about uunet. But that doesn't address: performance across FIOS in the Northeast - tied to rather high packet loss rates. Note that those packet losses seems to be bursts of 8-10 lost packets every 10 packets or so, and the really high delays on some of the traceroutes seem to indicate that it's happening somewhere in the middle of the path, not at my end. And, come to think of it, that might explain some of the horrid performance of the FIOS channel guide. Any thoughts? Anyone here from Verizon Northeast FIOS operations who might have a comment? Thanks, Miles Fidelman Miles Fidelman wrote:
Any Verizon folks here?
I've been having some rather weird network issues lately - just reading email via IMAP, from home. Over a 1gig FIOS connection to a machine in a nearby Tierpoint data center that has LOTS of good connectivity.
I just tried some traceroutes, and got some interesting results:
These originate on a machine connected to a 1gig FIOS feed, and end at a machine, located in a Tierpoint datacenter, about 10 miles from here.
traceroute to ntcorp.com (207.154.13.58), 64 hops max, 52 byte packets 1 * fios_quantum_gateway (192.168.1.1) 3.530 ms 2.822 ms 2 * * * 3 100.41.27.110 (100.41.27.110) 14.970 ms 5.323 ms 6.306 ms 4 0.csi1.bstnmafr-mse01-bb-su1.alter.net (140.222.10.32) 11.069 ms 8.477 ms 17.097 ms 5 * * * 6 0.ae1.br1.bos30.alter.net (140.222.236.253) 17.121 ms 19.027 ms 0.ae2.br1.bos30.alter.net (140.222.236.255) 19.795 ms 7 * * * 8 colo4-dalla.bear1.boston1.level3.net (4.53.61.86) 2205.648 ms 8.331 ms 13.161 ms 9 static-33-65-203-66.axsne.net (66.203.65.33) 16.951 ms 13.791 ms static-145-65-203-66.axsne.net (66.203.65.145) 21.503 ms 10 server1.ntcorp.com (207.154.13.58) 17.872 ms 15.902 ms 14.415 ms
Several things jump out:
1. alter.net is not a common path between here & there - usually a lower grade connection, when other backbones aren't working right
2. origin - alter.net - level.3 - endpoint is just bizarre, one would think that the regional FIOS network has a direct connection to level.3 (it also seems kind of odd that the packets are flowing from Acton MA, to Boston, and back out to Marlboro MA - there's an awful lot of fiber running along Rt. 495, and the networks are fairly dense around here)
3. The intermittent, high delays (factor of 10) jump out (also, when running ping tests, there seem to be intermittent periods of long sequences of timeouts)
All in all it's really mucking with both streaming services, and simply posting emails (SMTP timeouts).
All of which leads me to wonder if there's something mucked up with Verizon's routing tables (or a particular network interface).
Any insights (or fixes) to be had?
Thanks,
Miles Fidelman
-- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra Theory is when you know everything but nothing works. Practice is when everything works but no one knows why. In our lab, theory and practice are combined: nothing works and no one knows why. ... unknown
FYI: Funny thing, this morning, the packet loss rate has dropped to between 0 and 10% (from closer to 40%). Route seems to the same. One wonders if some piece of equipment has been updated. Miles Fidelman Miles Fidelman wrote:
Ok folks,
Thanks for the info about uunet. But that doesn't address:
3. The intermittent, high delays (factor of 10) jump out (also, when running ping tests, there seem to be intermittent periods of long sequences of timeouts) or, that, for about 4 years now, gamers seem to be reporting really poor performance across FIOS in the Northeast - tied to rather high packet loss rates.
Note that those packet losses seems to be bursts of 8-10 lost packets every 10 packets or so, and the really high delays on some of the traceroutes seem to indicate that it's happening somewhere in the middle of the path, not at my end.
And, come to think of it, that might explain some of the horrid performance of the FIOS channel guide.
Any thoughts? Anyone here from Verizon Northeast FIOS operations who might have a comment?
Thanks,
Miles Fidelman
Miles Fidelman wrote:
Any Verizon folks here?
I've been having some rather weird network issues lately - just reading email via IMAP, from home. Over a 1gig FIOS connection to a machine in a nearby Tierpoint data center that has LOTS of good connectivity.
I just tried some traceroutes, and got some interesting results:
These originate on a machine connected to a 1gig FIOS feed, and end at a machine, located in a Tierpoint datacenter, about 10 miles from here.
traceroute to ntcorp.com (207.154.13.58), 64 hops max, 52 byte packets 1 * fios_quantum_gateway (192.168.1.1) 3.530 ms 2.822 ms 2 * * * 3 100.41.27.110 (100.41.27.110) 14.970 ms 5.323 ms 6.306 ms 4 0.csi1.bstnmafr-mse01-bb-su1.alter.net (140.222.10.32) 11.069 ms 8.477 ms 17.097 ms 5 * * * 6 0.ae1.br1.bos30.alter.net (140.222.236.253) 17.121 ms 19.027 ms 0.ae2.br1.bos30.alter.net (140.222.236.255) 19.795 ms 7 * * * 8 colo4-dalla.bear1.boston1.level3.net (4.53.61.86) 2205.648 ms 8.331 ms 13.161 ms 9 static-33-65-203-66.axsne.net (66.203.65.33) 16.951 ms 13.791 ms static-145-65-203-66.axsne.net (66.203.65.145) 21.503 ms 10 server1.ntcorp.com (207.154.13.58) 17.872 ms 15.902 ms 14.415 ms
Several things jump out:
1. alter.net is not a common path between here & there - usually a lower grade connection, when other backbones aren't working right
2. origin - alter.net - level.3 - endpoint is just bizarre, one would think that the regional FIOS network has a direct connection to level.3 (it also seems kind of odd that the packets are flowing from Acton MA, to Boston, and back out to Marlboro MA - there's an awful lot of fiber running along Rt. 495, and the networks are fairly dense around here)
3. The intermittent, high delays (factor of 10) jump out (also, when running ping tests, there seem to be intermittent periods of long sequences of timeouts)
All in all it's really mucking with both streaming services, and simply posting emails (SMTP timeouts).
All of which leads me to wonder if there's something mucked up with Verizon's routing tables (or a particular network interface).
Any insights (or fixes) to be had?
Thanks,
Miles Fidelman
-- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra Theory is when you know everything but nothing works. Practice is when everything works but no one knows why. In our lab, theory and practice are combined: nothing works and no one knows why. ... unknown
participants (4)
-
Christopher Morrow -
Eric Kuhnke -
James Jun -
Miles Fidelman