Verizon connectivity issues
Hello all, We have heard from many of our customers in the Northeast region, specifically PA, MD, and VA who have difficulty connecting to our website (very slow loading times). We have noticed that if our data center provider specifically routes our outbound traffic over Hurricane Electric rather than XO Communications, that connectivity is restored to normal for our customers. The HE path goes from Salt Lake City to Denver where it is handed off to TeliaSonera, and they send it to Chicago where it gets handed off to Verizon. This works great and generally has a ping response of 20-25ms less than the bad route as posted below from one of our servers: Tracing route to pool-108-52-xxx-xxx.phlapa.fios.verizon.net [108.52.xxx.xxx] over a maximum of 30 hops: 1 <1 ms <1 ms <1 ms 172.xxx.xxx.xxx (Internal IP) 2 <1 ms 10 ms <1 ms 209-41-xxx-xxx.c7dc.com [209.41.xxx.xxx] 3 <1 ms <1 ms <1 ms e1-5.rt08.gp1.c7dc.com [192.41.0.97] 4 2 ms 1 ms 1 ms ip65-46-51-113.z51-46-65.customer.algx.net [65.46.51.113] 5 19 ms 18 ms 19 ms vb1611.rar3.sanjose-ca.us.xo.net [216.156.0.5] 6 21 ms 18 ms 18 ms 207.88.14.226.ptr.us.xo.net [207.88.14.226] 7 18 ms 18 ms 18 ms 206.111.6.122.ptr.us.xo.net [206.111.6.122]
Verizon owned device with an XO IP address 8 87 ms 86 ms 87 ms b100.phlapa-lcr-22.verizon-gni.net [130.81.209.187] Next hop goes from CA to PA, probably just hidden routing info 9 * * * Request timed out. 10 88 ms 89 ms 90 ms pool-108-52-xxx-xxx.phlapa.fios.verizon.net [108.52.xxx.xxx]
As you can see, XO sends this traffic from Salt Lake City to San Jose where it is handed off to Verizon. We've worked with XO and determined that there are no connectivity issues along their path, which seems to point to an issue within Verizon's network. I have a couple of tickets open with Verizon at the moment on behalf of some of our customers, but we have had trouble breaking through Tier 1. Our first thought was saturation of the peering point, but XO states that the link between hops 6 and 7 is using 5Gbps out of 20Gbps capacity, and the latency on hop 7 (the Verizon owned device with an XO IP) seems normal whenever we test. Any Verizon engineers out there willing to take a quick peek at this route for any obvious issues? It would be a lot easier for us if Verizon provided Looking Glass servers, but alas... Thank you, Ryan
On Mon, Aug 25, 2014 at 1:54 PM, Ryan Ruuska <ryanruuska+nanog@gmail.com> wrote:
Hello all,
We have heard from many of our customers in the Northeast region, specifically PA, MD, and VA who have difficulty connecting to our website
there's probably vz customer folk on-list, maybe 'what should they test for you' would be nice? :)
(very slow loading times). We have noticed that if our data center provider specifically routes our outbound traffic over Hurricane Electric rather than XO Communications, that connectivity is restored to normal for our customers. The HE path goes from Salt Lake City to Denver where it is handed off to TeliaSonera, and they send it to Chicago where it gets handed off to Verizon. This works great and generally has a ping response of 20-25ms less than the bad route as posted below from one of our servers:
Tracing route to pool-108-52-xxx-xxx.phlapa.fios.verizon.net [108.52.xxx.xxx] over a maximum of 30 hops:
1 <1 ms <1 ms <1 ms 172.xxx.xxx.xxx (Internal IP) 2 <1 ms 10 ms <1 ms 209-41-xxx-xxx.c7dc.com [209.41.xxx.xxx] 3 <1 ms <1 ms <1 ms e1-5.rt08.gp1.c7dc.com [192.41.0.97] 4 2 ms 1 ms 1 ms ip65-46-51-113.z51-46-65.customer.algx.net [65.46.51.113] 5 19 ms 18 ms 19 ms vb1611.rar3.sanjose-ca.us.xo.net [216.156.0.5] 6 21 ms 18 ms 18 ms 207.88.14.226.ptr.us.xo.net [207.88.14.226] 7 18 ms 18 ms 18 ms 206.111.6.122.ptr.us.xo.net [206.111.6.122]
Verizon owned device with an XO IP address 8 87 ms 86 ms 87 ms b100.phlapa-lcr-22.verizon-gni.net [130.81.209.187] Next hop goes from CA to PA, probably just hidden routing info 9 * * * Request timed out. 10 88 ms 89 ms 90 ms pool-108-52-xxx-xxx.phlapa.fios.verizon.net [108.52.xxx.xxx]
As you can see, XO sends this traffic from Salt Lake City to San Jose where it is handed off to Verizon. We've worked with XO and determined that there are no connectivity issues along their path, which seems to point to an issue within Verizon's network. I have a couple of tickets open with Verizon at the moment on behalf of some of our customers, but we have had trouble breaking through Tier 1.
Our first thought was saturation of the peering point, but XO states that the link between hops 6 and 7 is using 5Gbps out of 20Gbps capacity, and the latency on hop 7 (the Verizon owned device with an XO IP) seems normal whenever we test.
Any Verizon engineers out there willing to take a quick peek at this route for any obvious issues? It would be a lot easier for us if Verizon provided Looking Glass servers, but alas...
they have route data sent to routeviews at least...
Hi Chris! @Ryan, XO was accurate, the peering connection between 6 and 7 is not congested, nor is it taking any sort of errors. I took a look. The significant increase in rtt seems to happen between 7 and 8 (an Verizon LSP consisting of multiple physical routers). That hop traverses the continental US from San Jose to Philadelphia, so you would expect that, and I don't see anything going on in that path. The end to end rtt is pretty normal, and in any case, it shouldn't prevent your customers from connecting. I am not seeing any packet loss on that path. It's possible that whatever they saw was a transient issue and may be over now. So, please double check with your customers to verify that they still have problems. If so, continue to escalate the ticket. -- rj -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Christopher Morrow Sent: Wednesday, August 27, 2014 10:52 PM To: ryanruuska+nanog@gmail.com Cc: nanog list Subject: Re: Verizon connectivity issues On Mon, Aug 25, 2014 at 1:54 PM, Ryan Ruuska <ryanruuska+nanog@gmail.com> wrote:
Hello all,
We have heard from many of our customers in the Northeast region, specifically PA, MD, and VA who have difficulty connecting to our website
there's probably vz customer folk on-list, maybe 'what should they test for you' would be nice? :)
(very slow loading times). We have noticed that if our data center provider specifically routes our outbound traffic over Hurricane Electric rather than XO Communications, that connectivity is restored to normal for our customers. The HE path goes from Salt Lake City to Denver where it is handed off to TeliaSonera, and they send it to Chicago where it gets handed off to Verizon. This works great and generally has a ping response of 20-25ms less than the bad route as posted below from one of our servers:
Tracing route to pool-108-52-xxx-xxx.phlapa.fios.verizon.net [108.52.xxx.xxx] over a maximum of 30 hops:
1 <1 ms <1 ms <1 ms 172.xxx.xxx.xxx (Internal IP) 2 <1 ms 10 ms <1 ms 209-41-xxx-xxx.c7dc.com [209.41.xxx.xxx] 3 <1 ms <1 ms <1 ms e1-5.rt08.gp1.c7dc.com [192.41.0.97] 4 2 ms 1 ms 1 ms ip65-46-51-113.z51-46-65.customer.algx.net [65.46.51.113] 5 19 ms 18 ms 19 ms vb1611.rar3.sanjose-ca.us.xo.net [216.156.0.5] 6 21 ms 18 ms 18 ms 207.88.14.226.ptr.us.xo.net [207.88.14.226] 7 18 ms 18 ms 18 ms 206.111.6.122.ptr.us.xo.net [206.111.6.122]
Verizon owned device with an XO IP address 8 87 ms 86 ms 87 ms b100.phlapa-lcr-22.verizon-gni.net [130.81.209.187] Next hop goes from CA to PA, probably just hidden routing info 9 * * * Request timed out. 10 88 ms 89 ms 90 ms pool-108-52-xxx-xxx.phlapa.fios.verizon.net [108.52.xxx.xxx]
As you can see, XO sends this traffic from Salt Lake City to San Jose where it is handed off to Verizon. We've worked with XO and determined that there are no connectivity issues along their path, which seems to point to an issue within Verizon's network. I have a couple of tickets open with Verizon at the moment on behalf of some of our customers, but we have had trouble breaking through Tier 1.
Our first thought was saturation of the peering point, but XO states that the link between hops 6 and 7 is using 5Gbps out of 20Gbps capacity, and the latency on hop 7 (the Verizon owned device with an XO IP) seems normal whenever we test.
Any Verizon engineers out there willing to take a quick peek at this route for any obvious issues? It would be a lot easier for us if Verizon provided Looking Glass servers, but alas...
they have route data sent to routeviews at least...
On Thu, Aug 28, 2014 at 10:18 AM, Bacon, Ricky (RJ) <rj.bacon@verizon.com> wrote:
Hi Chris!
howdy!
@Ryan, XO was accurate, the peering connection between 6 and 7 is not
I had thought Ryan wasn't at XO but an XO customer... I could be mistaken though. congested, nor is it taking any sort of errors.
I took a look. The significant increase in rtt seems to happen between 7 and 8 (an Verizon LSP consisting of multiple physical routers). That hop traverses the
yikes :) lsp's from XT -> LCR now, neat. it's also possible (and likely) that the return path from VZ customer -> Ryan is across either the NYC or DCA links to XO (nyc4 or dca6?)... I agree that the ~65ms cross-country seems on-par with SJC -> PHL on a good day though. continental US from San Jose to Philadelphia, so you would expect that, and I don't see anything going on in that path. The end to end rtt is pretty normal, and in any case, it shouldn't prevent your customers from connecting. I am not seeing any packet loss on that path. It's possible that whatever they saw was a transient issue and may be over now.
So, please double check with your customers to verify that they still have problems. If so, continue to escalate the ticket.
hopefully ryan's having a better day today :)
-- rj
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Christopher Morrow Sent: Wednesday, August 27, 2014 10:52 PM To: ryanruuska+nanog@gmail.com Cc: nanog list Subject: Re: Verizon connectivity issues
On Mon, Aug 25, 2014 at 1:54 PM, Ryan Ruuska <ryanruuska+nanog@gmail.com> wrote:
Hello all,
We have heard from many of our customers in the Northeast region, specifically PA, MD, and VA who have difficulty connecting to our website
there's probably vz customer folk on-list, maybe 'what should they test for you' would be nice? :)
(very slow loading times). We have noticed that if our data center provider specifically routes our outbound traffic over Hurricane Electric rather than XO Communications, that connectivity is restored to normal for our customers. The HE path goes from Salt Lake City to Denver where it is handed off to TeliaSonera, and they send it to Chicago where it gets handed off to Verizon. This works great and generally has a ping response of 20-25ms less than the bad route as posted below from one of our servers:
Tracing route to pool-108-52-xxx-xxx.phlapa.fios.verizon.net [108.52.xxx.xxx] over a maximum of 30 hops:
1 <1 ms <1 ms <1 ms 172.xxx.xxx.xxx (Internal IP) 2 <1 ms 10 ms <1 ms 209-41-xxx-xxx.c7dc.com [209.41.xxx.xxx] 3 <1 ms <1 ms <1 ms e1-5.rt08.gp1.c7dc.com [192.41.0.97] 4 2 ms 1 ms 1 ms ip65-46-51-113.z51-46-65.customer.algx.net [65.46.51.113] 5 19 ms 18 ms 19 ms vb1611.rar3.sanjose-ca.us.xo.net [216.156.0.5] 6 21 ms 18 ms 18 ms 207.88.14.226.ptr.us.xo.net [207.88.14.226] 7 18 ms 18 ms 18 ms 206.111.6.122.ptr.us.xo.net [206.111.6.122]
Verizon owned device with an XO IP address 8 87 ms 86 ms 87 ms b100.phlapa-lcr-22.verizon-gni.net [130.81.209.187] Next hop goes from CA to PA, probably just hidden routing info 9 * * * Request timed out. 10 88 ms 89 ms 90 ms pool-108-52-xxx-xxx.phlapa.fios.verizon.net [108.52.xxx.xxx]
As you can see, XO sends this traffic from Salt Lake City to San Jose where it is handed off to Verizon. We've worked with XO and determined that there are no connectivity issues along their path, which seems to point to an issue within Verizon's network. I have a couple of tickets open with Verizon at the moment on behalf of some of our customers, but we have had trouble breaking through Tier 1.
Our first thought was saturation of the peering point, but XO states that the link between hops 6 and 7 is using 5Gbps out of 20Gbps capacity, and the latency on hop 7 (the Verizon owned device with an XO IP) seems normal whenever we test.
Any Verizon engineers out there willing to take a quick peek at this route for any obvious issues? It would be a lot easier for us if Verizon provided Looking Glass servers, but alas...
they have route data sent to routeviews at least...
participants (3)
-
Bacon, Ricky (RJ) -
Christopher Morrow -
Ryan Ruuska