On Mon, Aug 25, 2014 at 1:54 PM, Ryan Ruuska <ryanruuska+nanog@gmail.com> wrote:
Hello all,
We have heard from many of our customers in the Northeast region, specifically PA, MD, and VA who have difficulty connecting to our website
there's probably vz customer folk on-list, maybe 'what should they test for you' would be nice? :)
(very slow loading times). We have noticed that if our data center provider specifically routes our outbound traffic over Hurricane Electric rather than XO Communications, that connectivity is restored to normal for our customers. The HE path goes from Salt Lake City to Denver where it is handed off to TeliaSonera, and they send it to Chicago where it gets handed off to Verizon. This works great and generally has a ping response of 20-25ms less than the bad route as posted below from one of our servers:
Tracing route to pool-108-52-xxx-xxx.phlapa.fios.verizon.net [108.52.xxx.xxx] over a maximum of 30 hops:
1 <1 ms <1 ms <1 ms 172.xxx.xxx.xxx (Internal IP) 2 <1 ms 10 ms <1 ms 209-41-xxx-xxx.c7dc.com [209.41.xxx.xxx] 3 <1 ms <1 ms <1 ms e1-5.rt08.gp1.c7dc.com [192.41.0.97] 4 2 ms 1 ms 1 ms ip65-46-51-113.z51-46-65.customer.algx.net [65.46.51.113] 5 19 ms 18 ms 19 ms vb1611.rar3.sanjose-ca.us.xo.net [216.156.0.5] 6 21 ms 18 ms 18 ms 207.88.14.226.ptr.us.xo.net [207.88.14.226] 7 18 ms 18 ms 18 ms 206.111.6.122.ptr.us.xo.net [206.111.6.122]
Verizon owned device with an XO IP address 8 87 ms 86 ms 87 ms b100.phlapa-lcr-22.verizon-gni.net [130.81.209.187] Next hop goes from CA to PA, probably just hidden routing info 9 * * * Request timed out. 10 88 ms 89 ms 90 ms pool-108-52-xxx-xxx.phlapa.fios.verizon.net [108.52.xxx.xxx]
As you can see, XO sends this traffic from Salt Lake City to San Jose where it is handed off to Verizon. We've worked with XO and determined that there are no connectivity issues along their path, which seems to point to an issue within Verizon's network. I have a couple of tickets open with Verizon at the moment on behalf of some of our customers, but we have had trouble breaking through Tier 1.
Our first thought was saturation of the peering point, but XO states that the link between hops 6 and 7 is using 5Gbps out of 20Gbps capacity, and the latency on hop 7 (the Verizon owned device with an XO IP) seems normal whenever we test.
Any Verizon engineers out there willing to take a quick peek at this route for any obvious issues? It would be a lot easier for us if Verizon provided Looking Glass servers, but alas...
they have route data sent to routeviews at least...