From the traces provided the packet loss starts accruing at hop 4 and 5 and
Terri, Except the initial traces show that hop 6 is clean. If the problem really started at hop 4 then the issue would likely show up in hop 6 (unless there was a hashing on the return path based on the far end router's source IP that caused just some traffic to be thrown away). Hop 4 and 5 always have packet loss - I believe it's due to ICMPv6 rate limiters protecting their CPU: Frank From: Fullenkamp, Terri L [mailto:terri.l.fullenkamp@xo.com] Sent: Sunday, February 28, 2016 2:59 PM To: Frank Bulk <frnkblk@iname.com>; nanog@nanog.org Cc: # SUP - NCC Level 1 Data <NCCLevel1Data@xo.com> Subject: RE: Packet loss on XO's network that is before it gets into the XO network, If you are a customer of XO please contact care @ 888-575-6398 with any other questions Trace from hop 6 tfullenk@MCR2.Minneapolis-MN <mailto:tfullenk@MCR2.Minneapolis-MN> > traceroute enterprise.com traceroute6 to enterprise.com (2620:103:e049:31a7::b4) from 2610:18::30bc, 64 hops max, 12 byte packets 1 mcr1.minneapolis-mn.us.xo.net (2610:18::30b8) 29.325 ms 28.950 ms 31.327 ms MPLS Label=318081 CoS=0 TTL=1 S=0 MPLS Label=2 CoS=0 TTL=1 S=1 2 2610:18::5802 (2610:18::5802) 32.680 ms 29.684 ms 32.643 ms MPLS Label=19268 CoS=0 TTL=255 S=0 MPLS Label=2 CoS=0 TTL=255 S=1 3 2610:18::5804 (2610:18::5804) 30.629 ms 33.384 ms 81.985 ms MPLS Label=757571 CoS=0 TTL=1 S=0 MPLS Label=2 CoS=0 TTL=3 S=1 4 2610:18::5803 (2610:18::5803) 65.305 ms 28.876 ms 39.141 ms MPLS Label=419427 CoS=0 TTL=1 S=0 MPLS Label=2 CoS=0 TTL=4 S=1 5 2610:18::5201 (2610:18::5201) 36.762 ms 28.776 ms 28.718 ms MPLS Label=694064 CoS=0 TTL=1 S=0 MPLS Label=2 CoS=0 TTL=5 S=1 6 2610:18::2070 (2610:18::2070) 29.487 ms 28.920 ms 42.500 ms 7 2610:18:16::32 (2610:18:16::32) 28.619 ms 2600:803:2::9 (2600:803:2::9) 28.706 ms 28.692 ms 8 2600:807:21f::22 (2600:807:21f::22) 47.997 ms 48.042 ms 50.504 ms 9 2600:807:21f::22 (2600:807:21f::22) 48.026 ms 48.084 ms 49.172 ms Terri Fullenkamp NCC Specialist V / IP Data / <http://www.xo.com/> XO Communications / 2020 Westport Center Drive, Maryland Heights, MO 63146 T: 866-966-8975 F: 314-787- 7799 E: terri.l.fullenkamp@xo.com <mailto:terri.l.fullenkamp@xo.com> From: Frank Bulk [mailto:frnkblk@iname.com] Sent: Sunday, February 28, 2016 12:25 To: nanog@nanog.org <mailto:nanog@nanog.org> Subject: Packet loss on XO's network Since ~11:08 am Central our monitoring system has been reporting intermittent HTTPv6 timeouts to www.sprint.net <http://www.sprint.net> , www.qwest.com <http://www.qwest.com> , enterprise.com, and enterprise.ca. It goes bad for a while and then clears. Using mtr, these all have XO's network in common, and the packet loss to the endpoint sustains over 40%. Seems to be between 2610:18:18b:c000::11 and mcr1.minneapolis-mn.us.xo.net [2610:18::30b8] Frank To enterprise.com: Host Loss% Snt Last Avg Best Wrst StDev 1. router-core.mtcnet.net 0.0% 929 0.3 0.6 0.2 37.6 2.5 2. sxct.sxcy.mtcnet.net 0.0% 929 0.3 0.3 0.2 11.2 0.6 3. v6-premier.sxcy-mlx.fbnt.netins.net 0.0% 929 1.6 1.5 1.5 12.6 0.9 4. v6-ins-db4-te-0-6-0-4-219.desm.netins.net 52.4% 929 8.1 8.0 7.8 10.5 0.3 5. v6-ins-dc1-te-7-4.desm.netins.net 52.4% 929 7.9 8.0 7.6 10.9 0.5 6. 2610:18:18b:c000::11 0.0% 929 13.9 13.2 12.8 46.8 3.0 7. mcr1.minneapolis-mn.us.xo.net 50.3% 929 54.7 71.2 54.6 86.7 7.7 8. 2610:18::5802 50.8% 929 55.7 72.8 54.6 84.3 7.8 9. 2610:18::5202 51.4% 929 54.4 72.9 54.4 85.0 8.0 2610:18::5804 10. 2610:18::2070 52.3% 928 54.4 71.5 54.4 90.7 7.9 2610:18::5304 11. 2600:803:2::9 41.6% 928 54.5 56.8 47.8 96.7 3.4 2610:18::5303 12. 2610:18::5200 96.0% 928 54.5 55.8 54.4 75.6 3.7 13. 2610:18::5202 96.0% 928 55.5 56.6 54.6 58.5 1.2 14. 2610:18::2070 96.0% 928 61.8 55.1 54.6 61.8 1.2 15. 2600:803:2::9 95.9% 928 45.0 45.1 45.0 48.0 0.5 16. ??? To www.sprint.net <http://www.sprint.net> : Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. router-core.mtcnet.net 0.0% 846 0.4 0.8 0.2 91.5 3.9 2. sxct.sxcy.mtcnet.net 0.0% 846 0.9 0.3 0.2 10.3 0.4 3. v6-premier.sxcy-mlx.fbnt.netins.net 0.0% 846 1.5 1.6 1.5 12.0 0.8 4. v6-ins-db4-te-0-6-0-4-219.desm.netins.net 55.4% 846 7.9 8.1 7.8 14.8 0.4 5. v6-ins-dc1-te-7-4.desm.netins.net 54.3% 846 7.8 8.2 7.6 19.2 1.0 6. 2610:18:18b:c000::11 0.0% 846 12.9 14.0 12.8 73.0 5.4 7. mcr1.minneapolis-mn.us.xo.net 49.8% 846 54.7 70.0 54.6 100.1 8.1 8. 2610:18::5802 47.4% 846 58.3 71.9 54.6 82.9 8.2 9. 2610:18::5202 45.0% 846 54.4 72.1 54.4 83.3 8.4 2610:18::5804 10. 2610:18::2070 50.6% 845 54.4 70.4 54.3 88.8 8.1 2610:18::5304 11. sl-st31-ash-te0-15-2-0.v6.sprintlink.net 43.4% 845 57.5 59.1 53.1 80.5 2.3 2610:18::5303 12. sl-crs3-dc-be13.v6.sprintlink.net 46.7% 845 54.6 64.7 54.4 69.1 3.7 2610:18::5200 13. sl-crs1-ffx-be2.v6.sprintlink.net 44.1% 844 55.0 71.8 55.0 79.7 5.3 2610:18::5202 14. sl-crs1-orl-bu-1.v6.sprintlink.net 41.7% 844 54.9 75.7 54.7 96.3 7.4 2610:18::2070 15. sl-lkdstr2-p1-0.v6.sprintlink.net 41.0% 844 46.4 73.5 45.8 81.7 7.0 sl-st31-ash-te0-15-2-0.v6.sprintlink.net 16. www.sprint.net <http://www.sprint.net> 43.1% 844 54.5 74.6 53.5 81.6 5.0 sl-crs3-dc-be13.v6.sprintlink.net