Vadim, thank you for pointing this out. Or Network Engineers believe this is an issue with an optical repeater from our carrier. We will be doing intrusive testing to confirm. The second issue was a graveyard staff training issue and will be corrected as well. - Kevin -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Vadim Antonov Sent: Monday, February 07, 2000 4:07 AM To: nanog@merit.edu Subject: RR.COM has backbone problems I tried to contact their NOC, but the bozo on call didn't even want to hear what the problem is, advising me to contact my cable provider (how helpful). Please if there's anyone with a clue from RR.COM / MEDIAONE.NET or any other Time Warner holding - wake up, you've got a big problem. It doesn't show on your network monitors, but it is sure to screw your customers. And, please track down who was on the NOC telephone ((703) 345-3416) at this time (see message headers) and kick him hard for being an idiot and not taking a real trouble report. It appears that somehow long packets between boxes 24.218.189.197 and 24.218.188.118 get corrupted: kitty 67: ping -s1500 24.218.188.118 PING 24.218.188.118 (24.218.188.118): 1500 data bytes 1508 bytes from 24.218.188.118: icmp_seq=0 ttl=246 time=164.0 ms wrong data byte #1404 should be 0x7c but was 0xaa 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 <skipped> 1508 bytes from 24.218.188.118: icmp_seq=1 ttl=246 time=164.6 ms 1508 bytes from 24.218.188.118: icmp_seq=2 ttl=246 time=166.1 ms 1508 bytes from 24.218.188.118: icmp_seq=3 ttl=246 time=164.7 ms 1508 bytes from 24.218.188.118: icmp_seq=5 ttl=246 time=164.3 ms 1508 bytes from 24.218.188.118: icmp_seq=6 ttl=246 time=168.6 ms wrong data byte #1404 should be 0x7c but was 0xaa 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 <skipped> 1508 bytes from 24.218.188.118: icmp_seq=7 ttl=246 time=163.9 ms wrong data byte #924 should be 0x9c but was 0xaa 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 <skipped> kitty 70: ping -s1500 24.218.189.197 PING 24.218.189.197 (24.218.189.197): 1500 data bytes 1508 bytes from 24.218.189.197: icmp_seq=0 ttl=248 time=144.3 ms 1508 bytes from 24.218.189.197: icmp_seq=1 ttl=248 time=83.2 ms 1508 bytes from 24.218.189.197: icmp_seq=2 ttl=248 time=82.4 ms 1508 bytes from 24.218.189.197: icmp_seq=3 ttl=248 time=80.8 ms 1508 bytes from 24.218.189.197: icmp_seq=4 ttl=248 time=80.9 ms kitty 72: /usr/sbin/traceroute 24.218.189.170 traceroute: Warning: Multiple interfaces found; using 205.158.141.97 @ eth0 traceroute to 24.218.188.118 (24.218.188.118), 30 hops max, 40 byte packets 1 192.168.203.1 (192.168.203.1) 19.680 ms 19.540 ms 19.521 ms 2 mpk.core.rtr.wavepath.net (205.158.140.129) 19.825 ms 19.699 ms 21.328 ms 3 pop.access.rtr.wavepath.net (205.158.140.249) 21.097 ms 21.252 ms 21.030 ms 4 pop.border.rtr.wavepath.net (205.158.140.1) 21.204 ms 21.146 ms 23.449 ms 5 us-ca-scl-web1-e8.rtr.concentric.net (205.158.7.97) 22.883 ms 21.793 ms 22.879 ms 6 us-ca-scl-border2-fe4-0d0.rtr.concentric.net (207.88.95.9) 24.269 ms 101.210 ms 23.812 ms 7 us-ca-scl-core2-g5-0-0.rtr.concentric.net (207.88.1.3) 23.506 ms 24.663 ms 22.793 ms 8 PacBellrr.com (198.32.128.50) 23.761 ms 25.667 ms 26.280 ms 9 pabrt02-pabrt03.rr.com (24.218.189.197) 26.108 ms 57.919 ms 24.778 ms 10 24.218.188.118 (24.218.188.118) 110.427 ms * 116.969 ms 11 bsgsr01-nygsr01.rr.com (24.218.188.21) 116.396 ms 115.738 ms 116.053 ms 12 cambrt01-bsgsr01.rr.com (24.128.191.58) 113.668 ms 114.166 ms 115.836 ms 13 cmrtr01-srp3.ne.mediaone.net (24.218.189.170) 113.589 ms 114.720 ms 113.175 ms (Note that the problem affects all _transit_ packets): kitty 78: ping -s1500 24.218.188.21 PING 24.218.188.21 (24.218.188.21): 1500 data bytes 1508 bytes from 24.218.188.21: icmp_seq=1 ttl=245 time=525.0 ms 1508 bytes from 24.218.188.21: icmp_seq=2 ttl=245 time=171.2 ms 1508 bytes from 24.218.188.21: icmp_seq=3 ttl=245 time=173.8 ms 1508 bytes from 24.218.188.21: icmp_seq=4 ttl=245 time=174.3 ms wrong data byte #1404 should be 0x7c but was 0xaa 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 <skipped> Regards, --vadim