I recommend to 'ping' every host in 'traceroute' chain and check _when_ does this duping appear. It can result from some important misconfiguration in your own network, but can be result of some unimportant effect. Anyway, it looks suspiciows. I'v checked this addresses, and can't see 'dup's in my 'ping tests'. Looks as the dup's source somewhere in your network. For example, it can be ccaused by the broken interface card in your router, or by the broken switch, or result from the internal loop in your (or some intermediate) LAN. You should pay attention to this effect - in can indicate a serious lost of throughput. Run 'tcpdump', open some 'ftp' session and look - if you see dup TCP packets you can get complain to your provider (or youself) about _bad network behaviour_. Alex. On Mon, 15 Mar 1999, cache22 wrote:
Date: Mon, 15 Mar 1999 13:33:47 -0500 From: cache22 <cache22@webkorner.com> To: nanog@merit.edu Subject: dup packets
i reported a problem with packet duplication to the gridnet (worldcom) noc, and they checked it and they said everything is fine, some hosts give duplicate echo replies to pings. i get either one or three duplicates. are there any hosts, routers, firewalls, load balancers, etc. that do this? i couldn't find anything in the archives on this subject.
when i tried pinging from locations on other backbones i didn't receive dups so i feel the problem is within gridnet. i am reporting here as i would like to know if any others are suffering this problem. also, many hosts and networks may be receiving unnecessary traffic from duplicate acks as there is duplication of tcp packets also.
most of the problems are through connections from east to west coast on atm circuits--see traces below.
other unfriendly behavior--their default route appears to be pb-nap.eni.net. most unroutable packets, including reserved and rfc1918, are routed there--traces below. this morning, when g.root-servers.net was unreachable for a few minutes (via gridnet, did anyone else see it disappear?) i watched those packets go to eni. (sorry, didn't capture the trace.) are they selling /dev/rj0 service?
this (everything above) has been happening for at least several months.
matt sommer sysadmin webkorner.com
PING route-server.cerf.net (192.215.254.5): 56 data bytes 64 bytes from 192.215.254.5: icmp_seq=0 ttl=248 time=150.6 ms 64 bytes from 192.215.254.5: icmp_seq=0 ttl=248 time=151.1 ms (DUP!) 64 bytes from 192.215.254.5: icmp_seq=0 ttl=248 time=151.9 ms (DUP!) 64 bytes from 192.215.254.5: icmp_seq=0 ttl=248 time=152.4 ms (DUP!) 64 bytes from 192.215.254.5: icmp_seq=1 ttl=248 time=129.1 ms
--- route-server.cerf.net ping statistics --- 2 packets transmitted, 2 packets received, +3 duplicates, 0% packet loss round-trip min/avg/max = 129.1/147.0/152.4 ms
traceroute to route-server.cerf.net (192.215.254.5), 30 hops max, 40 byte packets 1 nemo.webkorner.com (207.53.77.1) 3.039 ms 2.013 ms 1.728 ms 2 core-atln1-hssi4/0.9.grid.net (207.205.2.1) 16.749 ms 19.756 ms 17.061 ms 3 core-snfx1-atm9-0.2.grid.net (206.80.170.70) 79.949 ms 81.523 ms 81.564 ms 4 snfc1.grid.net (198.32.136.48) 79.668 ms 93.924 ms 79.515 ms 5 atm8-0-155M.sjc-bb3.cerf.net (134.24.29.38) 97.64 ms 90.751 ms f1-0-0.sjc-bb1.cerf.net (134.24.88.55) 106.49 ms 6 atm0-0-155M.sfo-bb2.cerf.net (134.24.29.21) 114.862 ms atm8-0-155M.sjc-bb3.cerf.net (134.24.29.38) 101.439 ms atm0-0-155M.sfo-bb2.cerf.net (134.24.29.21) 105.387 ms 7 pos0-2-155M.sfo-bb3.cerf.net (134.24.29.197) 93.883 ms 97.39 ms 97.007 ms 8 atm10-0-155M.lax-bb1.cerf.net (134.24.29.41) 110.238 ms 113.954 ms 107.834 ms 9 atm1-0-1-622M.san-bb6.cerf.net (134.24.29.141) 106.82 ms atm10-0-155M.lax-bb1.cerf.net (134.24.29.41) 190.6 ms atm1-0-1-622M.san-bb6.cerf.net (134.24.29.141) 135.938 ms 10 pos10-0-0-155M.san-bb2.cerf.net (134.24.29.105) 114.09 ms 100.856 ms 103.303 ms 11 pos1-0-155M.san-hq5.cerf.net (134.24.29.54) 112.1 ms 118.083 ms 104.277 ms 12 pos1-0-155M.san-hq5.cerf.net (134.24.29.54) 105.645 ms 106.927 ms 113.921 ms 13 * route-server.cerf.net (192.215.254.5) 122.29 ms *
PING cerf.net (192.102.249.3): 56 data bytes 64 bytes from 192.102.249.3: icmp_seq=0 ttl=247 time=178.2 ms 64 bytes from 192.102.249.3: icmp_seq=0 ttl=247 time=178.9 ms (DUP!) 64 bytes from 192.102.249.3: icmp_seq=0 ttl=247 time=179.2 ms (DUP!) 64 bytes from 192.102.249.3: icmp_seq=0 ttl=247 time=179.7 ms (DUP!) 64 bytes from 192.102.249.3: icmp_seq=1 ttl=247 time=126.5 ms
--- cerf.net ping statistics --- 2 packets transmitted, 2 packets received, +3 duplicates, 0% packet loss round-trip min/avg/max = 126.5/168.5/179.7 ms
traceroute to cerf.net (192.102.249.3), 30 hops max, 40 byte packets 1 nemo.webkorner.com (207.53.77.1) 1.978 ms 1.959 ms 1.724 ms 2 core-atln1-hssi4/0.9.grid.net (207.205.2.1) 29.475 ms 17.064 ms 18.406 ms 3 core-snfx1-atm9-0.2.grid.net (206.80.170.70) 80.843 ms 89.331 ms 82.987 ms 4 snfc1.grid.net (198.32.136.48) 85.938 ms 78.285 ms 82.579 ms 5 f0-0-0.sjc-bb2.cerf.net (134.24.88.5) 85.739 ms 88.796 ms 90.437 ms 6 pos0-0-155M.san-bb6.cerf.net (134.24.29.130) 104.53 ms 102.469 ms 102.327 ms 7 pos0-0-155M.san-hq5.cerf.net (134.24.29.113) 102.044 ms pos0-0-155M.san-bb6.cerf.net (134.24.29.130) 100.354 ms pos0-0-155M.san-hq5.cerf.net (134.24.29.113) 101.231 ms 8 pos0-0-155M.san-hq5.cerf.net (134.24.29.113) 103.14 ms 98.93 ms 100.854 ms 9 pos9-0-0-155M.san-hq1.cerf.net (134.24.29.126) 108.44 ms 104.816 ms 104.518 ms 10 nic.cerf.net (192.102.249.3) 102.09 ms 110.655 ms 103.239 ms
PING cisco.com (192.31.7.130): 56 data bytes 64 bytes from 192.31.7.130: icmp_seq=0 ttl=246 time=89.0 ms 64 bytes from 192.31.7.130: icmp_seq=0 ttl=246 time=89.5 ms (DUP!) 64 bytes from 192.31.7.130: icmp_seq=0 ttl=246 time=90.0 ms (DUP!) 64 bytes from 192.31.7.130: icmp_seq=0 ttl=246 time=90.5 ms (DUP!) 64 bytes from 192.31.7.130: icmp_seq=1 ttl=246 time=104.7 ms
--- cisco.com ping statistics --- 2 packets transmitted, 2 packets received, +3 duplicates, 0% packet loss round-trip min/avg/max = 89.0/92.7/104.7 ms
traceroute to cisco.com (192.31.7.130), 30 hops max, 40 byte packets 1 nemo.webkorner.com (207.53.77.1) 1.946 ms 5.717 ms 2.183 ms 2 core-atln1-hssi4/0.9.grid.net (207.205.2.1) 16.692 ms 25.742 ms 40.845 ms 3 core-snfx1-atm5-0.4.grid.net (206.80.187.29) 98.354 ms 85.901 ms 82.039 ms 4 snfc1.grid.net (198.32.136.48) 79.189 ms 107.082 ms 79.548 ms 5 p2-0.sanjose1-nbr1.bbnplanet.net (4.0.3.193) 85.506 ms 100.916 ms 105.923 ms 6 p2-0.sanjose1-nbr1.bbnplanet.net (4.0.3.193) 92.522 ms 96.826 ms 100.667 ms 7 p4-0.paloalto-nbr2.bbnplanet.net (4.0.1.1) 91.881 ms 110.78 ms 106.679 ms 8 h1-0.cisco.bbnplanet.net (131.119.26.10) 95.844 ms 89.91 ms 89.474 ms 9 h1-0.cisco.bbnplanet.net (131.119.26.10) 92.13 ms pigpen.cisco.com (192.31.7.40) 98.463 ms h1-0.cisco.bbnplanet.net (131.119.26.10) 114.123 ms 10 cio-sys.cisco.com (192.31.7.130) 97.249 ms pigpen.cisco.com (192.31.7.40) 99.075 ms 95.722 ms
PING enmu.edu (192.94.216.250): 56 data bytes 64 bytes from 192.94.216.250: icmp_seq=0 ttl=51 time=322.1 ms 64 bytes from 192.94.216.250: icmp_seq=0 ttl=51 time=322.7 ms (DUP!) 64 bytes from 192.94.216.250: icmp_seq=1 ttl=51 time=239.1 ms
--- enmu.edu ping statistics --- 2 packets transmitted, 2 packets received, +1 duplicates, 0% packet loss round-trip min/avg/max = 239.1/294.6/322.7 ms
traceroute to enmu.edu (192.94.216.250), 30 hops max, 40 byte packets 1 nemo.webkorner.com (207.53.77.1) 2.014 ms 1.927 ms 2.617 ms 2 core-atln1-hssi4/0.9.grid.net (207.205.2.1) 19.161 ms 27.238 ms 16.684 ms 3 core-snfx1-atm5-0.4.grid.net (206.80.187.29) 81.792 ms 82.091 ms 80.942 ms 4 t8-0.San-Francisco.t3.ans.net (198.32.128.10) 95.598 ms 83.259 ms 83.83 ms 5 pacbell-nap.ans.net (198.32.128.67) 110.493 ms 97.621 ms 85.529 ms 6 h13-1.t16-0.Los-Angeles.t3.ans.net (140.223.9.14) 106.263 ms 119.991 ms 99.344 ms 7 h14-1.t112-0.Albuquerque.t3.ans.net (140.223.17.10) 136.081 ms 120.616 ms 119.887 ms 8 f0-0.cnss116.Albuquerque.t3.ans.net (140.222.112.196) 138.032 ms 122.725 ms 125.71 ms 9 h1-0.enss191.t3.ans.net (192.103.74.42) 126.823 ms 131.757 ms 130.943 ms 10 198.83.5.4 (198.83.5.4) 158.435 ms 144.432 ms 154.82 ms 11 192.65.78.82 (192.65.78.82) 175.607 ms 160.103 ms 168.386 ms 12 EM0X.ENMU.EDU (192.94.216.250) 244.882 ms 172.999 ms 183.95 ms
PING e.root-servers.net (192.203.230.10): 56 data bytes 64 bytes from 192.203.230.10: icmp_seq=0 ttl=251 time=112.5 ms 64 bytes from 192.203.230.10: icmp_seq=0 ttl=251 time=114.3 ms (DUP!) 64 bytes from 192.203.230.10: icmp_seq=1 ttl=251 time=117.3 ms
--- e.root-servers.net ping statistics --- 2 packets transmitted, 2 packets received, +1 duplicates, 0% packet loss round-trip min/avg/max = 112.5/114.7/117.3 ms
traceroute to e.root-servers.net (192.203.230.10), 30 hops max, 40 byte packets 1 nemo.webkorner.com (207.53.77.1) 2.072 ms 2.964 ms 1.746 ms 2 core-atln1-hssi4/0.9.grid.net (207.205.2.1) 17.125 ms 17.129 ms 16.715 ms 3 core-wash1-atm4/0.4.grid.net (206.80.188.33) 81.316 ms 69.578 ms 81.366 ms 4 mae-east.nsn.nasa.gov (192.41.177.125) 98.22 ms 87.618 ms 91.117 ms 5 s-ARC1-ATM.NSN.NASA.GOV (128.161.10.1) 116.303 ms 112.699 ms 113.355 ms 6 E.ROOT-SERVERS.NET (192.203.230.10) 123.275 ms * 111.486 ms
PING l.root-servers.net (198.32.64.12): 56 data bytes 64 bytes from 198.32.64.12: icmp_seq=0 ttl=244 time=174.9 ms 64 bytes from 198.32.64.12: icmp_seq=0 ttl=244 time=175.1 ms (DUP!) 64 bytes from 198.32.64.12: icmp_seq=1 ttl=244 time=157.6 ms
--- l.root-servers.net ping statistics --- 2 packets transmitted, 2 packets received, +1 duplicates, 0% packet loss round-trip min/avg/max = 157.6/169.2/175.1 ms
traceroute to l.root-servers.net (198.32.64.12), 30 hops max, 40 byte packets 1 nemo.webkorner.com (207.53.77.1) 3.558 ms 3.419 ms 1.765 ms 2 core-atln1-hssi4/0.9.grid.net (207.205.2.1) 18.912 ms 25.015 ms 17.186 ms 3 core-wash1-atm8-0.3.grid.net (206.80.180.34) 94.471 ms 87.77 ms 95.625 ms 4 * f6.peer1.wdc1.genuity.net (192.41.177.147) 109.531 ms 140.935 ms 5 h5-0-1.core1.wdc1.genuity.net (207.240.1.233) 136.672 ms * 154.67 ms 6 * core1.lax1.genuity.net (207.240.0.5) 169.329 ms 181.559 ms 7 fe-5-0.peer1.lax1.genuity.net (207.240.1.142) 244.746 ms 164.322 ms 184.965 ms 8 sandbox.ep.net (198.32.146.11) 168.058 ms 159.353 ms * 9 * l.root-servers.net (198.32.64.12) 159.825 ms 170.084 ms
PING altavista.com (204.123.2.66): 56 data bytes 64 bytes from 204.123.2.66: icmp_seq=0 ttl=54 time=96.8 ms 64 bytes from 204.123.2.66: icmp_seq=0 ttl=54 time=97.3 ms (DUP!) 64 bytes from 204.123.2.66: icmp_seq=1 ttl=54 time=99.8 ms
--- altavista.com ping statistics --- 2 packets transmitted, 2 packets received, +1 duplicates, 0% packet loss round-trip min/avg/max = 96.8/97.9/99.8 ms
traceroute to altavista.com (204.123.2.69), 30 hops max, 40 byte packets 1 nemo.webkorner.com (207.53.77.1) 1.96 ms 3.553 ms 1.782 ms 2 core-atln1-hssi4/0.9.grid.net (207.205.2.1) 16.567 ms 19.488 ms 21.282 ms 3 peer-atl-uunet-fddi.grid.net (206.80.190.201) 25.164 ms 17.09 ms 61.332 ms 4 104.ATM2-0.XR1.ATL1.ALTER.NET (146.188.232.50) 18.564 ms 17.982 ms 17.676 ms 5 295.ATM2-0.TR1.ATL1.ALTER.NET (146.188.232.90) 17.736 ms 86.312 ms 36.625 ms 6 109.ATM6-0.TR1.SCL1.ALTER.NET (146.188.136.66) 74.736 ms 80.162 ms 83.007 ms 7 199.ATM7-0.XR1.PAO1.ALTER.NET (146.188.147.117) 75.24 ms 107.525 ms 86.392 ms 8 189.ATM10-0-0.GW3.PAO1.ALTER.NET (146.188.147.249) 78.844 ms 81.42 ms 85.98 ms 9 208.195.253.18 (208.195.253.18) 94.185 ms 81.601 ms 84.882 ms 10 core-gw1.pa-x.dec.com (204.123.1.1) 81.878 ms 81.915 ms 82.955 ms 11 altavista.com (204.123.2.69) 90.244 ms 86.355 ms 84.804 ms
*** default route stuff ***
traceroute to 192.168.0.1 (192.168.0.1), 7 hops max, 40 byte packets 1 nemo.webkorner.com (207.53.77.1) 2.109 ms 1.987 ms 1.725 ms 2 core-atln1-hssi4/0.9.grid.net (207.205.2.1) 16.876 ms 17.136 ms 16.879 ms 3 core-wash1-atm4/0.4.grid.net (206.80.188.33) 80.955 ms 69.07 ms 84.384 ms 4 206.80.168.178 (206.80.168.178) 108.509 ms 108.005 ms 108.684 ms 5 pb-nap.eni.net (198.32.128.39) 116.592 ms 108.99 ms 108.275 ms 6 pb-nap.eni.net (198.32.128.39) 108.28 ms !H * * 7 * * *
[they blackholed this one, oddly enough]
traceroute to 172.16.0.1 (172.16.0.1), 7 hops max, 40 byte packets 1 nemo.webkorner.com (207.53.77.1) 1.956 ms 1.996 ms 1.695 ms 2 core-atln1-hssi4/0.9.grid.net (207.205.2.1) 16.623 ms 17.005 ms 20.315 ms 3 core-snfc1-hssi11-0.1.grid.net (206.80.180.214) 78.943 ms 76.954 ms 78.453 ms 4 * * * 5 * * * 6 * * * 7 * * *
traceroute to 172.17.0.1 (172.17.0.1), 7 hops max, 40 byte packets 1 nemo.webkorner.com (207.53.77.1) 2.184 ms 2.169 ms 1.69 ms 2 core-atln1-hssi4/0.9.grid.net (207.205.2.1) 17.204 ms 17.49 ms 96.326 ms 3 core-wash1-atm4/0.4.grid.net (206.80.188.33) 73.894 ms 80.646 ms 72.661 ms 4 206.80.168.178 (206.80.168.178) 114.916 ms 111.088 ms 107.569 ms 5 pb-nap.eni.net (198.32.128.39) 110.934 ms 109.213 ms 109.857 ms 6 * * * 7 * * *
traceroute to 172.32.0.1 (172.32.0.1), 7 hops max, 40 byte packets 1 nemo.webkorner.com (207.53.77.1) 2.002 ms 3.351 ms 1.696 ms 2 core-atln1-hssi4/0.9.grid.net (207.205.2.1) 17.148 ms 16.834 ms 18.726 ms 3 core-wash1-atm8-0.3.grid.net (206.80.180.34) 94.444 ms 94.4 ms 99.834 ms 4 206.80.168.178 (206.80.168.178) 140.235 ms 140.247 ms 137.033 ms 5 pb-nap.eni.net (198.32.128.39) 127.318 ms 133.545 ms 124.439 ms 6 * * * 7 * * *
traceroute to 10.0.0.1 (10.0.0.1), 7 hops max, 40 byte packets 1 nemo.webkorner.com (207.53.77.1) 2.998 ms 1.933 ms 2.636 ms 2 core-atln1-hssi4/0.9.grid.net (207.205.2.1) 16.79 ms 17.42 ms 17.519 ms 3 core-wash1-atm4/0.4.grid.net (206.80.188.33) 83.64 ms 80.487 ms 98.335 ms 4 206.80.168.178 (206.80.168.178) 126.873 ms 111.404 ms 110.231 ms 5 pb-nap.eni.net (198.32.128.39) 127.533 ms 109.746 ms 111.201 ms 6 pb-nap.eni.net (198.32.128.39) 114.215 ms !H * * 7 * * *
traceroute to 1.0.0.1 (1.0.0.1), 7 hops max, 40 byte packets 1 nemo.webkorner.com (207.53.77.1) 2.053 ms 2.003 ms 1.685 ms 2 core-atln1-hssi4/0.9.grid.net (207.205.2.1) 30.512 ms 17.147 ms 18.741 ms 3 core-wash1-atm4/0.4.grid.net (206.80.188.33) 76.457 ms 89.586 ms 69.123 ms 4 206.80.168.178 (206.80.168.178) 110.383 ms 117.309 ms 112.722 ms 5 pb-nap.eni.net (198.32.128.39) 112.76 ms 112.733 ms 111.573 ms 6 * * * 7 * * *
traceroute to 0.0.0.1 (0.0.0.1), 7 hops max, 40 byte packets 1 nemo.webkorner.com (207.53.77.1) 2.025 ms 2.825 ms 1.753 ms 2 core-atln1-hssi4/0.9.grid.net (207.205.2.1) 20.68 ms 17.356 ms 20.616 ms 3 core-wash1-atm4/0.4.grid.net (206.80.188.33) 73.731 ms core-wash1-atm8-0.3.grid.net (206.80.180.34) 94.608 ms core-wash1-atm4/0.4.grid.net (206.80.188.33) 68.392 ms 4 206.80.168.178 (206.80.168.178) 121.929 ms 107.539 ms 120.781 ms 5 pb-nap.eni.net (198.32.128.39) 112.065 ms 122.983 ms 109.885 ms 6 * * * 7 pb-nap.eni.net (198.32.128.39) 111.42 ms !H * *
Aleksei Roudnev, Network Operations Center, Relcom, Moscow (+7 095) 194-19-95 (Network Operations Center Hot Line),(+7 095) 230-41-41, N 13729 (pager) (+7 095) 196-72-12 (Support), (+7 095) 194-33-28 (Fax)