www.google.com latency/packet loss/very slow thru savvis
hey is any one seeing a slow google to day... packet loss 22 hops... visual traceroute show that it goes from chicago to los angeles to singapore to italy to amsterdam and then finally to google in sunnyvale ca. router@m40> ping google.com PING google.com (72.14.207.99): 56 data bytes 64 bytes from 72.14.207.99: icmp_seq=1 ttl=243 time=447.727 ms 64 bytes from 72.14.207.99: icmp_seq=2 ttl=244 time=445.543 ms 64 bytes from 72.14.207.99: icmp_seq=3 ttl=243 time=442.937 ms 64 bytes from 72.14.207.99: icmp_seq=4 ttl=244 time=443.776 ms 64 bytes from 72.14.207.99: icmp_seq=5 ttl=243 time=446.607 ms 64 bytes from 72.14.207.99: icmp_seq=6 ttl=243 time=451.655 ms 64 bytes from 72.14.207.99: icmp_seq=7 ttl=243 time=451.842 ms 64 bytes from 72.14.207.99: icmp_seq=8 ttl=244 time=447.052 ms 64 bytes from 72.14.207.99: icmp_seq=9 ttl=243 time=448.432 ms 64 bytes from 72.14.207.99: icmp_seq=10 ttl=244 time=433.543 ms 64 bytes from 72.14.207.99: icmp_seq=12 ttl=244 time=455.547 ms 64 bytes from 72.14.207.99: icmp_seq=13 ttl=243 time=451.316 ms 64 bytes from 72.14.207.99: icmp_seq=14 ttl=243 time=444.915 ms 64 bytes from 72.14.207.99: icmp_seq=16 ttl=243 time=444.318 ms ^C --- google.com ping statistics --- 17 packets transmitted, 14 packets received, 17% packet loss round-trip min/avg/max/stddev = 433.543/446.801/455.547/5.075 ms router@m40> traceroute google.com traceroute to google.com (72.14.207.99), 30 hops max, 40 byte packets 1 s216240-1.rtr1s.216240.uschcg2-j20c.savvis.net (64.240.49.173) 5.275 ms 4.889 ms 4.378 ms 2 dcr2-so-2-2-0.Chicago.savvis.net (208.172.3.81) 5.191 ms 60.075 ms 4.844 ms 3 bcs2-so-0-0-0.Dallas.savvis.net (204.70.192.98) 30.408 ms 30.099 ms 30.683 ms MPLS Label=595600 CoS=0 TTL=1 S=1 4 bcs1-so-6-0-0.Dallas.savvis.net (204.70.192.49) 30.093 ms 30.035 ms 30.314 ms MPLS Label=294001 CoS=0 TTL=1 S=1 5 dcr2-so-2-0-0.LosAngeles.savvis.net (204.70.192.86) 62.299 ms 62.363 ms 62.326 ms MPLS Label=442832 CoS=0 TTL=1 S=1 6 dcr1-as0-0.LosAngeles.savvis.net (204.70.192.117) 158.574 ms 62.171 ms 62.628 ms 7 bpr3-so-0-0-0.LosAngelesEquinix.savvis.net (208.172.44.106) 62.246 ms 62.182 ms 64.274 ms 8 singapore-telecommunications-ltd.LosAngelesEquinix.savvis.net (208.174.196.6) 64.209 ms 62.281 ms 62.374 ms 9 laxeq-cr1.ge-3-0-0.ix.singtel.com (203.208.182.45) 62.767 ms ge-0-3-0.laxeq-cr1.ix.singtel.com (203.208.168.221) 62.958 ms 62.604 ms MPLS Label=126512 CoS=0 TTL=1 S=1 10 so-3-3-2.sngc3-cr1.ix.singtel.com (203.208.154.25) 227.841 ms 225.874 ms 226.254 ms MPLS Label=100288 CoS=0 TTL=1 S=1 11 ge-1-0-0.sngc3-ar2.ix.singtel.com (203.208.172.158) 232.373 ms ge-0-0-0.sngc3-ar2.ix.singtel.com (203.208.172.166) 232.503 ms 239.240 ms 12 203.208.147.238 (203.208.147.238) 424.056 ms * 429.784 ms 13 pal6-pakistan-telecom-1-pk.pal.seabone.net (195.22.197.193) 477.452 ms 478.886 ms * 14 amsix-ams2-racc1.ams.seabone.net (195.22.213.221) 483.282 ms 491.266 ms 480.559 ms 15 core1.ams.net.google.com (195.69.144.247) 448.393 ms 442.054 ms 435.464 ms 16 216.239.43.90 (216.239.43.90) 452.569 ms 460.223 ms 451.533 ms MPLS Label=588419 CoS=0 TTL=1 S=1 17 72.14.236.216 (72.14.236.216) 429.854 ms 433.688 ms 448.938 ms MPLS Label=775643 CoS=0 TTL=1 S=1 18 66.249.94.98 (66.249.94.98) 448.831 ms 439.760 ms 453.181 ms 19 66.249.94.69 (66.249.94.69) 444.019 ms * 439.384 ms 20 72.14.236.183 (72.14.236.183) 446.575 ms 438.125 ms * 21 * 72.14.236.134 (72.14.236.134) 451.592 ms 457.254 ms 22 72.14.207.99 (72.14.207.99) 452.669 ms * 441.830 ms Thanks Erik
On Tuesday 13 December 2005 02:46 pm, you wrote:
router@m40> ping google.com PING google.com (72.14.207.99): 56 data bytes 64 bytes from 72.14.207.99: icmp_seq=1 ttl=243 time=447.727 ms 64 bytes from 72.14.207.99: icmp_seq=2 ttl=244 time=445.543 ms 64 bytes from 72.14.207.99: icmp_seq=3 ttl=243 time=442.937 ms 64 bytes from 72.14.207.99: icmp_seq=4 ttl=244 time=443.776 ms
Looks ok here from Denver, CO: PING google.com (64.233.187.99) 56(84) bytes of data. 64 bytes from 64.233.187.99: icmp_seq=1 ttl=235 time=56.4 ms 64 bytes from 64.233.187.99: icmp_seq=2 ttl=236 time=56.2 ms 64 bytes from 64.233.187.99: icmp_seq=3 ttl=235 time=55.9 ms 64 bytes from 64.233.187.99: icmp_seq=4 ttl=236 time=56.3 ms 64 bytes from 64.233.187.99: icmp_seq=5 ttl=235 time=56.2 ms 64 bytes from 64.233.187.99: icmp_seq=6 ttl=236 time=56.4 ms 64 bytes from 64.233.187.99: icmp_seq=7 ttl=235 time=56.0 ms 64 bytes from 64.233.187.99: icmp_seq=8 ttl=236 time=56.4 ms 64 bytes from 64.233.187.99: icmp_seq=9 ttl=235 time=56.1 ms 64 bytes from 64.233.187.99: icmp_seq=10 ttl=236 time=56.3 ms 64 bytes from 64.233.187.99: icmp_seq=11 ttl=235 time=56.0 ms 64 bytes from 64.233.187.99: icmp_seq=12 ttl=236 time=56.3 ms traceroute to google.com (72.14.207.99), 30 hops max, 38 byte packets 1 10.10.1.254 (10.10.1.254) 1.008 ms 0.346 ms 0.314 ms 2 ai-core.axint.net (38.116.133.1) 0.771 ms 0.597 ms 0.542 ms 3 f0-2.na01.b009854-0.den01.atlas.cogentco.com (38.112.6.145) 1.059 ms 8.405 ms 0.854 ms 4 g0-3.core01.den01.atlas.cogentco.com (66.250.10.217) 4.955 ms 2.292 ms 1.090 ms 5 p5-0.core01.mci01.atlas.cogentco.com (66.28.4.30) 14.825 ms 12.485 ms 11.917 ms 6 p5-0.core02.ord01.atlas.cogentco.com (66.28.4.34) 23.278 ms 23.379 ms 23.330 ms 7 p15-0.core01.ord03.atlas.cogentco.com (154.54.2.242) 105.414 ms 198.623 ms 224.269 ms 8 google.ord03.atlas.cogentco.com (154.54.11.186) 25.238 ms 23.616 ms 23.705 ms 9 66.249.95.253 (66.249.95.253) 23.871 ms 23.776 ms 23.610 ms 10 66.249.95.247 (66.249.95.247) 25.818 ms 23.816 ms 66.249.95.120 (66.249.95.120) 23.601 ms 11 72.14.236.20 (72.14.236.20) 37.726 ms 38.636 ms 37.839 ms 12 66.249.94.68 (66.249.94.68) 37.847 ms 37.883 ms 37.842 ms 13 72.14.236.181 (72.14.236.181) 40.689 ms 38.346 ms 47.902 ms 14 66.249.94.73 (66.249.94.73) 38.500 ms 38.306 ms 72.14.236.130 (72.14.236.130) 43.042 ms 15 72.14.207.99 (72.14.207.99) 38.486 ms 38.637 ms 38.375 ms
On Tue, 13 Dec 2005, Chris Stone wrote:
On Tuesday 13 December 2005 02:46 pm, you wrote:
router@m40> ping google.com PING google.com (72.14.207.99): 56 data bytes 64 bytes from 72.14.207.99: icmp_seq=1 ttl=243 time=447.727 ms 64 bytes from 72.14.207.99: icmp_seq=2 ttl=244 time=445.543 ms 64 bytes from 72.14.207.99: icmp_seq=3 ttl=243 time=442.937 ms 64 bytes from 72.14.207.99: icmp_seq=4 ttl=244 time=443.776 ms
Looks ok here from Denver, CO:
PING google.com (64.233.187.99) 56(84) bytes of data. 64 bytes from 64.233.187.99: icmp_seq=1 ttl=235 time=56.4 ms 64 bytes from 64.233.187.99: icmp_seq=2 ttl=236 time=56.2 ms 64 bytes from 64.233.187.99: icmp_seq=3 ttl=235 time=55.9 ms
You do realise you're talking to a 'different' google.com, right? That said, I tested both of the above IP's without any problems, via reach.com, if it helps. Mark.
Erik Sundberg wrote:
hey is any one seeing a slow google to day... packet loss 22 hops...
visual traceroute show that it goes from chicago to los angeles to singapore to italy to amsterdam and then finally to google in sunnyvale ca.
router@m40> ping google.com PING google.com (72.14.207.99): 56 data bytes 64 bytes from 72.14.207.99: icmp_seq=1 ttl=243 time=447.727 ms 64 bytes from 72.14.207.99: icmp_seq=2 ttl=244 time=445.543 ms 64 bytes from 72.14.207.99: icmp_seq=3 ttl=243 time=442.937 ms 64 bytes from 72.14.207.99: icmp_seq=4 ttl=244 time=443.776 ms 64 bytes from 72.14.207.99: icmp_seq=5 ttl=243 time=446.607 ms 64 bytes from 72.14.207.99: icmp_seq=6 ttl=243 time=451.655 ms 64 bytes from 72.14.207.99: icmp_seq=7 ttl=243 time=451.842 ms 64 bytes from 72.14.207.99: icmp_seq=8 ttl=244 time=447.052 ms 64 bytes from 72.14.207.99: icmp_seq=9 ttl=243 time=448.432 ms 64 bytes from 72.14.207.99: icmp_seq=10 ttl=244 time=433.543 ms 64 bytes from 72.14.207.99: icmp_seq=12 ttl=244 time=455.547 ms 64 bytes from 72.14.207.99: icmp_seq=13 ttl=243 time=451.316 ms 64 bytes from 72.14.207.99: icmp_seq=14 ttl=243 time=444.915 ms 64 bytes from 72.14.207.99: icmp_seq=16 ttl=243 time=444.318 ms ^C --- google.com ping statistics --- 17 packets transmitted, 14 packets received, 17% packet loss round-trip min/avg/max/stddev = 433.543/446.801/455.547/5.075 ms
router@m40> traceroute google.com traceroute to google.com (72.14.207.99), 30 hops max, 40 byte packets 1 s216240-1.rtr1s.216240.uschcg2-j20c.savvis.net (64.240.49.173) 5.275 ms 4.889 ms 4.378 ms 2 dcr2-so-2-2-0.Chicago.savvis.net (208.172.3.81) 5.191 ms 60.075 ms 4.844 ms 3 bcs2-so-0-0-0.Dallas.savvis.net (204.70.192.98) 30.408 ms 30.099 ms 30.683 ms MPLS Label=595600 CoS=0 TTL=1 S=1 4 bcs1-so-6-0-0.Dallas.savvis.net (204.70.192.49) 30.093 ms 30.035 ms 30.314 ms MPLS Label=294001 CoS=0 TTL=1 S=1 5 dcr2-so-2-0-0.LosAngeles.savvis.net (204.70.192.86) 62.299 ms 62.363 ms 62.326 ms MPLS Label=442832 CoS=0 TTL=1 S=1 6 dcr1-as0-0.LosAngeles.savvis.net (204.70.192.117) 158.574 ms 62.171 ms 62.628 ms 7 bpr3-so-0-0-0.LosAngelesEquinix.savvis.net (208.172.44.106) 62.246 ms 62.182 ms 64.274 ms 8 singapore-telecommunications-ltd.LosAngelesEquinix.savvis.net (208.174.196.6) 64.209 ms 62.281 ms 62.374 ms 9 laxeq-cr1.ge-3-0-0.ix.singtel.com (203.208.182.45) 62.767 ms ge-0-3-0.laxeq-cr1.ix.singtel.com (203.208.168.221) 62.958 ms 62.604 ms MPLS Label=126512 CoS=0 TTL=1 S=1 10 so-3-3-2.sngc3-cr1.ix.singtel.com (203.208.154.25) 227.841 ms 225.874 ms 226.254 ms MPLS Label=100288 CoS=0 TTL=1 S=1 11 ge-1-0-0.sngc3-ar2.ix.singtel.com (203.208.172.158) 232.373 ms ge-0-0-0.sngc3-ar2.ix.singtel.com (203.208.172.166) 232.503 ms 239.240 ms 12 203.208.147.238 (203.208.147.238) 424.056 ms * 429.784 ms 13 pal6-pakistan-telecom-1-pk.pal.seabone.net (195.22.197.193) 477.452 ms 478.886 ms * 14 amsix-ams2-racc1.ams.seabone.net (195.22.213.221) 483.282 ms 491.266 ms 480.559 ms 15 core1.ams.net.google.com (195.69.144.247) 448.393 ms 442.054 ms 435.464 ms 16 216.239.43.90 (216.239.43.90) 452.569 ms 460.223 ms 451.533 ms MPLS Label=588419 CoS=0 TTL=1 S=1 17 72.14.236.216 (72.14.236.216) 429.854 ms 433.688 ms 448.938 ms MPLS Label=775643 CoS=0 TTL=1 S=1 18 66.249.94.98 (66.249.94.98) 448.831 ms 439.760 ms 453.181 ms 19 66.249.94.69 (66.249.94.69) 444.019 ms * 439.384 ms 20 72.14.236.183 (72.14.236.183) 446.575 ms 438.125 ms * 21 * 72.14.236.134 (72.14.236.134) 451.592 ms 457.254 ms 22 72.14.207.99 (72.14.207.99) 452.669 ms * 441.830 ms
Thanks
Erik
From seattle.. it looks fine.. ritalin[~]$ traceroute 72.14.207.99 traceroute to 72.14.207.99 (72.14.207.99), 30 hops max, 38 byte packets 1 vl101-d1.acc.sea2.hopone.net (209.160.32.2) 1.221 ms 1.307 ms 0.638 ms 2 ge6-0.core1.sea2.hopone.net (209.160.60.242) 1.462 ms 0.269 ms 0.152 ms 3 ge6-0.core2.sea2.hopone.net (209.160.60.243) 0.222 ms 0.301 ms 0.285 ms 4 ge-3-0-0--0.tkwl.eli.net (66.36.224.193) 0.729 ms 1.000 ms 1.104 ms 5 ge-4-1-0--0.cr02.tkwl.eli.net (207.173.115.209) 1.775 ms 0.849 ms 1.167 ms 6 so-0-0-0--0.cr01.mpls.eli.net (207.173.115.166) 37.471 ms 37.276 ms 37.863 ms 7 so-3-0-0--0.cr02.mpls.eli.net (207.173.114.186) 46.812 ms 37.432 ms 37.278 ms 8 so-1-0-0--0.cr01.chcg.eli.net (207.173.115.233) 47.082 ms 47.461 ms 46.652 ms 9 ge-1-0-0--0.er01.chcg.eli.net (207.173.115.138) 46.698 ms 46.376 ms 46.406 ms 10 core1-2-2-0.ord.net.google.com (206.223.119.21) 98.684 ms 47.532 ms 46.726 ms 11 216.239.46.1 (216.239.46.1) 46.967 ms 47.061 ms 46.759 ms 12 72.14.236.20 (72.14.236.20) 61.060 ms 61.015 ms 60.896 ms 13 72.14.236.183 (72.14.236.183) 61.848 ms 61.650 ms 61.801 ms 14 72.14.236.134 (72.14.236.134) 70.552 ms 64.497 ms 66.249.94.72 (66.249.94.72) 61.790 ms 15 72.14.236.130 (72.14.236.130) 65.446 ms 61.785 ms 70.031 ms 16 72.14.236.134 (72.14.236.134) 65.919 ms 64.584 ms 72.297 ms 17 72.14.207.99 (72.14.207.99) 62.743 ms 62.596 ms 62.203 ms and from Tysons Corner, Mclean VA $ traceroute 72.14.207.99 traceroute to 72.14.207.99 (72.14.207.99), 64 hops max, 40 byte packets 1 c-vl101-d2.acc.dca2.hopone.net (66.36.230.3) 0.386 ms 0.392 ms 0.507 ms 2 ge4-0.core2.dca2.hopone.net (66.36.224.243) 0.493 ms 0.257 ms 0.460 ms 3 ge2-0-241.core1.iad1.hopone.net (66.36.224.34) 1.221 ms 1.418 ms 1.483 ms 4 core1-2-2-0.ord.net.google.com (206.223.119.21) 28.714 ms 28.847 ms 29.247 ms 5 216.239.46.1 (216.239.46.1) 29.560 ms 66.249.95.249 (66.249.95.249) 29.081 ms 28.739 ms 6 66.249.95.245 (66.249.95.245) 28.622 ms 28.971 ms 28.824 ms 7 72.14.236.20 (72.14.236.20) 43.094 ms 42.856 ms 42.717 ms 8 72.14.236.183 (72.14.236.183) 43.652 ms 43.427 ms 43.673 ms 9 72.14.236.134 (72.14.236.134) 44.317 ms 72.14.236.181 (72.14.236.181) 43.343 ms 43.510 ms 10 72.14.236.130 (72.14.236.130) 44.430 ms 43.862 ms 72.14.207.99 (72.14.207.99) 43.532 ms -- Bill Sehmel - bsehmel@HopOne.net -- 1-206-242-2743 Systems Administrator, HopOne Internet Corp. SEA2 NOC Bandwidth & full range of carrier/web host colo + networking services: http://www.hopone.net#
On Tue, Dec 13, 2005 at 03:46:43PM -0600, Erik Sundberg wrote:
hey is any one seeing a slow google to day... packet loss 22 hops...
visual traceroute show that it goes from chicago to los angeles to singapore to italy to amsterdam and then finally to google in sunnyvale ca.
...yadayadayada...
8 singapore-telecommunications-ltd.LosAngelesEquinix.savvis.net (208.174.196.6) 64.209 ms 62.281 ms 62.374 ms 9 laxeq-cr1.ge-3-0-0.ix.singtel.com (203.208.182.45) 62.767 ms ge-0-3-0.laxeq-cr1.ix.singtel.com (203.208.168.221) 62.958 ms 62.604 ms MPLS Label=126512 CoS=0 TTL=1 S=1 10 so-3-3-2.sngc3-cr1.ix.singtel.com (203.208.154.25) 227.841 ms 225.874 ms 226.254 ms MPLS Label=100288 CoS=0 TTL=1 S=1 11 ge-1-0-0.sngc3-ar2.ix.singtel.com (203.208.172.158) 232.373 ms ge-0-0-0.sngc3-ar2.ix.singtel.com (203.208.172.166) 232.503 ms 239.240 ms 12 203.208.147.238 (203.208.147.238) 424.056 ms * 429.784 ms 13 pal6-pakistan-telecom-1-pk.pal.seabone.net (195.22.197.193) 477.452 ms 478.886 ms * 14 amsix-ams2-racc1.ams.seabone.net (195.22.213.221) 483.282 ms 491.266 ms 480.559 ms 15 core1.ams.net.google.com (195.69.144.247) 448.393 ms 442.054 ms 435.464 ms
You'd be better off with an AS-PATH, but just follow the chain... Savvis -> Singtel -> Pakistan Telecom -> Seabone -> Google. Where I'm from, we call that a routing leak. Doesn't take too much work to guess where either. :) -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
On Tue, Dec 13, 2005 at 05:02:42PM -0500, Richard A Steenbergen wrote:
You'd be better off with an AS-PATH, but just follow the chain... Savvis -> Singtel -> Pakistan Telecom -> Seabone -> Google. Where I'm from, we call that a routing leak. Doesn't take too much work to guess where either. :)
Oh and FYI it is still going on, though the route just changed 4 mins ago: [BGP/170] 00:04:21, localpref 200 AS path: 7473 17557 17557 17557 17557 5400 15169 I Singtel -> Pakistan Telecom -> British Telecom -> Google. AS17557 is leaking its BGP table, and AS7473 is not filtering its customer. Bad network, bad. No cookie. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
On Tue, 13 Dec 2005, Richard A Steenbergen wrote:
Oh and FYI it is still going on, though the route just changed 4 mins ago:
[BGP/170] 00:04:21, localpref 200 AS path: 7473 17557 17557 17557 17557 5400 15169 I
Singtel -> Pakistan Telecom -> British Telecom -> Google. AS17557 is leaking its BGP table, and AS7473 is not filtering its customer. Bad network, bad. No cookie.
and... sbgp would have fixed this for us! :) Seriously though, in the last 12 months there have been atleast 4 major route leaks like this, eh? Is the lesson not being learned to filter custoemrs?
On Tue, 13 Dec 2005, Christopher L. Morrow wrote:
[BGP/170] 00:04:21, localpref 200 AS path: 7473 17557 17557 17557 17557 5400 15169 I
Singtel -> Pakistan Telecom -> British Telecom -> Google. AS17557 is leaking its BGP table, and AS7473 is not filtering its customer. Bad network, bad. No cookie.
and... sbgp would have fixed this for us! :)
SBGP would have. But SIDR produced security extension might not (at least not initially) because cross-peer authentication is not part of the initial work requirement at wg formation. -- William Leibzon Elan Networks william@elan.net
participants (7)
-
Bill Sehmel -
Chris Stone -
Christopher L. Morrow -
Erik Sundberg -
Mark Foster -
Richard A Steenbergen -
william(at)elan.net