I'm OK to that IP as well, but when I query www.google.com, I get multiple IPs, but here are the ones that in in 147: DNS Server IP Route (for me) 205.234.170.217 (tiggee) 74.125.79.147 Amsterdam 208.67.222.222 (opendns) 64.233.183.147 Amsterdam 4.2.2.1 (verizon) 74.125.19.147 San Jose 198.6.1.3 (uu.net/verizon) 74.125.47.147 Washington DC (yay) That's a lot of different answers for the same question! So maybe the problem is that some DNS servers are smarter than others. I am on Verizon FIOS in Northern VA (DC), and use tiggee's free resolving service, http://www.resolvingnameserver.com/freerns.html (they are great for DNS hosting). I used to use 198.6.1.1, but it seems so did everyone else who used to work at UUnet, and they shut it off. Rather than using 198.6.1.3 everywhere (which I'm sure the network folk would appreciate), I found Tiggee's resolving name service, which is nicely anycasted. So what's the deal here? Caching? Beckman On Fri, 6 Feb 2009, Wallace Keith wrote:

Looks ok from Boston-

3 core2.po1-bbnet1.bsn.pnap.net (63.251.128.18) 2.590 ms 3.988 ms 3.181 ms 4 207.88.182.33.ptr.us.xo.net (207.88.182.33) 26.636 ms 7.651 ms 11.977 ms 5 207.88.182.18.ptr.us.xo.net (207.88.182.18) 7.603 ms 8.174 ms 7.405 ms 6 216.239.49.217 (216.239.49.217) 8.219 ms 8.388 ms 7.510 ms 7 72.14.236.213 (72.14.236.213) 8.468 ms 66.249.94.235 (66.249.94.235) 16.799 ms 72.14.236.213 (72.14.236.213) 9.196 ms 8 72.14.238.138 (72.14.238.138) 27.405 ms 209.85.248.216 (209.85.248.216) 16.352 ms 15.785 ms 9 72.14.238.235 (72.14.238.235) 15.691 ms 14.641 ms 15.628 ms 10 209.85.255.194 (209.85.255.194) 39.077 ms 72.14.239.136 (72.14.239.136) 31.206 ms 64.233.174.46 (64.233.174.46) 32.528 ms 11 209.85.254.247 (209.85.254.247) 28.897 ms 209.85.254.249 (209.85.254.249) 29.192 ms 29.649 ms 12 209.85.255.194 (209.85.255.194) 29.326 ms gw-in-f100.google.com (74.125.67.100) 28.393 ms 209.85.255.194 (209.85.255.194) 35.464 ms

-----Original Message----- From: Peter Beckman [mailto:beckman@angryox.com] Sent: Friday, February 06, 2009 11:01 AM To: nanog@nanog.org Subject: L3: Google from DC via the Netherlands?

Seems strange. Had a partial outage on Verizon network this morning around 9:50am EST, then when it came back around 10:05am, google routed via the Netherlands. My guess is that there's some sort of routing problem making my fastest or least cost route go to the Netherlands, but I wanted to find out if there was an ongoing issue, or if the Netherlands simply now serves the US East Coast for google.

Isn't there anything closer? Or is this just indicative of an ongoing L3 outage? I'm guessing it's cheaper/faster to get to Google somewhere in the US, but maybe someone on NANOG knows better than I do.

HOST: octal.angryox.com Loss% Snt Last Avg Best Wrst StDev 1. tomato.angryox.com 0.0% 20 0.6 0.6 0.5 0.7 0.0 2. 10.1.41.15 0.0% 20 2.1 4.2 1.9 42.8 9.1 3. p4-2.lcr-02.washdc.verizon-g 0.0% 20 1.5 1.5 1.3 2.3 0.2 4. 130.81.29.218 0.0% 20 2.0 7.3 2.0 45.0 11.9 5. 0.so-1-2-0.xl4.iad8.alter.ne 0.0% 20 2.4 2.4 2.3 3.3 0.2 6. 0.ge-3-3-0.br2.iad8.alter.ne 0.0% 20 2.4 2.4 2.3 2.5 0.1 7. te-11-3-0.edge1.washington4. 0.0% 20 2.4 12.2 2.3 151.2 33.6 8. vlan99.csw4.washington1.leve 0.0% 20 3.1 7.3 3.0 15.0 4.3 9. ae-92-92.ebr2.washington1.le 0.0% 20 3.2 3.3 3.1 3.6 0.1 10. ae-42-42.ebr2.frankfurt1.lev 0.0% 20 94.1 94.4 93.7 104.0 2.3 11. ae-2-2.ebr1.dusseldorf1.leve 0.0% 20 106.1 100.6 95.8 107.0 4.1 12. ae-46-106.ebr2.dusseldorf1.l 0.0% 20 100.4 103.1 98.0 111.0 4.3 13. ae-41.ebr1.amsterdam1.level3 0.0% 20 107.2 99.9 94.4 107.7 5.7 14. ae-11-53.car1.amsterdam1.lev 0.0% 20 94.5 95.0 94.1 102.6 2.0 15. 212.72.42.14 0.0% 20 94.9 98.8 94.5 133.9 12.0 16. 209.85.254.250 0.0% 20 95.4 95.4 94.8 97.0 0.5 17. 72.14.233.114 0.0% 20 100.9 107.2 100.7 140.6 9.7 18. 209.85.255.166 0.0% 20 108.9 109.1 107.5 117.3 2.1 19. 209.85.255.110 5.0% 20 109.5 112.4 106.3 119.9 4.7 20. ew-in-f147.google.com 5.0% 20 107.9 108.0 107.5 108.9 0.3

Fri Feb 6 10:56:09 2009 Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. tomato.angryox.com 0.0% 8 0.6 0.6 0.5 0.6 0.0 2. 10.1.41.15 0.0% 8 2.1 2.4 2.1 3.3 0.4 3. P4-2.LCR-02.WASHDC.verizon-gni.net 0.0% 8 1.4 1.4 1.3 1.5 0.1 4. 130.81.29.218 0.0% 8 2.1 9.1 1.9 59.0 20.2 5. 0.so-1-2-0.XL4.IAD8.ALTER.NET 0.0% 8 2.4 2.4 2.3 2.6 0.1 0.so-6-1-0.XL4.IAD8.ALTER.NET 6. 0.ge-7-1-0.BR2.IAD8.ALTER.NET 0.0% 8 2.5 2.5 2.4 2.6 0.1 0.ge-3-3-0.BR2.IAD8.ALTER.NET 0.ge-5-1-0.BR2.IAD8.ALTER.NET 0.ge-7-0-0.BR2.IAD8.ALTER.NET 7. te-11-3-0.edge1.Washington4.level3.net 0.0% 7 2.4 6.0 2.3 27.2 9.4 8. vlan79.csw2.Washington1.Level3.net 0.0% 7 3.2 8.3 3.2 13.5 4.2 9. ae-72-72.ebr2.Washington1.Level3.net 0.0% 7 3.7 3.5 3.1 4.0 0.3 10. ae-44-44.ebr2.Frankfurt1.Level3.net 0.0% 7 94.5 94.9 94.4 97.2 1.0 11. ae-2-2.ebr1.Dusseldorf1.Level3.net 0.0% 7 106.2 100.7 96.4 106.7 4.6 12. ae-46-106.ebr2.Dusseldorf1.Level3.net 0.0% 7 107.6 106.1 100.1 108.9 3.0 13. ae-41.ebr1.Amsterdam1.Level3.net 0.0% 7 95.5 100.6 95.4 108.3 5.9 14. ae-11-55.car1.Amsterdam1.Level3.net 0.0% 7 95.3 95.8 94.4 98.7 1.5 15. GOOGLE-INC.car1.Amsterdam1.Level3.net 0.0% 7 97.0 96.7 96.5 97.0 0.2 16. 209.85.254.250 0.0% 7 102.0 97.0 95.7 102.0 2.2 17. 72.14.233.114 0.0% 7 107.5 104.8 101.5 107.5 2.7 209.85.248.79 18. 209.85.255.70 0.0% 7 109.8 109.1 107.9 110.3 0.8 72.14.239.123 209.85.255.166 209.85.255.20 19. 209.85.255.98 0.0% 7 118.3 114.9 109.7 120.2 4.6 209.85.255.102 209.85.255.110 20. ew-in-f103.google.com 0.0% 7 110.0 109.4 108.0 110.6 1.1

------------------------------------------------------------------------ --- Peter Beckman Internet Guy beckman@angryox.com http://www.angryox.com/ ------------------------------------------------------------------------ ---

--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman@angryox.com http://www.angryox.com/ ---------------------------------------------------------------------------

Peter Beckman wrote:

SO. Who's problem is this to fix? Is it:

1. Me? Am I a dope for using a very reliable but anycasted resolving name service? Clearly, I could just use the handy dandy easy to remember because I worked there 198.6.1.x, or is that an Internet faux pas because technically I wasn't given permission to use it?

We are network operators. We all use robust distributed name servers, along with our own (usually hidden) primary. But as a network operator, why aren't you running your own caching resolver? Not since the '80s have I ever needed to point at somebody else's resolver, the DNS is much better distributed now.

IMHO, off the top of my head, on a weekend where I haven't had enough coffee yet: 3. Anycasted DNS Providers? Not sure how they could fix it, other than flag certain domains as special, and do something special for them, but man that smells like a hack. Anycast is a good thing, but when geo-location style concerns are factored in maybe they should have region-based anycast addresses. Interestingly, with Google there could be another similar concern WRT the IPv6 "trusted tester program" (or whatever the correct name of that is) where the DNS resolver / organization could have sufficient IPv6 connectivity to qualify, but that capability might not expand to the clients of / hosts within the service. /TJ

-----Original Message----- From: Peter Beckman [mailto:beckman@angryox.com] Sent: Friday, February 06, 2009 2:51 PM To: nanog@nanog.org Subject: RE: L3: Google from DC via the Netherlands?

On Fri, 6 Feb 2009, Peter Beckman wrote:

...

I'm OK to that IP as well, but when I query www.google.com, I get multiple IPs, but here are the ones that in in 147:

DNS Server IP Route (for me) 205.234.170.217 (tiggee) 74.125.79.147 Amsterdam 208.67.222.222 (opendns) 64.233.183.147 Amsterdam 4.2.2.1 (verizon) 74.125.19.147 San Jose 198.6.1.3 (uu.net/verizon) 74.125.47.147 Washington DC (yay)

So someone from Google has been helpful in pointing out that the resolver IP, not YOUR IP, is the one that determines where you get routed to when you make a request for www.google.com. Which is simply due to the way things are implemented, which makes sense.

The problem is, here I am, just some guy, and 99%* of the Internet resolves to the same IP(s) regardless of who I ask. But then the other 1%*, and this would likely be larger players who are diversified and have systems in multiple locations and networks, do something funky and give a different address depending on where your DNS server is in the network.

Then throw in the possibility that YOUR DNS servers are anycasted for great justice, or at least for good reliability. Now when you base YOUR answer on the caching server's IP address, well, it may not make sense. It seems that Tiggee and OpenDNS are anycasted, as is DNS Advantage, as well as some root nameservers.

Thus my problem -- because I ask two free resolving name services, which I believe to be anycasted, where to go, I get routed to Amsterdam instead of a few miles down the road in Ashburn, VA, and spend 100ms instead of 10ms travelling the globe, costing someone more money for Atlantic Ocean transit when it was unnecessary.

SO. Who's problem is this to fix? Is it:

1. Me? Am I a dope for using a very reliable but anycasted resolving name service? Clearly, I could just use the handy dandy easy to remember because I worked there 198.6.1.x, or is that an Internet faux pas because technically I wasn't given permission to use it?

2. Google? They probably have an interest in making sure my experience to their services are fast and as close to me as possible, but I'm probably a minority and not worth the effort of refactoring a giant DNS implementation just to fix my one problem, nay, inconvenience.

3. Anycasted DNS Providers? Not sure how they could fix it, other than flag certain domains as special, and do something special for them, but man that smells like a hack.

4. My ISP? Does the ISP have to gripe at Google for providing bad results that causes traffic to go over expensive lines when it could have easily gone locally and much more cheaply? I'm assuming that sending my traffic over the Atlantic to the Netherlands costs SOMEONE more money than if I had gone to a datacenter nearby, both physically and network-wise.

5. Nobody? Is it just the price the customer (me, who helps generate income for Google by using Google and clicking AdWords ads all day) pays for the reliability, redundancy and fault tolerance that Google has implemented?

I think things are working as implemented -- it's not "broken," but it seems it could be better. Then again, sometimes better is more expensive than the status quo, either in time or money or both.

NOTE: I do not admit to knowing that 100% of what I've written is fact, and if you know better than I, please correct me and show me the light.

* Numbers have no basis, just a guess.

Beckman --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman@angryox.com http://www.angryox.com/ ---------------------------------------------------------------------------

In message <alpine.BSF.2.00.0902081439461.72677@nog.angryox.com>, Peter Beckman writes:

After a few emails traded with David Ulevitch from OpenDNS, it is clear to me that they do NOT suffer from this issue, and have a work-around. My apologies to David and to OpenDNS for lumping them in and not doing better due dilligence when researching this issue.

On Sat, 7 Feb 2009, TJ wrote:

...

IMHO, off the top of my head, on a weekend where I haven't had enough coffe e yet:

3. Anycasted DNS Providers? Not sure how they could fix it, other than flag certain domains as special, and do something special for them, but man that smells like a hack.

Anycast is a good thing, but when geo-location style concerns are factored in maybe they should have region-based anycast addresses.

Anycast is extremely useful for fault tolerance, agreed. But what I personally didn't consider, and I don't think other people consider, when they chose to use an alternative DNS caching resolution providers is what might break or not operate as expected.

Having traded a few private emails from people smarter than I at Google and OpenDNS, I understand the issue much better than when I first posted. Thank you to you both.

Here's a theoretical solution to this problem that I'd like to open for discussion.

In each location where a provider hosts their anycasted service, there is likely a local, non-anycasted IP address for each server. When receiving a DNS request that is not in the local cache, or has expired, make the new request on that local IP address interface, rather than on the anycasted IP address interface. In those cases, GSLB records would likely return a more accurate set of results for clients making DNS requests of it, and when those records were requested from the anycasted DNS resolving service, the cached records would more likely be closer from a network standpoint to the actual service.

Obviously there are some issues: * need to patch BIND or PowerDNS to use a different interface for making new requests

query-source ....;

* possibility of the responding anycasted DNS server being close to server farm A, while being far away from DNS record requestor B

I'm curious to find out if others on the list know what other companies are using GSLB, and what the actual impact of anycasted DNS caching nameservers has on GSLB records. If enough people are using anycasted DNS resolution services, implementing a fix like this would reduce network traffic. By how much, I don't know.

Beckman --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman@angryox.com http://www.angryox.com/ ---------------------------------------------------------------------------

-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: Mark_Andrews@isc.org

On Fri, Feb 06, 2009 at 12:05:41PM -0500, Peter Beckman wrote:

I'm OK to that IP as well, but when I query www.google.com, I get multiple IPs, but here are the ones that in in 147:

DNS Server IP Route (for me) 205.234.170.217 (tiggee) 74.125.79.147 Amsterdam 208.67.222.222 (opendns) 64.233.183.147 Amsterdam 4.2.2.1 (verizon) 74.125.19.147 San Jose 198.6.1.3 (uu.net/verizon) 74.125.47.147 Washington DC (yay)

That's a lot of different answers for the same question!

And you can add this one to your list: DNS Server IP 212.27.40.240 (free/proxad) 2001:4860:A003:0:0:0:0:68 $ host -t AAAA www.google.com 212.27.40.240 www.google.com CNAME www.l.google.com www.l.google.com AAAA 2001:4860:A003:0:0:0:0:68 As you can see, no need for http://ipv6.google.com/ to reach Google over IPv6 using default DNS resolver provided by some French ISPs. www.google.com starts to be resolved as an IPv4/IPv6 website. -- Nicolas Hyvernat