(Caution: Chris is a chemical engineer, not an anycast engineer) On Wed, 1 Sep 2004, Steve Francis wrote:

Bill Woodcock wrote:

...

...

...how good/bad using DNS anycast is as a kludgey traffic optimiser?

I'd hardly call it a kludge. It's been standard best-practice for over a decade.

If I read your original request correctly you were planning on: 1) having presence in multiple datacenters (assume multiple providers as well) 2) having a 'authoritative' DNS server in each facility (or 2/3/4 whatever per center) 3) return datacenter-1-host-1 from datacenter-1-authserver-1, datacenter-2-host-2 from datacenter-2-authserver-1, and so forth. This isn't really 'anycast' so much as 'different A records depending on server which was asked' So, you'd be dependent on: 1) order of DNS requests made to AUTH NS servers for your domain/host 2) speed of network(s) between requestor and responder 3) effects of using caching DNS servers along the route You are not, now, making your decision on 'network closeness' so much as 'application swiftness'. I suspect you'd really also introduce some major troubleshooting headaches with this setup, not just for you, but for your users as well. I think in the end you probably want to obtain PI space from ARIN and use that as the 'home' for your DNS and Application servers, or atleast the application servers. There was some mention, and research I believe(?), about the value of having a partial Anycast deployment, so 3/4ths of your capacity on Anycast servers and 1/4th on 'normal' hosts to guard against route flaps and dampening of prefixes... I'm sure that some of the existing anycast users could provide much mode relevant real-world experiences though. -chris

On Wed, 1 Sep 2004, Steve Francis wrote:

Christopher L. Morrow wrote:

...

If I read your original request correctly you were planning on: 1) having presence in multiple datacenters (assume multiple providers as well) 2) having a 'authoritative' DNS server in each facility (or 2/3/4 whatever per center) 3) return datacenter-1-host-1 from datacenter-1-authserver-1, datacenter-2-host-2 from datacenter-2-authserver-1, and so forth.

This isn't really 'anycast' so much as 'different A records depending on server which was asked'

Well, there'd be one NS record returned for the zone in question. That NS record would be an IP address that is anycasted from all the datacenters. So end users (or their DNS servers) would all query the same IP address as the NS for that zone, but would end up at different datacenters depending on the whims of the anycasted BGP space.

Hmm, why not anycast the service/application ips? Having inconsistent DNS info seems like a problem waiting to bite your behind.

...

I suspect you'd really also introduce some major troubleshooting headaches with this setup, not just for you, but for your users as well.

I don't doubt that. :-)

which I'd think you'd want to minimize as much as possible, right?

On Wed, 1 Sep 2004, James wrote:

On Wed, Sep 01, 2004 at 08:00:53PM +0000, Christopher L. Morrow wrote:

...

On Wed, 1 Sep 2004, Steve Francis wrote:

...

Christopher L. Morrow wrote:

Hmm, why not anycast the service/application ips? Having inconsistent DNS info seems like a problem waiting to bite your behind.

Which begs the question.. is anyone doing this right now? I've been wondering about the potential issues wrt anycasting tcp applications.. TCP sessions would be affected negatively during a route change..

short-lived tcp is probably ok though (like static webpages or something of that sort) you'll also have to watch out for maintaining state for distributed application servers (I suppose). TCP anycast has many more complicated implications than UDP/DNS things, or so it seems to my untrained/educated eye.

...

This isn't really 'anycast' so much as 'different A records depending on server which was asked'

right.

Well, there'd be one NS record returned for the zone in question. That NS record would be an IP address that is anycasted from all the datacenters. So end users (or their DNS servers) would all query the same IP address as the NS for that zone, but would end up at different datacenters depending on the whims of the anycasted BGP space.

that's generic dns anycast. it's safe if your routing team is very strong.

Once they reached a name server, then yes, it changes to 'different A records depending on server which was asked'

that's incoherent dns. when i first began castigating people in public for this, i coined the term "stupid dns tricks" to describe this behaviour. cisco now has products that will do this for you. many web hosting companies offer this incoherence as though it were some kind of feature. akamai at one time depended on it, speedera at one time did not, i don't know what's happening currently, perhaps they've flipflopped. dns is not a redirection service, and incoherence is bad. when you make a query you're asking for a mapping of <name,class,type,time> to an rrset. offering back a different rrset based on criteria like source ip address, bgp path length, ping rtt, or the phase of the moon, is a protocol violation, and you shouldn't do it. the only way to make this not be a protocol violation is to use zero TTL's to prohibit caching/reuse, which is also bad but for a different reason.

...

I suspect you'd really also introduce some major troubleshooting headaches with this setup, not just for you, but for your users as well.

I don't doubt that. :-)

not only is it bad dns, it's bad web service. the fact that a current routing table gives a client's query to a particular anycasted DNS server does not mean that the web services mirror co-located with that DNS server is the one that would give you the best performance. for one thing, the client's dns forwarding/caching resolver might have a different position in the connectivity graph than the web client. for another thing, as-path length doesn't tell you anything about current congestion or bandwidth -- BGP is not IGRP (and thank goodness!). if you want a web client to get its web data from the best possible web services host/mirror out of a distributed cluster, then you will have to do something a hell of a lot smarter than incoherent dns. there are open source packages to help you do this. they involve sending back an HTTP redirect to clients who would be best served by some other member of the distributed mirror cluster. -- Paul Vixie

On Wed, 1 Sep 2004, Steve Francis wrote:

Paul Vixie wrote:

...

not only is it bad dns, it's bad web service. the fact that a current routing table gives a client's query to a particular anycasted DNS server does not mean that the web services mirror co-located with that DNS server is the one that would give you the best performance. for one thing, the client's dns forwarding/caching resolver might have a different position in the connectivity graph than the web client. for another thing, as-path length doesn't tell you anything about current congestion or bandwidth -- BGP is not IGRP (and thank goodness!).

I'm aware that web clients are not colocated with the client's name server, and that BGP does not attempt to optimise performance.

However, I suspect that in most cases, the client is close enough to the name server, and the BGP best path is close enough to the best path if it were based on latency, that most clients would be happy with the result most of the time. I'm not aiming for 100%, just Good Enough.

This is not always a good assumption: 1) dial clients sometimes get their DNS info from their radius profile (I believe) sometimes that dns server isn't on the same ASN as the dialup link. 2) many people have hardcoded DNS servers over the years, ones that have drifted from 'close' to 'far' 3) corporations with multiple exit points and larger internal networks might have DNS servers that exit in one country but are queried internally from other countries/states/locations. I think Paul's partly pointing out that you are using DNS for the wrong thing here, and partly pointing out that you are going to increase your troubleshooting overhead/complexity... Users on network X that you expect to use datacenter Y are really accessing datacenter Z because their dns cache server is located on network U :( I'm glad to see Joe/Paul/Bill jump in though... they do know quite a bit more about the practice of anycasting services on large networks.

On Wed, 1 Sep 2004, Steve Francis wrote: >>> I'm sure there is research out there... >> Why? :-) > Usual - if I build it myself, will it work well enough, or should I pony > up for a CDN? Uh, what about that makes you sure that there's research out there? > I thought it was standard best practice for availability, like for root > name servers. I thought it was not a good "closest server" selection > mechanism, as you'll be going to the closest server as determined by BGP > - which may have little relationship to the server with lowest RTT. And the lowest RTT doesn't necessarily have much to do with what's closest. If you want lowest RTT, that's what the DNS client already does for you, so you don't need to do anything at all. -Bill

So there is no need to anycast the DNS servers and rely on BGP topology for selection. Instead use bind's behaviour so that each resolving nameserver will be querying the authoritative nameserver that responds the fastest.

However, note that only BIND does this. djbdns always selects nameservers randomly and the Windows selection algorithm is somewhat of a mystery. See http://www.nanog.org/mtg-0310/wessels.html Duane W.

On 2 Sep 2004, at 06:05, Bill Woodcock wrote:

If you want nearest server, anycast will give you that essentially 100% of the time.

Just to clarify this slightly, since I've known people to misinterpret this point: a clear, contextual understanding of the word "nearest" is important in understanding this sentence. Here's an example: France Telecom was an early supporter of F-root's anycast deployment in Hong Kong. Due to the peering between OpenTransit and F at the HKIX, the nearest F-root server to OT customers in Paris was in Asia, despite the fact that there were other F-root nodes deployed in Europe. Those OT customers were indeed reaching the nearest F-root node, or maybe they weren't, depending on what you understand by the word "near". Another one: where anycast nodes are deployed within the scope of an IGP, topological nearness does not necessarily indicate best performance (since not all circuits will have the same loading, in general, and maybe a short, congested hop is not as "near" as several uncongested hops). For F, we don't worry too much about which flavour of "near" we achieve for every potential client: redundancy/diversity/reliability/availability is more important than minimising the time to do a lookup, and the fact that the "near" we achieve in many cases corresponds to what human users expect it to mean is really just a bonus. However, in the general case it's important to understand what kind of "near" you need, and to deploy accordingly. Joe

On 01.09 15:18, Matt Larson wrote:

I can give you one data point: VeriSign anycasts j.root-servers.net from all the same locations (minus one) where the com/net authoritative servers (i.e., *.gtld-servers.net) are located. An informal examination of query rates among all the J root instances (traffic distribution via BGP) vs. query rates among all the com/net servers (traffic distribution via iterative resolver algorithms, which means round trip time in the case of BIND and Microsoft) shows much more even distribution when the iterative resolvers get to pick vs. BGP. ....

For what it's worth,

Thanks Matt for sharing this observation. Based on my own informal observations this has to be taken with a truck load of NaCl. The load characteristics of TLD servers and root servers are vastly different. The root servers usually get large amounts of (bogus) load from relatively few sources whereas the sources of load for TLD servers are more evenly distributed to start with. Daniel