Re: [dnsop] DNS Anycast revisited (fwd)
This was Vixie's last post on the subject of Anycast on DNSOP. NB: Patrick Gilmore and Chris Morrow, note that Vixie agrees that HTTP anycast is a bad idea. Note the nonsense about anycast being "completely coherent". Note also that Vixie continues to ignore per-packet load balancing issues, and focuses on route-change times, instead. ---------- Forwarded message ---------- Date: 29 Mar 2005 22:46:27 +0000 From: Paul Vixie <vixie@vix.com> To: dnsop@lists.uoregon.edu Subject: Re: [dnsop] DNS Anycast revisited david.conrad@nominum.com (David Conrad) writes:
In my experience, shared unicast DNS provides quite a few benefits, particularly in the context of ISPs or services that need to be highly available, at the cost of some additional routing configuration complexity. There are, of course, situations in which the costs of shared unicast DNS outweigh the benefits, but I've found those situations to be rare in larger networks.
i figure this is as good a time to mention this as any. david conrad was the first voice for wide scale ipv4 anycast of root name servers, and when f-root started deploying this (in the months before the october 2002 ddos) it was because david and i had been sharing an office and talking about it. ("and it makes for great security/resiliency slideware.") for the record, i remain convinced that unowned anycast (where the prefix being advertised isn't solely controlled by a single entity worldwide) is dangerous and should not be done except in cases like AS112 (www.as112.net). ("but it makes for great socialist-internet slideware.") while i'm on the subject, i also remain convinced that using anycast to do distributed load balancing for applications like WWW, on the assumption that the path you heard a dns query on is instructive as to what content would be best to answer with, is silly, and will more often do harm or do nothing than do good. (and i've told akamai and speedera this many times.) ("but it makes for great marketing slideware.") lest anyone be confused, ultradns's anycast for .ORG is completely coherent and doesn't admit the possibility of giving out different responses from different anycast nodes for policy reasons or any other reason, and so it's an example of "good" anycast the way i count such things. finally, a word about tcp. even the most pessimistic route-change measurements (from verisign and IIJ) wouldn't affect tcp performance for transactions as short-lived as occur with dns queries. but that's not a justification for switching to tcp. if we believe that EDNS0's buffer size management isn't good enough, then we can bring back the MD bit from an old EDNS1 proposal. but we won't be holding full tcp session state in dns servers. nope nope nope. -- Paul Vixie . dnsop resources:_____________________________________________________ web user interface: http://darkwing.uoregon.edu/~llynch/dnsop.html mhonarc archive: http://darkwing.uoregon.edu/~llynch/dnsop/index.html
On Tue, 3 May 2005, Mark Boolootian wrote:
Note the nonsense about anycast being "completely coherent".
If you check, I think you'll see that he actually said "ultradns's anycast for .ORG is completely coherent".
There seems to be no possibility for anycast to be "completely coherent", so ultradns' anycast couldn't be "completely coherent" either. But Vixie mentions it to respond to comments by others about Ultradns' particularly pervasive use of anycast. --Dean -- Av8 Internet Prepared to pay a premium for better service? www.av8.net faster, more reliable, better service 617 344 9000
----- Original Message ----- From: "Dean Anderson" <dean@av8.com> To: "Mark Boolootian" <booloo@ucsc.edu> Cc: <Nanog@merit.edu> Sent: Tuesday, May 03, 2005 6:33 PM Subject: Re: [dnsop] DNS Anycast revisited (fwd)
On Tue, 3 May 2005, Mark Boolootian wrote:
Note the nonsense about anycast being "completely coherent".
If you check, I think you'll see that he actually said "ultradns's anycast for .ORG is completely coherent".
There seems to be no possibility for anycast to be "completely coherent", so ultradns' anycast couldn't be "completely coherent" either. But Vixie mentions it to respond to comments by others about Ultradns' particularly pervasive use of anycast.
it may not be possible to make every service *consistent*, but it is perfectly possible to make it coherent (i'm talking about coherency of copies of a shared resource). i'm curious to see how you can substantiate this claim, since any backend which supports distributed transaction semantics will give you this. i can't comment on the veracity of paul's statement comme applique ultradns, since i'm not familiar with how they do things, but that doesn't change the fact that you've just made a statement which appears blatantly false to anyone with any distributed systems experience. -p --- paul galynin
On Tue, May 03, 2005 at 06:59:45PM -0400, Paul G wrote:
----- Original Message ----- From: "Dean Anderson" <dean@av8.com> To: "Mark Boolootian" <booloo@ucsc.edu> Cc: <Nanog@merit.edu> Sent: Tuesday, May 03, 2005 6:33 PM Subject: Re: [dnsop] DNS Anycast revisited (fwd)
On Tue, 3 May 2005, Mark Boolootian wrote:
Note the nonsense about anycast being "completely coherent".
If you check, I think you'll see that he actually said "ultradns's anycast for .ORG is completely coherent".
There seems to be no possibility for anycast to be "completely coherent", so ultradns' anycast couldn't be "completely coherent" either. But Vixie mentions it to respond to comments by others about Ultradns' particularly pervasive use of anycast.
it may not be possible to make every service *consistent*, but it is perfectly possible to make it coherent (i'm talking about coherency of copies of a shared resource). i'm curious to see how you can substantiate this claim, since any backend which supports distributed transaction semantics will give you this. i can't comment on the veracity of paul's statement comme applique ultradns, since i'm not familiar with how they do things, but that doesn't change the fact that you've just made a statement which appears blatantly false to anyone with any distributed systems experience.
Coherency of DNS service is dependent upon the operational clue of the hostmaster in charge of such DNS systems and ensuring that each specific system is maintained to provide consistent service. (i.e. ensuring zones are properly updated on all servers, ensuring each system can answer the queries by enduser with proper answer) Anycast obviously opens a small set of can of caveats and notes while providing benefits. A properly configured and maintained anycast DNS setup can provide just as much coherency than using different unicast addresses, and ofcourse YMMV. And this has been done and is already being done by more ISPs to simplify their resolver DNS service. Honestly I've heard more success stories with anycast deployments than failure stories from the operational community thus far. Failure stories I've heard so far with respect to operational caveats of anycast were primarily brought up by those researching different and varying scenarios, many of which are very uncommon on the normal internet, but not primarily from those who actually operate a network and striving to provide service. Just as with anycast, people implementing per-packet load balancing should be aware of its caveats and implications before turning it on without reading the fine manual. Also note that most of popular setup of per-packet load balancing is done on a setup where multiple circuits are bound to the _same_ router, to provide added throughput. For most people, the extensive jitter provided by enabling PPLB accross different uplink POPs/ISPs is enough to fail the Return On Investment (ROI) of doing so. Before we even get to jitter, I would hate to see a sporadic sick-looking traceroute for all hops beyond the point where PPLB is enabled, due to PPLB being enabled between different IGP paths or uplink carriers. I myself would agree with the most of the anycast-supportive operators in that we should rather worry about tangible reality operational problems, not theories or lab-environment-generated proof of concepts. To really understand just what anycast is, I find this pdf introduction pretty useful for many folks who are new to the anycast topic and are looking to find some value in this long thread :) -- <http://www.nanog.org/mtg-0310/miller.html> -J (another person who deploys anycast for a few networks and have been very happy with the results it provided)
-p
--- paul galynin
-- James Jun Infrastructure and Technology Services TowardEX Technologies Office +1-617-459-4051 x179 | Mobile +1-978-394-2867 james@towardex.com | www.towardex.com
On 3 May 2005, at 20:07, James wrote:
Anycast obviously opens a small set of can of caveats and notes while providing benefits.
Comments on (and contributions to) draft-ietf-grow-anycast-00 would be gratefully received (by private mail, probably, rather than on this list). Joe
On Tue, 3 May 2005, Paul G wrote:
There seems to be no possibility for anycast to be "completely coherent", so ultradns' anycast couldn't be "completely coherent" either. But Vixie mentions it to respond to comments by others about Ultradns' particularly pervasive use of anycast.
it may not be possible to make every service *consistent*, but it is perfectly possible to make it coherent (i'm talking about coherency of copies of a shared resource).
This seems to be a trivial interpretation of "coherent". It is assumed that the copies of DNS _zones_ are kept in sync regardless of whether the servers are to traditional replicas or to anycasted replicas. No one ever claimed that zone transfers between the copies would be affected by anycast. The "in-sync"-ness of the zone data is competely orthogonal to anycast. Roots are updated via back channels on non-anycast addresses, and not with AXFR. Though, the whole "complete coherency" statement seems to be nonsense, so I can't really say that it wasn't what he meant. If it was, it had no bearing on anything, certainly not on any concerns having to do with Ultradns' operations. The concern about Ultradns was the pervasive use of anycast. As just pointed out, it seems they've altered this, perhaps as a result of these concerns being aired. (You're welcome, but save the thanks until we have a solid solution) The following is interesting: *some* Vixie associates thought that asking Nanog how many roots were anycast was considered off topic: (I think this query also resulted from DNSOP discussions in Oct/Nov '04.) (now, wasn't randy bush one of the people who was doing ad hominems on me just now? So he knows I have a valid point, but took a shot anyway. Nice. Get 'em while you can, I guess) ========================================================== On Thu, Nov 11, 2004 at 06:03:42PM -0800, Randy Bush wrote:
which roots are anycast? c f i j k? b m
thanks.
which are widely anycast, i.e. at more than three or four locations OR on three or more continents?
randy
and the good folks on nanog would know this why? --bill ========================================================== -- Av8 Internet Prepared to pay a premium for better service? www.av8.net faster, more reliable, better service 617 344 9000
On Tue, 3 May 2005, Paul G wrote:
There seems to be no possibility for anycast to be "completely coherent", so ultradns' anycast couldn't be "completely coherent" either. But Vixie mentions it to respond to comments by others about Ultradns'
----- Original Message ----- From: "Dean Anderson" <dean@av8.com> To: "Paul G" <paul@rusko.us> Cc: <nanog@merit.edu> Sent: Tuesday, May 03, 2005 8:35 PM Subject: Re: [dnsop] DNS Anycast revisited (fwd) particularly
pervasive use of anycast.
it may not be possible to make every service *consistent*, but it is perfectly possible to make it coherent (i'm talking about coherency of copies of a shared resource).
This seems to be a trivial interpretation of "coherent". It is assumed that the copies of DNS _zones_ are kept in sync regardless of whether the servers are to traditional replicas or to anycasted replicas. No one ever claimed that zone transfers between the copies would be affected by anycast. The "in-sync"-ness of the zone data is competely orthogonal to anycast. Roots are updated via back channels on non-anycast addresses, and not with AXFR.
i'm terribly sorry, but i'm unable to extract any meaning at all from these statements. when i parse them, they make no sense at all (not in terms of being wrong, just not understandable). could you rephrase them? coherency and consistency are well-defined terms in systems engineering. we are talking about dns queries and hence coherency of zone data (the shared resource). i fail to see how this is open to any interpretation at all. i snipped the rest for obvious reasons. -p
On Tue, 3 May 2005, Paul G wrote:
i'm terribly sorry, but i'm unable to extract any meaning at all from these statements. when i parse them, they make no sense at all (not in terms of being wrong, just not understandable). could you rephrase them?
coherency and consistency are well-defined terms in systems engineering. we are talking about dns queries and hence coherency of zone data (the shared resource). i fail to see how this is open to any interpretation at all.
Sorry, The original statement Vixie made is nonsense. Here is the original statement again: Vixe writes: lest anyone be confused, ultradns's anycast for .ORG is completely coherent and doesn't admit the possibility of giving out different responses from different anycast nodes for policy reasons or any other reason, and so it's an example of "good" anycast the way i count such things. Vixie seems to be responding to concern raised for Ultradns' pervasive use of anycasting. This was the only issue raised involving Ultradns. During the anycast discussion on DNSOP, the subject of zone coherency (as normally used) was not an issue. So there is no question of zone coherency for Ultradns' servers. We assumed (and did not dispute) that zone updates were unaffected by anycast. Zone updates happen over private secure channels on non-anycasted IP addreses. They ought to be as coherent as DNS gets. They ought not be affected by anycast. Vixie ends by saying essentially, that because of Ultradns' coherency, it is an example of "good anycast". But the two issues (coherency and anycast) have no relationship. There is no reason to conclude that coherency means anycast is either good or bad. Hence, his statement is nonsense. Alternately, one might read "coherency" as refering to the shared TCP state of the group of servers. The "shared TCP state" of a group of anycast servers is conceptual. There is no actual coherency for this shared TCP state. So that is nonsense, too. I suspect the statement is just nonsense, that is hoped to sound good. Many people will not really read or parse his whole statement, but will remember 'Ultradns is an example of good anycast, according to Vixie' But, given that the original statement is nonsense, I can't really say that any of these are correct. --Dean -- Av8 Internet Prepared to pay a premium for better service? www.av8.net faster, more reliable, better service 617 344 9000
Dean Anderson wrote:
On Tue, 3 May 2005, Paul G wrote:
i'm terribly sorry, but i'm unable to extract any meaning at all from these statements. when i parse them, they make no sense at all (not in terms of being wrong, just not understandable). could you rephrase them?
coherency and consistency are well-defined terms in systems engineering. we are talking about dns queries and hence coherency of zone data (the shared resource). i fail to see how this is open to any interpretation at all.
Sorry, The original statement Vixie made is nonsense. Here is the original statement again:
Vixe writes: lest anyone be confused, ultradns's anycast for .ORG is completely coherent and doesn't admit the possibility of giving out different responses from different anycast nodes for policy reasons or any other reason, and so it's an example of "good" anycast the way i count such things.
Vixie seems to be responding to concern raised for Ultradns' pervasive use of anycasting. This was the only issue raised involving Ultradns.
During the anycast discussion on DNSOP, the subject of zone coherency (as normally used) was not an issue. So there is no question of zone coherency for Ultradns' servers. We assumed (and did not dispute) that zone updates were unaffected by anycast. Zone updates happen over private secure channels on non-anycasted IP addreses. They ought to be as coherent as DNS gets. They ought not be affected by anycast.
Vixie ends by saying essentially, that because of Ultradns' coherency, it is an example of "good anycast". But the two issues (coherency and anycast) have no relationship. There is no reason to conclude that coherency means anycast is either good or bad. Hence, his statement is nonsense.
Context helps. In the previous paragraph Vixie said:
while i'm on the subject, i also remain convinced that using anycast to do distributed load balancing for applications like WWW, on the assumption that the path you heard a dns query on is instructive as to what content would be best to answer with, is silly, and will more often do harm or do nothing than do good. (and i've told akamai and speedera this many times.) ("but it makes for great marketing slideware.")
In other words this is a bad idea: [FT@fenrir FT]$ dig a248.e.akamai.net @69.45.79.10 ;; ANSWER SECTION: a248.e.akamai.net. 20 IN A 80.67.72.214 a248.e.akamai.net. 20 IN A 80.67.72.201 FT@inuyasha:~$ dig a248.e.akamai.net @69.45.79.10 ;; ANSWER SECTION: a248.e.akamai.net. 20 IN A 69.45.79.15 a248.e.akamai.net. 20 IN A 69.45.79.16 While I'm not a mind reader, It seems he's saying that, since Ultradns doesn't use anycast to do this, it is an example of 'good anycast.'
On May 3, 2005, at 10:28 PM, Nicholas Suan wrote:
In the previous paragraph Vixie said:
while i'm on the subject, i also remain convinced that using anycast to do distributed load balancing for applications like WWW, on the assumption that the path you heard a dns query on is instructive as to what content would be best to answer with, is silly, and will more often do harm or do nothing than do good. (and i've told akamai and speedera this many times.) ("but it makes for great marketing slideware.")
In other words this is a bad idea:
[FT@fenrir FT]$ dig a248.e.akamai.net @69.45.79.10
;; ANSWER SECTION: a248.e.akamai.net. 20 IN A 80.67.72.214 a248.e.akamai.net. 20 IN A 80.67.72.201
FT@inuyasha:~$ dig a248.e.akamai.net @69.45.79.10
;; ANSWER SECTION: a248.e.akamai.net. 20 IN A 69.45.79.15 a248.e.akamai.net. 20 IN A 69.45.79.16
While I'm not a mind reader, It seems he's saying that, since Ultradns doesn't use anycast to do this, it is an example of 'good anycast.'
I'm not a mind reader either, but I read English. (Well, I try sometimes. :) Paul said: <quote> i also remain convinced that using anycast to do distributed load balancing for applications like WWW, ... is silly, and will more often do harm or do nothing than do good. (and i've told akamai and speedera this many times.) </quote> The fact your digs returned different IPs for the same hostname has _nothing_ to do with anycast. Whether (or not) UltraDNS doing this (or not) is good (or not) seems, to me at least, to be apples & oranges. -- TTFN, patrick
Patrick W. Gilmore wrote:
<quote> i also remain convinced that using anycast to do distributed load balancing for applications like WWW, ... is silly, and will more often do harm or do nothing than do good. (and i've told akamai and speedera this many times.) </quote>
The fact your digs returned different IPs for the same hostname has _nothing_ to do with anycast.
I never said that anycast was required to do this. I could do the same thing with BIND's views. I was interpreting it as 'using anycast to provide incoherent results is a bad idea'.
nsuan@nonexiste.net (Nicholas Suan) writes:
In the previous paragraph Vixie said:
while i'm on the subject, i also remain convinced that using anycast to do distributed load balancing for applications like WWW, on the assumption that the path you heard a dns query on is instructive as to what content would be best to answer with, is silly, and will more often do harm or do nothing than do good. (and i've told akamai and speedera this many times.) ("but it makes for great marketing slideware.")
In other words this is a bad idea:
[FT@fenrir FT]$ dig a248.e.akamai.net @69.45.79.10
;; ANSWER SECTION: a248.e.akamai.net. 20 IN A 80.67.72.214 a248.e.akamai.net. 20 IN A 80.67.72.201
FT@inuyasha:~$ dig a248.e.akamai.net @69.45.79.10
;; ANSWER SECTION: a248.e.akamai.net. 20 IN A 69.45.79.15 a248.e.akamai.net. 20 IN A 69.45.79.16
While I'm not a mind reader, It seems he's saying that, since Ultradns doesn't use anycast to do this, it is an example of 'good anycast.'
yes. i thought that was clear. oops. (why is this all happening on nanog@ rather than dnsop@?) -- Paul Vixie
On Tue, 3 May 2005, Mark Boolootian wrote:
Note the nonsense about anycast being "completely coherent".
If you check, I think you'll see that he actually said "ultradns's anycast for .ORG is completely coherent".
And last time I checked -- on this list, mind you -- it certainly was not. Cf. people trying to run and hide, or lash out at me for complaining, when I pointed out how two anycast routes pointing to the same dead node made the .ORG anycast implementation unusable. I reserve judgment on whether their implementation has been fixed in the meantime; I have no evidence either way at the moment. -- -- Todd Vierling <tv@duh.org> <tv@pobox.com> <todd@vierling.name>
TV> Date: Tue, 3 May 2005 22:21:45 -0400 (Eastern Daylight Time) TV> From: Todd Vierling [ trimming CC list before it grows too long ] TV> And last time I checked -- on this list, mind you -- it certainly TV> was not. Cf. people trying to run and hide, or lash out at me for TV> complaining, when I pointed out how two anycast routes pointing to TV> the same dead node made the .ORG anycast implementation unusable. Akamai's service uses non-coherent DNS by design. Your post referenced a failure case in which DNS service was not coherent by virtue of certain pods not responding; UDNS attempts to provide coherent DNS service. TV> I reserve judgment on whether their implementation has been fixed in the "me too" TV> meantime; I have no evidence either way at the moment. One of the challenges of anycast is failure detection and mitigation. <mumbles> flooding clusters via source-based routing tunneling anycast-destined OAM packets via unicast ns-to-machine affinity within pods tight coupling of DNS service to anycast route injection </mumbles> Anycast implementation _does_ present new operational challenges, but they're hardly insurmountable. Eddy -- Everquick Internet - http://www.everquick.net/ A division of Brotsman & Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita ________________________________________________________________________ DO NOT send mail to the following addresses: davidc@brics.com -*- jfconmaapaq@intc.net -*- sam@everquick.net Sending mail to spambait addresses is a great way to get blocked. Ditto for broken OOO autoresponders and foolish AV software backscatter.
On 5/3/05 7:21 PM, "Todd Vierling" <tv@duh.org> wrote:
If you check, I think you'll see that he actually said "ultradns's anycast for .ORG is completely coherent".
And last time I checked -- on this list, mind you -- it certainly was not.
If you want to be taken seriously, perhaps you might want to research the use of the term coherency in terms of DNS. Or not. Hint: It doesn't mean that one node responds and another doesn't. Rodney Joffe CenterGate Research Group, LLC http://www.centergate.com "Technology so advanced, even WE don't understand it"(R)
On Tue, 3 May 2005, Rodney Joffe wrote:
If you check, I think you'll see that he actually said "ultradns's anycast for .ORG is completely coherent".
And last time I checked -- on this list, mind you -- it certainly was not.
If you want to be taken seriously, perhaps you might want to research the use of the term coherency in terms of DNS.
Hint: It doesn't mean that one node responds and another doesn't.
<feedback loop="trollbait"> It certainly breaks coherency in my mind when one query host receives SERVFAIL for a while, then nothing at all, but another query host is receiving records that the first query host *should* have seen. </feedback> After all these years, I should know better than to feed back the bait. Maybe it's just fun to do. :) That said, I'll credit you for catching my brain fart. I read right past 'coherent' in vix's text and saw the phrase '"good" anycast' in relation to the .ORG zone. It was not technically a *coherency* flaw to which I referred, but rather a fundamental *architecture* flaw. (It's too bad that the architectural flaw was never really discussed, or for that matter subjected to a mea culpa and/or public postmortem. But I suppose that is the pride/ego component of running such a large zone -- even one intended for the "public interest".) I wrote and maintain a query host deterministic, but inter-site coherent, anycasted DNS server for my $orkplace, so don't presume to tell me what you think I know. It only makes you look like a holier-than-thou ass [again]. Feel free to feed the trolls elsewhere, though, if you wish; it can be fun to watch. -- -- Todd Vierling <tv@duh.org> <tv@pobox.com> <todd@vierling.name>
If you check, I think you'll see that he actually said "ultradns's anycast for .ORG is completely coherent".
And last time I checked -- on this list, mind you -- it certainly was not.
well, for balance, every time i've checked, it certainly WAS. i was recently engaged as a consultant to study ultradns's .ORG architecture with regard to some complaints seen on an ICANN discussion board around the .NET bid. (since my employer was offering services to a different bidder, i was considered to be an objective evaluator.) ultradns's .ORG architecture is designed to achieve, and actually does achieve, excellent DNS coherence. -- Paul Vixie
On May 3, 2005, at 5:50 PM, Dean Anderson wrote:
This was Vixie's last post on the subject of Anycast on DNSOP.
NB: Patrick Gilmore and Chris Morrow, note that Vixie agrees that HTTP anycast is a bad idea.
Reasonable people can disagree. I think Paul is reasonable, I hope he thinks I am reasonable. We disagree. Doesn't mean I won't buy him a drink at the next NANOG. :-) Tell you what, if you come to Seattle, I will buy you a drink too, just to smooth things over. Maybe we can agree to disagree, reasonably. Also, whether Paul thinks it is a bad idea, or even if it really is a bad idea, that does not change the fact it was in wide use in the past, and probably still is used by many people without any serious problems. NB [translation, "operational content"]: Akamai does not use any anycast for HTTP. I am not at all certain why Paul is telling us this is a bad idea, since we don't do it. Then again, we might in the future, I am not privy to every decision in the company. (No, that is not a "hint", I really do not think we will do anycast HTTP for content delivery, but I also really do not know everything we will do in the future.) Either way, Paul's advice is always welcome, even if we don't listen sometimes. =) -- TTFN, patrick
PWG> Date: Tue, 3 May 2005 18:03:12 -0400 PWG> From: Patrick W. Gilmore PWG> NB [translation, "operational content"]: Akamai does not use any PWG> anycast for HTTP. I am not at all certain why Paul is telling us PWG> this is a bad idea, since we don't do it. Then again, we might in PWG> the future, I am not privy to every decision in the company. (No, PWG> that is not a "hint", I really do not think we will do anycast HTTP PWG> for content delivery, but I also really do not know everything we PWG> will do in the future.) One also should distinguish between TCP _to_ an anycasted address and TCP _from_ an anycasted address. The latter is trickier, as asymmetric routing increases the chances that the session will need to be transferred to another pod: ac LAX --> uc PHX uc PHX --> ac DFW This presses the question of synchronization (how do LAX and DFW know about each other?) and source-based routing. IMNSHO there's more overlap between these two topics than one might expect, too. When anycast gets _really_ interesting is when an anycasted client makes a request [from an anycasted address] to an anycasted server. The results resemble persistent BGP oscillation due to poor MED choices: ac LAX --> ac PHX ac PHX --> ac DFW ac DFW --> ac ORD ac ORD --> ac SJC ac SJC --> ac LAX I could see that happening with anycasted SMTP, but it would be less likely (although certainly not impossible) with HTTP. i.e., an HTTP client isn't as likely to be anycasted as an SMTP client. Far too much to cover in a noisy NANOG thread. :-( Hence the need to agree on just what "anycast" is before debating the merits. Eddy -- Everquick Internet - http://www.everquick.net/ A division of Brotsman & Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita ________________________________________________________________________ DO NOT send mail to the following addresses: davidc@brics.com -*- jfconmaapaq@intc.net -*- sam@everquick.net Sending mail to spambait addresses is a great way to get blocked. Ditto for broken OOO autoresponders and foolish AV software backscatter.
On May 3, 2005, at 7:38 PM, Edward B. Dreger wrote:
PWG> Date: Tue, 3 May 2005 18:03:12 -0400 PWG> From: Patrick W. Gilmore
PWG> NB [translation, "operational content"]: Akamai does not use any PWG> anycast for HTTP. I am not at all certain why Paul is telling us PWG> this is a bad idea, since we don't do it. Then again, we might in PWG> the future, I am not privy to every decision in the company. (No, PWG> that is not a "hint", I really do not think we will do anycast HTTP PWG> for content delivery, but I also really do not know everything we PWG> will do in the future.)
One also should distinguish between TCP _to_ an anycasted address and TCP _from_ an anycasted address. The latter is trickier, as asymmetric routing increases the chances that the session will need to be transferred to another pod:
Just to make life fun, there is the whole "anycast a bunch of name servers, each with different zone files pointing at local HTTP servers". Since the "anycast" portion is over UDP, it avoids a lot of the problems (real or otherwise) mentioned here, and the HTTP is still unicast but distributed and can be made resilient to failure. Of course, the DNS backend is then .. uh .. "de-coherent"? :-) But it works, and works well, in many currently operational configurations. Does PPLB (or anything else) break this? I'm certain I could find things that would break this if I looked hard enough. But as I've said many times, reality trumps NANOG posts. Since this is a _working_ configuration today, I would say that disproves any claims that it cannot or will not work. -- TTFN, patrick
PWG> Date: Tue, 3 May 2005 21:58:37 -0400 PWG> From: Patrick W. Gilmore PWG> Just to make life fun, there is the whole "anycast a bunch of name PWG> servers, each with different zone files pointing at local HTTP PWG> servers". Since the "anycast" portion is over UDP, it avoids a lot PWG> of the problems (real or otherwise) mentioned here, and the HTTP is PWG> still unicast but distributed and can be made resilient to failure. Actually, it's not because the anycast portion is over UDP _per se_. Someone hits an anycasted [Akamai] DNS server, which then transfers the session to a unicast IP address c/o DNS. Once the data is flowing via unicast, there's no further chance of transferring the session to another anycast box. DNS --> <whatever> is a quick-and-dirty way of accomplishing the session transfer once and for all without requiring additional protocol intelligence. It's also visible to the world, as opposed to behind-the-scenes session transfer. PWG> Of course, the DNS backend is then .. uh .. "de-coherent"? :-) But Comparing anycasted DNS service with CDNs is about as valid as comparing TCP with BGP. They serve different purposes at different levels, and one relies on the other. PWG> it works, and works well, in many currently operational configurations. Indeed. One could move beyond CDNs and cite many corporate WANs as examples of working configurations that lack coherent DNS. Unicast nameservers can lack coherency. As several others have said, anycast and coherency are orthogonal. PWG> Does PPLB (or anything else) break this? I'm certain I could find PWG> things that would break this if I looked hard enough. But as I've The trick is finding something that breaks your config without affecting "typical" ones. PWG> said many times, reality trumps NANOG posts. Since this is a PWG> _working_ configuration today, I would say that disproves any claims PWG> that it cannot or will not work. I'd not go so far as to say "disproves", but it shifts the burden of proof. Eddy -- Everquick Internet - http://www.everquick.net/ A division of Brotsman & Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita ________________________________________________________________________ DO NOT send mail to the following addresses: davidc@brics.com -*- jfconmaapaq@intc.net -*- sam@everquick.net Sending mail to spambait addresses is a great way to get blocked. Ditto for broken OOO autoresponders and foolish AV software backscatter.
Sorry if it was not clear, but my post had _nothing_ to do with "CDNs". I was just talking about people setting up anycast name servers, each of which pointed at a different HTTP server (or other service), to spread load. In many cases, the two servers are the same. -- TTFN, patrick On May 3, 2005, at 10:27 PM, Edward B. Dreger wrote:
PWG> said many times, reality trumps NANOG posts. Since this is a PWG> _working_ configuration today, I would say that disproves any claims PWG> that it cannot or will not work.
I'd not go so far as to say "disproves", but it shifts the burden of proof.
No, it disproves. You say "it will not / cannot work". Showing you a working instance in production absolutely disproves your statement. You can say "it might break", but that's a different statement. -- TTFN, patrick
PWG> Date: Tue, 3 May 2005 23:56:48 -0400 PWG> From: Patrick W. Gilmore PWG> I was just talking about people setting up anycast name servers, each PWG> of which pointed at a different HTTP server (or other service), to PWG> spread load. In many cases, the two servers are the same. Ah, okay... which again helps demonstrate the lack of coupling between *cast and coherency. A single unicast DNS server can serve split views just as easily. PWG> No, it disproves. You say "it will not / cannot work". Showing you PWG> a working instance in production absolutely disproves your statement. PWG> PWG> You can say "it might break", but that's a different statement. Yes. I misread/misthought "disproves will not / cannot work" as "proves can / will work". Speaking of things that, and people who, are incoherent... ;-) Eddy -- Everquick Internet - http://www.everquick.net/ A division of Brotsman & Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita ________________________________________________________________________ DO NOT send mail to the following addresses: davidc@brics.com -*- jfconmaapaq@intc.net -*- sam@everquick.net Sending mail to spambait addresses is a great way to get blocked. Ditto for broken OOO autoresponders and foolish AV software backscatter.
On May 4, 2005, at 12:08 AM, Edward B. Dreger wrote:
PWG> I was just talking about people setting up anycast name servers, each PWG> of which pointed at a different HTTP server (or other service), to PWG> spread load. In many cases, the two servers are the same.
Ah, okay... which again helps demonstrate the lack of coupling between *cast and coherency. A single unicast DNS server can serve split views just as easily.
Views and anycast serve different, although sometimes similar or overlapping, functions. (Which you know very well, Eddy.) Think we've gone down the path enough for people to understand at least some of anycast's pros & cons. If anyone here would like a compare/contrast on views vs. anycast, let us know and we'll discuss. But not tonight. It's late, and I wouldn't want to interrupt the flame-fest with actual operational content. :) -- TTFN, patrick
On Tue, 3 May 2005, Edward B. Dreger wrote:
When anycast gets _really_ interesting is when an anycasted client makes a request [from an anycasted address] to an anycasted server.
Why would anyone use an anycast address as a client? Wouldn't it be simpler to make all client connections from the machine's unicast address? Tony. -- f.a.n.finch <dot@dotat.at> http://dotat.at/ BISCAY: WEST 5 OR 6 BECOMING VARIABLE 3 OR 4. SHOWERS AT FIRST. MODERATE OR GOOD.
TF> Date: Wed, 4 May 2005 10:48:56 +0100 TF> From: Tony Finch TF> Why would anyone use an anycast address as a client? Wouldn't it be TF> simpler to make all client connections from the machine's unicast address? Maybe, maybe not. Take an anycast DNS provider that AXFR/IXFRs zones from its customers. Notifying them of all anycast addresses and keeping ACLs up-to-date isn't feasible. The obvious answer is to have a couple hosts pull zones from unicasted addresses. However, this creates a few small targets... the question is if DNS slaves would benefit sufficiently from increased splay to warrant the additional implementation complexity. Eddy -- Everquick Internet - http://www.everquick.net/ A division of Brotsman & Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita ________________________________________________________________________ DO NOT send mail to the following addresses: davidc@brics.com -*- jfconmaapaq@intc.net -*- sam@everquick.net Sending mail to spambait addresses is a great way to get blocked. Ditto for broken OOO autoresponders and foolish AV software backscatter.
On 4 May 2005, at 09:52, Edward B. Dreger wrote:
TF> Date: Wed, 4 May 2005 10:48:56 +0100 TF> From: Tony Finch
TF> Why would anyone use an anycast address as a client? Wouldn't it be TF> simpler to make all client connections from the machine's unicast address?
Maybe, maybe not.
Take an anycast DNS provider that AXFR/IXFRs zones from its customers. Notifying them of all anycast addresses and keeping ACLs up-to-date isn't feasible.
The obvious answer is to have a couple hosts pull zones from unicasted addresses.
Actually, the obvious answer is to use TSIG instead of address-based ACLs to authenticate zone transfers. But in cases where that's not possible (because the master servers don't want to implement it, and insist on address-based ACLs), there are hacks available. See http://www.isc.org/pubs/tn/isc-tn-2004-1.html#anchor14 for an example. Joe
participants (12)
-
Dean Anderson -
Edward B. Dreger -
James -
Joe Abley -
Mark Boolootian -
Nicholas Suan -
Patrick W. Gilmore -
Paul G -
Paul Vixie -
Rodney Joffe -
Todd Vierling -
Tony Finch