Re: Slashdot: Providers Ignoring DNS TTL?
In your note below you speak of 'moving on to something else' when PPLB comes.
PPLB destabilizes TCP. It elicits erroneous retransmissions, squanders capacity and lowers performance.
I would actually dispute this. I agree that PPLB will *occasionally* lead to out-of-order packets, which will lead to lower TCP performance *when it happens*. To many customers this is acceptable as long as PPLB gives them improved performance *most of the time*. And this is what we saw very clearly at my previous employer - PPLB worked very well, and gave clearly increased performance, *most of the time*. As mentioned in another message, I don't really believe PPLB is coming. Instead I believe PPLB is something which is probably being *less* used now than a few years ago, since other link bundling methods are more easily available now (than they were a few years ago) - and these link bundling methods occur at a layer below TCP, and are invisible to TCP (no packet reordering problems). Steinar Haug, Nethelp consulting, sthaug@nethelp.no
Steinar: There is a large body of work from competent and well known researchers that assert the claim. I certainly lack standing to question their results. Empirically, download speeds to home are nearly cut in half (18Mbps) from sources that are subjected to packet reordering along the path. More to the point however, I note that Jay is the author of RFC2100. I think he's just having a little bit of fun. My apologies for belaboring the performance issue. Bob At 04:23 PM 4/24/2005, sthaug@nethelp.no wrote:
In your note below you speak of 'moving on to something else' when PPLB comes.
PPLB destabilizes TCP. It elicits erroneous retransmissions, squanders capacity and lowers performance.
I would actually dispute this. I agree that PPLB will *occasionally* lead to out-of-order packets, which will lead to lower TCP performance *when it happens*. To many customers this is acceptable as long as PPLB gives them improved performance *most of the time*. And this is what we saw very clearly at my previous employer - PPLB worked very well, and gave clearly increased performance, *most of the time*.
As mentioned in another message, I don't really believe PPLB is coming. Instead I believe PPLB is something which is probably being *less* used now than a few years ago, since other link bundling methods are more easily available now (than they were a few years ago) - and these link bundling methods occur at a layer below TCP, and are invisible to TCP (no packet reordering problems).
Steinar Haug, Nethelp consulting, sthaug@nethelp.no
On Sun, 24 Apr 2005, Robert M. Enger wrote:
Steinar:
There is a large body of work from competent and well known researchers that assert the claim. I certainly lack standing to question their results.
Empirically, download speeds to home are nearly cut in half (18Mbps) from sources that are subjected to packet reordering along the path.
I'm trying to sort out the various claims here, since I think right now this is a case of people talking past each other, and arguing completely different points. First of all, let's ditch the term "PPLB." The usual alternative to per packet load balancing (what's been being talked about here) is per prefix load balancing, which would also be "PPLB." The abbreviation is therefore more confusing than anything else. Now, onto the argument that's going on here. Dean says "per packet load balancing is coming," and then goes on to assume it's going to be used in such a way that it will cause packets to route through widely divergent paths. Several others have responded that that would cause packet reordering and break TCP. Robert says that even used correctly (on identical circuits between the same set of routers), per packet load balancing can cause packet reordering. Steiner says that when used correctly, per packet load balancing causes packet reordering only rarely, and speeds things up enough when it doesn't that the slowdowns caused by occasional packet reordering may be worth putting up with. Robert says well known researchers say that packet reordering is bad. So, as far as I can tell, everybody except perhaps Dean agrees that: - Used incorrectly (on divergent paths), per packet load balancing can cause packet reordering. - Used correctly (on non-diverging paths), packet reordering doesn't happen often. - Packet reordering is bad, and should be avoided. I'm less clear on Dean's position, but I think it's something along the lines of: "Per packet load balancing over divergent paths is coming, by fiat from marketing departments even if engineers don't like it, and anything that doesn't play well with it needs fixing." While Dean focuses on anycast, that would presumably extend to TCP and to anything jitter sensitive, such as streaming audio or video. Anything that's being missed here, or does this sum it all up? -Steve
On Sun, 24 Apr 2005, Steve Gibbard wrote:
On Sun, 24 Apr 2005, Robert M. Enger wrote:
Steinar:
There is a large body of work from competent and well known researchers that assert the claim. I certainly lack standing to question their results.
Empirically, download speeds to home are nearly cut in half (18Mbps) from sources that are subjected to packet reordering along the path.
I'm trying to sort out the various claims here, since I think right now this is a case of people talking past each other, and arguing completely different points.
First of all, let's ditch the term "PPLB." The usual alternative to per packet load balancing (what's been being talked about here) is per prefix load balancing, which would also be "PPLB." The abbreviation is therefore more confusing than anything else.
Now, onto the argument that's going on here.
Dean says "per packet load balancing is coming," and then goes on to assume it's going to be used in such a way that it will cause packets to route through widely divergent paths.
Several others have responded that that would cause packet reordering and break TCP.
Robert says that even used correctly (on identical circuits between the same set of routers), per packet load balancing can cause packet reordering.
Steiner says that when used correctly, per packet load balancing causes packet reordering only rarely, and speeds things up enough when it doesn't that the slowdowns caused by occasional packet reordering may be worth putting up with.
Robert says well known researchers say that packet reordering is bad.
So, as far as I can tell, everybody except perhaps Dean agrees that:
- Used incorrectly (on divergent paths), per packet load balancing can cause packet reordering.
- Used correctly (on non-diverging paths), packet reordering doesn't happen often.
- Packet reordering is bad, and should be avoided.
I'm less clear on Dean's position, but I think it's something along the lines of:
"Per packet load balancing over divergent paths is coming, by fiat from marketing departments even if engineers don't like it, and anything that doesn't play well with it needs fixing." While Dean focuses on anycast, that would presumably extend to TCP and to anything jitter sensitive, such as streaming audio or video.
Anything that's being missed here, or does this sum it all up?
I think thats a fair summary. So agreeing for a second with Dean that indeed this behaviour would appear to be prohibited or at least inconsistent with the RFCs, the fact is anycast is widely deployed and is proven to be stable. Perhaps a solution to this is to look at what would be the best consistent view and to write an RFC to clarify this and obsolete the old ones that produce the inconsistency. I'm not sure what that would look like but that would appear to be a way to eliminate the theoretical problem.. Steve
On Mon, 25 Apr 2005, Stephen J. Wilcox wrote:
So agreeing for a second with Dean that indeed this behaviour would appear to be prohibited or at least inconsistent with the RFCs, the fact is anycast is widely deployed and is proven to be stable.
"vixie-cast" is deployed on around 60 or so root DNS servers. (don't know the exact number) That covers a wide spread of root DNS servers, but I wouldn't call that 'widely deployed'. I haven't been able to find any users of HTTP anycast/'vixie-cast' that Patrick Gilmore referred. There are also very few TCP DNS queries to the roots, so it isn't widely used at present, and hasn't been widely used in the past. I don't think it can be claimed that "vixie-cast" has been proven to be stable. ISC's assertions of stablity at a 2002 Nanog are what probably brought it to Dr. Bernstein's attention. Those assertions of stability are what's being challenged. You cannot assume them true.
Perhaps a solution to this is to look at what would be the best consistent view and to write an RFC to clarify this and obsolete the old ones that produce the inconsistency. I'm not sure what that would look like but that would appear to be a way to eliminate the theoretical problem..
Another solution is to urge OS vendors to implement RFC 1546 TCP anycast. In order to use RFC 1546 TCP anycast, it is necessary to implement changes in all clients that might access TCP anycast servers (as well as in the servers). This would probably require a long time frame, but still good to encourage. It might be easier to require this for IPV6---though I don't know that it isn't already required for IPV6. Another solution is not to do Vixie-cast. This may require clarification to DNS RFCs to specify that TCP queries will not be made to root DNS servers. It was previously thought that DNSSEC would require TCP, but this isn't the case in the latest round of RFC drafts. I can't think of anything else in the pipe that might require TCP DNS queries to root servers. Non-root servers usually don't need to do anycast, and aren't required to do TCP. So one could do anycast without TCP, if one wanted. But one ought know that anycast'ing DNS precludes TCP DNS. The "vixie-cast" HTTP doesn't *seem* to be widely in use, and there are numerous other solutions for HTTP. So simply recommending a halt to that would seem to be low-impact. --Dean -- Av8 Internet Prepared to pay a premium for better service? www.av8.net faster, more reliable, better service 617 344 9000
On Sat, 30 Apr 2005, Dean Anderson wrote:
On Mon, 25 Apr 2005, Stephen J. Wilcox wrote:
So agreeing for a second with Dean that indeed this behaviour would appear to be prohibited or at least inconsistent with the RFCs, the fact is anycast is widely deployed and is proven to be stable.
"vixie-cast" is deployed on around 60 or so root DNS servers. (don't know the exact number) That covers a wide spread of root DNS servers, but I wouldn't call that 'widely deployed'. I haven't been able to find any users of HTTP anycast/'vixie-cast' that Patrick Gilmore referred. There
we do this 'http anycast' as part of another service... it seems to work well enough.
On Sun, 24 Apr 2005, Steve Gibbard wrote:
On Sun, 24 Apr 2005, Robert M. Enger wrote:
Steinar:
There is a large body of work from competent and well known researchers that assert the claim. I certainly lack standing to question their results.
Empirically, download speeds to home are nearly cut in half (18Mbps) from sources that are subjected to packet reordering along the path.
I'm trying to sort out the various claims here, since I think right now this is a case of people talking past each other, and arguing completely different points.
First of all, let's ditch the term "PPLB." The usual alternative to per packet load balancing (what's been being talked about here) is per prefix load balancing, which would also be "PPLB." The abbreviation is therefore more confusing than anything else.
Err. No, that would be worse. "Per prefix" load balancing is an artifact of the Cisco route cache. The route engine (ie the route table) isn't queried for every packet. Instead the route in the route cache is used. One doesn't configure "per prefix" load balancing. One configures load balancing, which adds multiple routes into the route table. The route cache then causes only one of these routes to be used. On cisco, to enable PPLB, you turn off the route cache. On Juniper, you configure it to put multiple routes in the route table. Its actaully more likely to happen on Junipers, because unless you configure additonal policies, you get load balancing on divergent links as well as non-divergent links. On Cisco, the route cache is controlled on a per-interface basis.
Now, onto the argument that's going on here.
Dean says "per packet load balancing is coming," and then goes on to assume it's going to be used in such a way that it will cause packets to route through widely divergent paths.
Almost a correct statement of my position. But if its used in any case--anywhere--even on internal links, in any condition other than between exactly two routers, it will cause a problem with Vixie's TCP anycast. We aren't seeing a lot of problems yet because few people are using PPLB, and fewer are using TCP DNS, and so few are using Vixie's TCP anycast. If we change any our terms in this discussion, it should be to change our use of "TCP anycast". The behavior described by Vixie for DNS TCP "anycast" is unapproved** and non-standard. It does not conform to RFC 1546 TCP Anycast. RFC 1546 outlines changes to the TCP stack to support TCP anycast. To avoid confusion, I propose we call it "vixie-cast". And I also want to emphasize that RFC 1546 TCP anycast doesn't have any problems with PPLB. And of course UDP anycast also has no problem. It is only "Vixie-cast TCP" that has a problem. ** One of the criticisms of DNS Root Anycast is that Vixie recommended deployment of anycast to root server operators without first discussing this on the DNSOP WG. Dr. Bernstein first brought it to the WG's attention in 2002. And one of the recent discoveries is that a good number of DNS root server operators are using it, and had assumed the Vixie was the IETF liason communicating with the IETF. The IETF has never approved of this, and in fact, Vixie's version of TCP anycast is not what is described in RFC 1546. Vixie's version really shouldn't be called anycast, since it creates confusion with RFC 1546 recommendations. Indeed, if RFC 1546 recommendations for TCP anycast were widely implemented in TCP stacks, then there wouldn't be any problem. The problem is caused by violating RFC 1546.
So, as far as I can tell, everybody except perhaps Dean agrees that:
- Used incorrectly (on divergent paths), per packet load balancing can cause packet reordering.
I agree that used on divergent paths, a problem occurs with Vixie-cast. Additionally, packet reordering can happen, which can cause problems with TCP stacks that aren't able to handle insertions. This reordering problem is symptomatic of a badly implement TCP stack. Other things can cause reordering, besides PPLB.
- Used correctly (on non-diverging paths), packet reordering doesn't happen often.
- Packet reordering is bad, and should be avoided.
I'm less clear on Dean's position, but I think it's something along the lines of:
"Per packet load balancing over divergent paths is coming, by fiat from marketing departments even if engineers don't like it, and anything that doesn't play well with it needs fixing." While Dean focuses on anycast, that would presumably extend to TCP and to anything jitter sensitive, such as streaming audio or video.
No, that isn't quite it. (While it may indeed come by fiat of marketing, but I don't know that and don't assert that) But I assert that only "vixie-cast" has a problem. Streaming video and audio is thought to be reorder sensitive, but this is the same problem as with TCP reordering, and has a similar solution. If the jitter buffer can do inserts efficiently there is no problem. And, of course, the out-of-order packets must arrive before they are to be played. Some jitter buffers can do this inserting well, others can't. Indeed, TCP stacks that properly implement RFC 1546 TCP anycast recommendations will work without problem. The problem is that "vixie-cast" doesn't conform to RFC 1546. Also, the reorder problem only causes a performance problem and never causes incorrect operation. Vixie-cast causes incorrect operation. --Dean -- Av8 Internet Prepared to pay a premium for better service? www.av8.net faster, more reliable, better service 617 344 9000
On Fri, Apr 29, 2005 at 11:56:01PM -0400, Dean Anderson wrote: [ snip ]
Err. No, that would be worse. "Per prefix" load balancing is an artifact of the Cisco route cache. The route engine (ie the route table) isn't queried for every packet. Instead the route in the route cache is used. [ snip ] On Juniper, you configure it to put multiple routes in the route table. Its actaully more likely to happen on Junipers,
What? -J -- James Jun Infrastructure and Technology Services TowardEX Technologies Office +1-617-459-4051 x179 | Mobile +1-978-394-2867 james@towardex.com | www.towardex.com
On Sat, 30 Apr 2005, James wrote:
On Fri, Apr 29, 2005 at 11:56:01PM -0400, Dean Anderson wrote:
[ snip ]
Err. No, that would be worse. "Per prefix" load balancing is an artifact of the Cisco route cache. The route engine (ie the route table) isn't queried for every packet. Instead the route in the route cache is used. [ snip ] On Juniper, you configure it to put multiple routes in the route table. Its actaully more likely to happen on Junipers,
What?
This wasn't clear. See section on junipers in my message to sthaug that I just sent.
-J
-- Av8 Internet Prepared to pay a premium for better service? www.av8.net faster, more reliable, better service 617 344 9000
On Sun, Apr 24, 2005 at 04:40:50PM -0400, Robert M. Enger wrote:
More to the point however, I note that Jay is the author of RFC2100. I think he's just having a little bit of fun. My apologies for belaboring the performance issue.
I'm pleased that you noticed... but possibly less so if it means you *didn't read the clarification I posted on what I actually meant*. :-} Cheers, -- jra -- Jay R. Ashworth jra@baylink.com Designer Baylink RFC 2100 Ashworth & Associates The Things I Think '87 e24 St Petersburg FL USA http://baylink.pitas.com +1 727 647 1274 If you can read this... thank a system administrator. Or two. --me
participants (8)
-
Christopher L. Morrow -
Dean Anderson -
James -
Jay R. Ashworth -
Robert M. Enger -
Stephen J. Wilcox -
Steve Gibbard -
sthaugļ¼ nethelp.no