On Sun, 24 Apr 2005, Steve Gibbard wrote:
On Sun, 24 Apr 2005, Robert M. Enger wrote:
Steinar:
There is a large body of work from competent and well known researchers that assert the claim. I certainly lack standing to question their results.
Empirically, download speeds to home are nearly cut in half (18Mbps) from sources that are subjected to packet reordering along the path.
I'm trying to sort out the various claims here, since I think right now this is a case of people talking past each other, and arguing completely different points.
First of all, let's ditch the term "PPLB." The usual alternative to per packet load balancing (what's been being talked about here) is per prefix load balancing, which would also be "PPLB." The abbreviation is therefore more confusing than anything else.
Err. No, that would be worse. "Per prefix" load balancing is an artifact of the Cisco route cache. The route engine (ie the route table) isn't queried for every packet. Instead the route in the route cache is used. One doesn't configure "per prefix" load balancing. One configures load balancing, which adds multiple routes into the route table. The route cache then causes only one of these routes to be used. On cisco, to enable PPLB, you turn off the route cache. On Juniper, you configure it to put multiple routes in the route table. Its actaully more likely to happen on Junipers, because unless you configure additonal policies, you get load balancing on divergent links as well as non-divergent links. On Cisco, the route cache is controlled on a per-interface basis.
Now, onto the argument that's going on here.
Dean says "per packet load balancing is coming," and then goes on to assume it's going to be used in such a way that it will cause packets to route through widely divergent paths.
Almost a correct statement of my position. But if its used in any case--anywhere--even on internal links, in any condition other than between exactly two routers, it will cause a problem with Vixie's TCP anycast. We aren't seeing a lot of problems yet because few people are using PPLB, and fewer are using TCP DNS, and so few are using Vixie's TCP anycast. If we change any our terms in this discussion, it should be to change our use of "TCP anycast". The behavior described by Vixie for DNS TCP "anycast" is unapproved** and non-standard. It does not conform to RFC 1546 TCP Anycast. RFC 1546 outlines changes to the TCP stack to support TCP anycast. To avoid confusion, I propose we call it "vixie-cast". And I also want to emphasize that RFC 1546 TCP anycast doesn't have any problems with PPLB. And of course UDP anycast also has no problem. It is only "Vixie-cast TCP" that has a problem. ** One of the criticisms of DNS Root Anycast is that Vixie recommended deployment of anycast to root server operators without first discussing this on the DNSOP WG. Dr. Bernstein first brought it to the WG's attention in 2002. And one of the recent discoveries is that a good number of DNS root server operators are using it, and had assumed the Vixie was the IETF liason communicating with the IETF. The IETF has never approved of this, and in fact, Vixie's version of TCP anycast is not what is described in RFC 1546. Vixie's version really shouldn't be called anycast, since it creates confusion with RFC 1546 recommendations. Indeed, if RFC 1546 recommendations for TCP anycast were widely implemented in TCP stacks, then there wouldn't be any problem. The problem is caused by violating RFC 1546.
So, as far as I can tell, everybody except perhaps Dean agrees that:
- Used incorrectly (on divergent paths), per packet load balancing can cause packet reordering.
I agree that used on divergent paths, a problem occurs with Vixie-cast. Additionally, packet reordering can happen, which can cause problems with TCP stacks that aren't able to handle insertions. This reordering problem is symptomatic of a badly implement TCP stack. Other things can cause reordering, besides PPLB.
- Used correctly (on non-diverging paths), packet reordering doesn't happen often.
- Packet reordering is bad, and should be avoided.
I'm less clear on Dean's position, but I think it's something along the lines of:
"Per packet load balancing over divergent paths is coming, by fiat from marketing departments even if engineers don't like it, and anything that doesn't play well with it needs fixing." While Dean focuses on anycast, that would presumably extend to TCP and to anything jitter sensitive, such as streaming audio or video.
No, that isn't quite it. (While it may indeed come by fiat of marketing, but I don't know that and don't assert that) But I assert that only "vixie-cast" has a problem. Streaming video and audio is thought to be reorder sensitive, but this is the same problem as with TCP reordering, and has a similar solution. If the jitter buffer can do inserts efficiently there is no problem. And, of course, the out-of-order packets must arrive before they are to be played. Some jitter buffers can do this inserting well, others can't. Indeed, TCP stacks that properly implement RFC 1546 TCP anycast recommendations will work without problem. The problem is that "vixie-cast" doesn't conform to RFC 1546. Also, the reorder problem only causes a performance problem and never causes incorrect operation. Vixie-cast causes incorrect operation. --Dean -- Av8 Internet Prepared to pay a premium for better service? www.av8.net faster, more reliable, better service 617 344 9000