Hi, Are there any ISPs that do, or desire, splitting traffic across different ASes for prefixes learnt via an exterior gateway protocol (say BGP)? For example, an ISP can learn two different equal cost routes to a foo.com server via two different autonomous domains. It can thus split different flows (based on src-dest IP, src-dest Port, TOS, etc) across these two paths. Do operators currently do this? Folks can send me replies offline in case this constitues a "trade secret"! Thanks, Glen
Glen Kent <glen.kent@gmail.com> writes:
For example, an ISP can learn two different equal cost routes to a foo.com server via two different autonomous domains. It can thus split different flows (based on src-dest IP, src-dest Port, TOS, etc) across these two paths.
Do operators currently do this?
Folks can send me replies offline in case this constitues a "trade secret"!
Works great with a flow-based router (or layer-three-switch-pronounced- 'crippled-router'). The downside, of course, is that you now have a flow-based router in your network, which has been shown to Not Work Well under other specified conditions (worm outbreaks come immediately to mind). ---rob
On Tue, 24 Jan 2006, Robert E.Seastrom wrote:
Glen Kent <glen.kent@gmail.com> writes:
For example, an ISP can learn two different equal cost routes to a foo.com server via two different autonomous domains. It can thus split different flows (based on src-dest IP, src-dest Port, TOS, etc) across these two paths.
Do operators currently do this?
Folks can send me replies offline in case this constitues a "trade secret"!
Works great with a flow-based router (or layer-three-switch-pronounced- 'crippled-router'). The downside, of course, is that you now have a flow-based router in your network, which has been shown to Not Work Well under other specified conditions (worm outbreaks come immediately to mind).
I could be mistaken, but this is also a feature in mbgp, effectively loadsharing across two external paths. I presume the paths would have to be completely equal all the way down to the router-id and (probably) age-of-route ...
"Christopher L. Morrow" <christopher.morrow@verizonbusiness.com> writes:
On Tue, 24 Jan 2006, Robert E.Seastrom wrote:
Glen Kent <glen.kent@gmail.com> writes:
For example, an ISP can learn two different equal cost routes to a foo.com server via two different autonomous domains. It can thus split different flows (based on src-dest IP, src-dest Port, TOS, etc) across these two paths.
Do operators currently do this?
Folks can send me replies offline in case this constitues a "trade secret"!
Works great with a flow-based router (or layer-three-switch-pronounced- 'crippled-router'). The downside, of course, is that you now have a flow-based router in your network, which has been shown to Not Work Well under other specified conditions (worm outbreaks come immediately to mind).
I could be mistaken, but this is also a feature in mbgp, effectively loadsharing across two external paths. I presume the paths would have to be completely equal all the way down to the router-id and (probably) age-of-route ...
He said "via two different autonomous domains", which I took to mean two upstreams... and my understanding is that (on ciscos anyway) you're talking per-packet, not per-flow load balancing. What happens when you intentionally bugger stuff up so that you are per-packet load balancing your outbound traffic between two diverse (ie, non-congruent-to-the-same-upstream-and-pop) paths is left as an exercise to the reader... but I don't think TCP is gonna like it. :) ---rob
On 24-Jan-2006, at 12:07, Robert E.Seastrom wrote:
He said "via two different autonomous domains", which I took to mean two upstreams... and my understanding is that (on ciscos anyway) you're talking per-packet, not per-flow load balancing.
If you can get two candidate routes for the same destination into the FIB, then you'll get per-flow load balancing as long as CEF is running, no? Joe
Joe Abley <jabley@isc.org> writes:
On 24-Jan-2006, at 12:07, Robert E.Seastrom wrote:
He said "via two different autonomous domains", which I took to mean two upstreams... and my understanding is that (on ciscos anyway) you're talking per-packet, not per-flow load balancing.
If you can get two candidate routes for the same destination into the FIB, then you'll get per-flow load balancing as long as CEF is running, no?
Yes and no. CEF is {src, dst} hash IIRC, and "per-flow" usually means {src, srcport, dst, dstport, [proto, tos]} hash in my experience. ---Rob
On 24-Jan-2006, at 13:09, Robert E.Seastrom wrote:
Joe Abley <jabley@isc.org> writes:
If you can get two candidate routes for the same destination into the FIB, then you'll get per-flow load balancing as long as CEF is running, no?
Yes and no. CEF is {src, dst} hash IIRC, and "per-flow" usually means {src, srcport, dst, dstport, [proto, tos]} hash in my experience.
Even if the hash is only calculated over source and destination IP addresses, the end effect is still that packets associated with a single flow still follow the same route where there is more than one candidate route available. Joe
Joe Abley <jabley@isc.org> writes:
On 24-Jan-2006, at 13:09, Robert E.Seastrom wrote:
Joe Abley <jabley@isc.org> writes:
If you can get two candidate routes for the same destination into the FIB, then you'll get per-flow load balancing as long as CEF is running, no?
Yes and no. CEF is {src, dst} hash IIRC, and "per-flow" usually means {src, srcport, dst, dstport, [proto, tos]} hash in my experience.
Even if the hash is only calculated over source and destination IP addresses, the end effect is still that packets associated with a single flow still follow the same route where there is more than one candidate route available.
And conversely, that different flows that ought to be load-balanced aren't. But we're splitting semantic hairs here... ;-) ---rob
Robert E Seastrom writes:
Yes and no. CEF is {src, dst} hash IIRC, and "per-flow" usually means {src, srcport, dst, dstport, [proto, tos]} hash in my experience.
Correct. The Catalyst 6500/7600 OSR with Sup2/Sup32/Sup720 can be configured to hash based on L4 ports in addition to the IP addresses (for IPv4): http://puck.nether.net/pipermail/cisco-nsp/2005-December/026952.html This is handy when you have multiple "striped" TCP connections between a single pair of hosts, and want them to be able to use multiple equal-cost paths, but still want to avoid reordering inside each connection (as you would inevitably get with per-packet load sharing). -- Simon.
On 24-Jan-2006, at 13:05, Joe Abley wrote:
On 24-Jan-2006, at 12:07, Robert E.Seastrom wrote:
He said "via two different autonomous domains", which I took to mean two upstreams... and my understanding is that (on ciscos anyway) you're talking per-packet, not per-flow load balancing.
If you can get two candidate routes for the same destination into the FIB, then you'll get per-flow load balancing as long as CEF is running, no?
(Substitute "other, similar, hash-based ECMP route selection schemes" for CEF, for routers not made by cisco. Apologies for my transient lapse into assumptions about vendor choice. And also for replying to my own mail. If anybody needs me, I'll be standing in the corner looking embarrassed.)
On Tue, 24 Jan 2006, Joe Abley wrote:
On 24-Jan-2006, at 12:07, Robert E.Seastrom wrote:
He said "via two different autonomous domains", which I took to mean two upstreams... and my understanding is that (on ciscos anyway) you're talking per-packet, not per-flow load balancing.
If you can get two candidate routes for the same destination into the FIB, then you'll get per-flow load balancing as long as CEF is running, no?
that was my thought... and yes, it could get ugly for tcp services. Why would you knowningly induce this complication?
On Tue, 24 Jan 2006, Christopher L. Morrow wrote:
that was my thought... and yes, it could get ugly for tcp services. Why would you knowningly induce this complication?
When you want single flows to go faster than a single member link? (not that I am saying this is a good idea) Actually, TCP handles out of order packets rather well as long as the reordering isn't too severe. You see a bunch of SACKs flying around, but as long as it doesn't get too out of hand it doesn't affect throughput. It is the non-TCP protocols that often suffer. Many of them implement sequence numbers and simply drop out of order packets. From my own experience, RealPlayer UDP streams and PPTP are two examples that fail (or at least feel like 50% packetloss) under heavy reodering, where TCP continues to work reasonably well. Years ago, I had ISDN and IDSL between home and the ISP I worked at, and out of curiosity I experimented with per-packet load balancing across these links. Reordering was rather severe, as these links had slow uneven speeds, and uneven latencies. TCP transfers got about 192kbit (75% total link capacity, 1.5 times single link capacity), but things like RealPlayer and PPTP VPNs were downright unusable.
On 24-Jan-2006, at 14:17, Matt Buford wrote:
Actually, TCP handles out of order packets rather well as long as the reordering isn't too severe.
There's packet reordering, and there's oscillating RTT on segments that travel by different paths. I suspect the veracity of your statement depends (a) on whose implementation of TCP you're using, and (b) what exactly you mean by "rather well". Joe
In message <0d2801c62152$4d32fd60$fe00000a@speedy>, "Matt Buford" writes:
Actually, TCP handles out of order packets rather well as long as the reordering isn't too severe. You see a bunch of SACKs flying around, but as long as it doesn't get too out of hand it doesn't affect throughput.
Actually, it isn't that great. See draft-ietf-tcpm-tcp-dcr-06 and its references. --Steven M. Bellovin, http://www.cs.columbia.edu/~smb
participants (7)
-
Christopher L. Morrow -
Glen Kent -
Joe Abley -
Matt Buford -
Robert E.Seastrom -
Simon Leinen -
Steven M. Bellovin