Re: Slashdot: Providers Ignoring DNS TTL?
First of all, let's ditch the term "PPLB." The usual alternative to per packet load balancing (what's been being talked about here) is per prefix load balancing, which would also be "PPLB." The abbreviation is therefore more confusing than anything else.
Err. No, that would be worse. "Per prefix" load balancing is an artifact of the Cisco route cache. The route engine (ie the route table) isn't queried for every packet. Instead the route in the route cache is used. One doesn't configure "per prefix" load balancing. One configures load balancing, which adds multiple routes into the route table.
Modern Cisco routers do not use a "route cache", they use a fully populated forwarding table. And load balancing is automatic if you have several equal cost routes.
The route cache then causes only one of these routes to be used. On cisco, to enable PPLB, you turn off the route cache.
Many modern Cisco routers can perform per-packet load balancing without doing process switching (but this needs to be explicitly configured).
On Juniper, you configure it to put multiple routes in the route table. Its actaully more likely to happen on Junipers, because unless you configure additonal policies, you get load balancing on divergent links as well as non-divergent links. On
Modern Juniper routers cannot do per-packet load balancing *at all*. It is correct that the configuration statement says "per-packet", however it is really per-flow (and this is well documented). See for instance the description of Internet Processor II ASIC load balancing at http://www.juniper.net/techpubs/software/junos/junos70/swconfig70-policy/htm... I'm afraid your statements show a certain lack of knowledge about modern router architectures. Steinar Haug, Nethelp consulting, sthaug@nethelp.no
On Sat, 30 Apr 2005 sthaug@nethelp.no wrote:
First of all, let's ditch the term "PPLB." The usual alternative to per packet load balancing (what's been being talked about here) is per prefix load balancing, which would also be "PPLB." The abbreviation is therefore more confusing than anything else.
Err. No, that would be worse. "Per prefix" load balancing is an artifact of the Cisco route cache. The route engine (ie the route table) isn't queried for every packet. Instead the route in the route cache is used. One doesn't configure "per prefix" load balancing. One configures load balancing, which adds multiple routes into the route table.
Modern Cisco routers do not use a "route cache",
You'll need to define what you mean by "modern" with respect to cisco. This statement seems to be incorrect.
they use a fully populated forwarding table. And load balancing is automatic if you have several equal cost routes.
This sounds very much like the Juniper description for the Internet Processor ASIC behavior. I'd say that's worse.
The route cache then causes only one of these routes to be used. On cisco, to enable PPLB, you turn off the route cache.
Many modern Cisco routers can perform per-packet load balancing without doing process switching (but this needs to be explicitly configured).
Well, 7500 and 7200 have interface processors that can route packets using the route cache without interrupting the main processor. So, if you don't consider 7500's and 7200s to be "modern", this feature above doesn't seem like a big deal: They could do that before. It was called CEF and DCEF.
On Juniper, you configure it to put multiple routes in the route table. Its actaully more likely to happen on Junipers, because unless you configure additonal policies, you get load balancing on divergent links as well as non-divergent links. On
Modern Juniper routers cannot do per-packet load balancing *at all*. It is correct that the configuration statement says "per-packet", however it is really per-flow (and this is well documented). See for instance the description of Internet Processor II ASIC load balancing at
http://www.juniper.net/techpubs/software/junos/junos70/swconfig70-policy/htm...
I don't have Junipers, so I'm just going by what the manual says. And your link says: "On routing platforms with an Internet Processor ASIC, when per-packet load balancing is configured, traffic between routers with multiple paths is spread using the hash algorithm across the available interfaces. The forwarding table balances the traffic headed to a destination, transmitting it in round-robin fashion among the multiple next hops (up to a maximum of eight equal-cost load-balanced paths). The traffic is ^^^^^^^^^^^^^^^^ load-balanced on a per-packet basis." ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ "On routing platforms with the Internet Processor II ASIC, when per-packet load balancing is configured, traffic between routers with multiple paths is divided into individual traffic flows (up to a maximum of 16 equal-cost load-balanced paths). Packets for each individual flow are kept on a ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ single interface." ^^^^^^^^^^^^^^^^ I would gues that since both processers are described, that they are both still supported, and that probably means that both are widely used. But, I should qualify that doing PPLB on diverse paths is more likely to happen on the Internet Processor ASIC. It would seem like the Internet Processor II ASIC has an architecture more like the cisco 7500s, and only allows per flow load balancing. So, I guess it depends on how many people might still be using the Internet Processor ASIC platforms. And of course, whether this behavior might be disabled in some future release for the 'II ASIC or if some new platform might have PPLB on diverse paths.
I'm afraid your statements show a certain lack of knowledge about modern router architectures.
I'm afraid your statements show a certain lack of knowledge about whats being used in datacenters to route packets. And perhaps some arrogance about whats "modern". I'd still call cisco 7500 and 7200 series routers "modern", and they have route caches. I don't know that much about GSRs, but they didn't seem to get much traction. I can't say whether they have a route cache or not. And the multi-rack monster router that cisco just announced a while back doesn't seem to be too popular either. I don't know its architecture, either. As I look around datacenters, the 7500 and 7200's and to some lesser extent Junipers are the workhorses doing most of the routing. And that basic technology will stay around in the enterprise for many more years to come. But again, note that RFC 1546 give a rule about internetwork architecture: An internetwork has no obligation to deliver two successive packets sent to the same anycast address to the same host. Whether it used to be impossible to utilize this rule, and whether anyone actually presently uses this rule is irrelevant to the question of what rules one needs to follow when building anycast systems. RFC 1546 gives some rules to follow, and they are violated at the peril of the internetwork. And "vixie-cast" violates this rule. It imposes the new rule that "an internetwork must deliver to successive packets sent to the same anycast address to the same host." And no one has thought much about the implications of that rule. Assurances that no one can do PPLB on diverse paths offer no defense to having violated the design principles given for anycast in RFC 1546. It is also objectionable to calling something "TCP anycast" that isn't TCP anycast according to RFC 1546. --Dean -- Av8 Internet Prepared to pay a premium for better service? www.av8.net faster, more reliable, better service 617 344 9000
On Sun, May 01, 2005 at 12:44:09AM -0400, Dean Anderson wrote:
Modern Cisco routers do not use a "route cache",
You'll need to define what you mean by "modern" with respect to cisco. This statement seems to be incorrect.
For someone with so little clue, you sure do manage to talk a lot. :P I'm not under any delusion that anything I say here will help you understand anything, but if it helps some other soul than it may be worth it. Historically, routers were entirely CPU driven beasts. The most basic and fully functional type of routing lookup (and also the slowest) was known as process switching, where the destination address was fully evaluated against the "master routing table" (RIB) on the CPU. The data structure used to hold this RIB, one of the easiest algorithms used to implement "longest prefix match" lookups, is known as a PATRICIA (Practical Algorithm To Retrieve Information Coded in Alphanumeric) tree. Unfortunately, these full RIB lookups were (and still are) a relatively slow process. In Cisco speak, a "route cache" is any of a variety of mechanisms that are used to cache routing lookups, using a mechanism which is "faster" than a full RIB lookup. Some of these mechanisms include the "fast cache" (where the most popular destinations are stored in a smaller cache after the first packet is looked up via process switching), "flow cache" (where individual layer 4 flows are stored), and "cef" or Cisco Express Forwarding. On modern Cisco routers, CEF is the only thing you will find used to actually do routing lookups. However, CEF is more of a brand name than a detailed description of how the routing lookup occurs, and the implementation varies greatly from platform to platform. The only thing that CEF really describes across all varieties (other than the fact that you will soon be experiencing its other meaning, the Customer Enragement Feature :P) is that the route cache will use a "pre-populated" FIB (forwarding information base). A FIB is a data structure which exists solely to do routing lookups, and is created from a normal RIB. A classic software implementation of a FIB uses a multi-bit trie (mtrie) to fully map the destination next-hops of the entire address space. This consumes a little bit of memory (several megabytes), but gives you a data structure which is fully "pre-populated" and delivers consistant results. Tihs means that there is no mode where the first lookup can take longer than the rest, the memory usage does not increase as you look up more destinations, and the number of memory accesses is roughly the same for all addresses (vs a patricia tree where the number of access can vary greatly depending on the tree depth). All of this is historical of course, as "modern" routers do all of their packet lookups in hardware using designs which look nothing like any of this. While Cisco does call everything "ip route-cache", in modern routers the commands are just there for historical compatability.
they use a fully populated forwarding table. And load balancing is automatic if you have several equal cost routes.
This sounds very much like the Juniper description for the Internet Processor ASIC behavior. I'd say that's worse.
Load balancing has nothing to do with CEF or a FIB persay, the FIB is just a good spot to slap multiple next-hops. Juniper's implementation uses a "pre-populated FIB", the same as everyone else, they just do it using a tree primitive on an ASIC and controlled by a CPU based switch board.
The route cache then causes only one of these routes to be used. On cisco, to enable PPLB, you turn off the route cache.
Many modern Cisco routers can perform per-packet load balancing without doing process switching (but this needs to be explicitly configured).
Well, 7500 and 7200 have interface processors that can route packets using the route cache without interrupting the main processor. So, if you don't consider 7500's and 7200s to be "modern", this feature above doesn't seem like a big deal: They could do that before. It was called CEF and DCEF.
7200 most certainly does not have interface processors. 7500 does have processors on the VIPs that do forwarding lookups in a distributed fashion, but the same procedure for software forwarding apply, there just happen to be a few more CPUs floating around. DCEF is just CEF plus copying the FIB structure to the VIPs. And no I don't think any sane person would consider 7500s or 7200s to be "modern", even though you can still make use of them.
On Juniper, you configure it to put multiple routes in the route table. Its actaully more likely to happen on Junipers, because unless you configure additonal policies, you get load balancing on divergent links as well as non-divergent links. On
Modern Juniper routers cannot do per-packet load balancing *at all*. It is correct that the configuration statement says "per-packet", however it is really per-flow (and this is well documented). See for instance the description of Internet Processor II ASIC load balancing at
http://www.juniper.net/techpubs/software/junos/junos70/swconfig70-policy/htm...
I don't have Junipers, so I'm just going by what the manual says. And your link says: ... I would gues that since both processers are described, that they are both still supported, and that probably means that both are widely used.
The original poster is entirely correct. The original Internet Processor is still supported, but it is about as far from "widely used" as you can get. If you're still using it you have bigger problems. The IP2 is only capable of doing per-flow load balancing, which is probably a good thing.
But, I should qualify that doing PPLB on diverse paths is more likely to happen on the Internet Processor ASIC. It would seem like the Internet Processor II ASIC has an architecture more like the cisco 7500s, and only allows per flow load balancing.
The IP vs IP2 has no architecture that is or isn't more like the 7500s. The IP2 is just a newer version of the ASIC on the switch fabric cards. Classic Juniper architecture (everything pre M320/T series) actually uses all centralized routing (and other forwarding/filtering) lookups.
I'm afraid your statements show a certain lack of knowledge about modern router architectures.
I'm afraid your statements show a certain lack of knowledge about whats being used in datacenters to route packets. And perhaps some arrogance about whats "modern". I'd still call cisco 7500 and 7200 series routers "modern", and they have route caches. I don't know that much about GSRs, but they didn't seem to get much traction. I can't say whether they have a route cache or not. And the multi-rack monster router that cisco just announced a while back doesn't seem to be too popular either. I don't know its architecture, either. As I look around datacenters, the 7500 and 7200's and to some lesser extent Junipers are the workhorses doing most of the routing. And that basic technology will stay around in the enterprise for many more years to come.
What ever happened to the days when people who didn't understand something would sit down, shut up, and listen to the people who did? You my friend don't even have a basic grasp on the old hardware (which is most assuredly not modern), let alone a firm grasp, let alone any idea how modern (you know, that stuff made in the last 5+ years) works. Instead of spouting off about things which you don't understand, you might do well to listen and learn. Well, that is if you actually want to learn of course. Maybe you just want to talk. :)
And "vixie-cast" violates this rule. It imposes the new rule that "an internetwork must deliver to successive packets sent to the same anycast address to the same host." And no one has thought much about the implications of that rule.
Assurances that no one can do PPLB on diverse paths offer no defense to having violated the design principles given for anycast in RFC 1546.
It is also objectionable to calling something "TCP anycast" that isn't TCP anycast according to RFC 1546.
I've successfully managed to tune out this thread until now, so I don't know what babble I've missed so far, but let me sum it up for you in no-nonsense terms: Nothing says that you can't have out of order packets on the Internet until you are blue in the face. However, it tends to do very nasty things to the TCP algorithm, which makes it perform poorly. If you are fine with poorly performing TCP then you go right ahead and re-order your packets, but I know that I'm not fine with this, nor are my customers or the vast majority of other Internet users. Therefore, if vendors want to design a product that end-run the problem by maintaining packet ordering when they load balance, good for them. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
7200 most certainly does not have interface processors. 7500 does have processors on the VIPs that do forwarding lookups in a distributed fashion, but the same procedure for software forwarding apply, there just happen to be a few more CPUs floating around.
Also, it may not be clear from a manufacturer's literature just how many CPUs are in a router. FPGAs and other ASICs can include one or more processor cores in the custom logic. A modern router is actually a box containing a network of specialized computers interconnected with various high-speed network fabrics to do jobs like move packets and internal coordination of activities. Routing engineers tend to have a higher level abstract concept of what the router does, based on older (less modern) implementations which implemented new concepts like CEF, etc.
The original poster is entirely correct. The original Internet Processor
is still supported, but it is about as far from "widely used" as you can
get. If you're still using it you have bigger problems. The IP2 is only capable of doing per-flow load balancing, which is probably a good thing.
And, if the few older models in existence are actually in a PPLB configuration, it is highly likely that they will only be dealing with load balancing over n identical circuits with between point A and point B in which all the A ends are on the same router and all the B ends are on the same router. This happens to be the scenario in which PPLB doesn't trigger major problems.
Nothing says that you can't have out of order packets on the Internet until you are blue in the face. However, it tends to do very nasty things to the TCP algorithm, which makes it perform poorly.
Which is precisely the reason why PPLB is not an issue with anycasting. People who tried PPLB got burnt and got rid of the config that was causing the pain. Therefore, per packet load balancing is extremely rare in the Internet today, and getting rarer. Now can we leave the discussion of theoretical problems behind? If anything, this should be taken up at the IETF so that the downsides of per packet load balancing are publicized and PPLB can be deprecated. --Michael Dillon
On Tue, 3 May 2005 Michael.Dillon@radianz.com wrote:
7200 most certainly does not have interface processors. 7500 does have processors on the VIPs that do forwarding lookups in a distributed fashion, but the same procedure for software forwarding apply, there just happen to be a few more CPUs floating around.
7200 _is_ an interface processor. It takes PA cards that usually go into a VIP. -- Av8 Internet Prepared to pay a premium for better service? www.av8.net faster, more reliable, better service 617 344 9000
Err. No, that would be worse. "Per prefix" load balancing is an artifact of the Cisco route cache. The route engine (ie the route table) isn't queried for every packet. Instead the route in the route cache is used. One doesn't configure "per prefix" load balancing. One configures load balancing, which adds multiple routes into the route table.
Modern Cisco routers do not use a "route cache",
You'll need to define what you mean by "modern" with respect to cisco. This statement seems to be incorrect.
the statement is largely correct -- at least from an operational standpoint. it is true that IOS still has 'route-cache'-based forwarding and 'flow'-based forwarding schemes (ip route-cache, ip-route-cache flow), BUT given we're talking about internet routing here, you would defintely want to be using CEF which isn't a cache demand-populated method. the distinction between demand-populated forwarding (FIB) versus prepopulated forwarding tables is relatively straight-forward, as are the reasons why it is a "good thing"<tm>. of course, hindsight is a wonderful thing.
they use a fully populated forwarding table. And load balancing is automatic if you have several equal cost routes.
This sounds very much like the Juniper description for the Internet Processor ASIC behavior. I'd say that's worse.
umm, no, i'd say it "isn't worse". i can't speak for how J does it (or what methods they may use for loadbalancing across distributed forwarding hardware and/or multiple switch-fabric(s)), but in the case of C, the default (per-prefix) loadbalancing provides deterministic loadbalancing which won't reorder packets within the same src/dst tuple (tuple could be L3 or L3+L4-based).
Many modern Cisco routers can perform per-packet load balancing without doing process switching (but this needs to be explicitly configured).
Well, 7500 and 7200 have interface processors that can route packets using the route cache without interrupting the main processor. So, if you don't consider 7500's and 7200s to be "modern", this feature above doesn't seem like a big deal: They could do that before. It was called CEF and DCEF.
umm, what you're saying is largely orthogonal to what Steinar is saying. distributed versus centralized forwarding is a different topic of discussion. you seem familiar with the methods commonly used to gain per-packet loadbalancing from about 6 years ago. CEF can provide the same functionality but without 'process-switching'.
I'm afraid your statements show a certain lack of knowledge about whats being used in datacenters to route packets. And perhaps some arrogance about whats "modern". I'd still call cisco 7500 and 7200 series routers "modern", and they have route caches.
"best practice" would be to use CEF for pre-populated Forwarding Tables rather than 'fast-switching' methods which use demand-based population methods. cheers, lincoln.
The questions of what various routers do now or did in the past is irrelevant. So, to wrap it up: RFC 1546 give this rule about internetwork architecture on page 5: An internetwork has no obligation to deliver two successive packets sent to the same anycast address to the same host. Whether it used to be impossible to utilize this rule, and whether anyone actually presently uses this rule is irrelevant to the question of what rules one needs to follow when building anycast systems. RFC 1546 gives some rules to follow, and they are violated at the peril of the internetwork. TCP "vixie-cast" violates this rule. It imposes the new rule that "an internetwork MUST deliver to successive packets sent to the SAME anycast address to the SAME host." And no one has thought much about the implications of that rule, (other than the original architects of RFC 1546). Sure, it sort of happens most of the time with current routers and current configurations, but load balancing over diverse paths isn't limited to being slow and per-flow. There are no IETF rules that require that behavior. Implementors of networks and routers are free to use the RFC 1546 design rule. Assurances that typically, it happens that no one can "deliver two successive packets sent to same anyast IP address to different hosts" is no defense for TCP "vixie-cast" having violated the design principles given for anycast in RFC 1546. It is also objectionable to calling something "TCP anycast" that isn't TCP anycast according to RFC 1546. --Dean -- Av8 Internet Prepared to pay a premium for better service? www.av8.net faster, more reliable, better service 617 344 9000
participants (5)
-
Dean Anderson -
Lincoln Dale -
Michael.Dillon＠radianz.com -
Richard A Steenbergen -
sthaug＠nethelp.no