RE: gigabit router (was Re: Getting a "portable" /19 or /20)
Vendors have known how to solve this problem for many years. Failure to do so is a poor implementation and has nothing to do with centralized forwarding being better/worse than distributed forwarding. Prabhu ---------------------------------------------------------------------- Prabhu Kavi Phone: 1-978-264-4900 x125 Director, Adv. Prod. Planning Fax: 1-978-264-0671 Tenor Networks Email: prabhu_kavi@tenornetworks.com 100 Nagog Park WWW: www.tenornetworks.com Acton, MA 01720
-----Original Message----- From: Matt Zimmerman [mailto:mdz@csh.rit.edu] Sent: Wednesday, April 11, 2001 4:47 PM To: nanog@merit.edu Subject: Re: gigabit router (was Re: Getting a "portable" /19 or /20)
On Wed, Apr 11, 2001 at 12:26:54AM -0700, David Schwartz wrote:
Why do you think central fowarding is superior to distributed forwarding?
Because you will have consistency problem. You are nearly 100% guaranteed to have them.
Alex
Ahh, so that's what you're thinking.
If you have forwarding table F(X) at time X and forwarding table F(X+1) at time X+1, a packet that arrives between times X and X+2 can reasonably be forwarded by any of the tables. There is no special sequencing present or required between the packets that involve routing protocols and the data packets.
I think Alex was referring to internal consistency within the router (between linecards), not external consistency. For example, if linecard X believes that a packet should be forwarded to linecard Y, but linecard Y's forwarding table is older than X's, Y could misforward the packet, causing a forwarding loop or a dropped packet. Thus, it can be the case that neither the old path nor the new path is taken.
Yes, there are ways to approach this problem, but it is a problem that central-forwarding systems will not have.
We misroute packets between routers because routing table updates don't happen fast enough. It's not a problem -- IP is designed to tolerate packet losses and has never guaranteed sequencing.
It is true that IP does not make guarantees about delivery, but packet loss has a detrimental effect on performance nonetheless.
The added occasional misroutes due to inconsistency will be proportional to the ratio of the average network transport time for a routing protocol packet to the average delay in propogating forwarding table changes to a linecard. You do the math.
I think a more useful model is this:
S(X) = (% of time that a router X spends in a consistent state) * (packets/sec through router X)
For the percentage of packets which will be successfully routed. The total end-to-end loss is 1 - S(X)^N for N identical routers. N >= 20 is not uncommon these days, and packets/sec gets higher all the time.
-- - mdz
Vendors have known how to solve this problem for many years. Failure to do so is a poor implementation and has nothing to do with centralized forwarding being better/worse than distributed forwarding.
Yet another person who does not understand the KISS principle. I am sure in theory it all works great, though I am seeing way too many comments similiar to: "The connectivity issues have been resolved. This appears to be the same CEF related issues we experienced Monday evening, and we have a case open with Cisco. As we get more information from Cisco, we will be passing it along." Alex
On Wed, Apr 11, 2001 at 05:16:15PM -0400, Kavi, Prabhu wrote:
Vendors have known how to solve this problem for many years. Failure to do so is a poor implementation and has nothing to do with centralized forwarding being better/worse than distributed forwarding.
It _does_ show that distributed forwarding will be significantly more complex and more difficult to implement. I am not arguing that distributed forwarding is a bad thing; on the contrary, it has a logical and demonstrated performance benefit. However, it comes with a cost in overall system complexity. -- - mdz
In message <20010412100249.B15380@alcor.net>, Matt Zimmerman writes:
It _does_ show that distributed forwarding will be significantly more complex and more difficult to implement. I am not arguing that distributed forwarding is a bad thing; on the contrary, it has a logical and demonstrated performance benefit. However, it comes with a cost in overall system complexity.
Having implemented distributed forwarding once, and consulted on it a couple of times, the answer is that it adds complexity but not where you'd think. You usually only have to generate one forwarding table (the outbound interface through which you send the packet doesn't change depending on whether you make the decision centrally or at the inbound interfaces). You can try to reduce the size of the forwarding table by excising stuff (e.g., info on VPNs that don't run through that interface) but forwarding tables are pretty small so you don't have to do that if you don't want to. Provided you don't do something stupid and have the outbound interface re-evaluate the inbound interface's routing decision, then all you need is a way to copy the forwarding table to each interface (and if the interfaces receive the table at slightly staggered times that's OK, because they don't have to be perfectly consistent). Now I just said "all you need" -- that's a lie :-). The major complexity is actually getting the forwarding tables out to the inbound interfaces. There are at least two issues: (a) bandwidth [a small forwarding table, times 32 or 64 ports rapidly becomes big, especially for the control data path which is typically short on bandwidth] and (b) finding a way to update the forwarding table in a way that is not too disruptive to forwarding [incremental updates are often mindbreakingly hard to do, esp. if the update changes the table size, or invalidates some entries -- you can have the forwarding table doing housecleaning instead of forwarding]. We used a bank swap -- we had an active forwarding table and an inactive one. Updates were sent to the inactive bank and when you had a fully updated bank, you send a message to the forwarding processor saying "swap banks." And, of course, the software to track which forwarding engine had which bank live and the current state of each bank's updates got a little exciting. In short, generating the tables is easy, consistency isn't an issue, but getting the tables to the right place, in a timely manner, and without getting confused and trashing a table, is a challenge. Thanks! Craig
On Thu, Apr 12, 2001 at 10:16:00AM -0400, Craig Partridge wrote:
Having implemented distributed forwarding once, and consulted on it a couple of times, the answer is that it adds complexity but not where you'd think. [...] Now I just said "all you need" -- that's a lie :-). The major complexity is actually getting the forwarding tables out to the inbound interfaces. There are at least two issues: (a) bandwidth [a small forwarding table, times 32 or 64 ports rapidly becomes big, especially for the control data path which is typically short on bandwidth] and (b) finding a way to update the forwarding table in a way that is not too disruptive to forwarding [incremental updates are often mindbreakingly hard to do, esp. if the update changes the table size, or invalidates some entries -- you can have the forwarding table doing housecleaning instead of forwarding]. We used a bank swap -- we had an active forwarding table and an inactive one. Updates were sent to the inactive bank and when you had a fully updated bank, you send a message to the forwarding processor saying "swap banks." And, of course, the software to track which forwarding engine had which bank live and the current state of each bank's updates got a little exciting.
Yes, I had thought about this kind of consistency as well, within a single forwarding table. You have to provide a transactional system to update a complete table (or a self-contained subset) in an atomic operation. In practice, many of the forwarding tables for different interfaces will contain identical entries, correct? Most traffic for a given destination will leave through one interface, with the same next-hop. Is it possible (or useful) to attempt to send forwarding table entries (or chunks) across the bus to multiple interfaces at once, to approach the bandwidth issue? Thanks for sharing your experience with the list. -- - mdz
participants (4)
-
alex＠yuriev.com -
Craig Partridge -
Kavi, Prabhu -
Matt Zimmerman