Craig did a great job describing the problem and one valid approach to solving it. There are other valid approaches as well. Creating modern packet forwarding devices is based upon learning what did not work well in previous incarnations (such as how to synchronize these forwarding tables) and elmininating the problems (Too little bandwidth in the control plane? Add bandwidth) in the next version. Many vendors learned about the synchronized forwarding table problem years ago, either with experience on their own routers, or by getting talent from places (like BBN) that previously dealt with this. And any recent system architecture created by competent engineers should not have this problem. Prabhu

-----Original Message----- From: Craig Partridge [mailto:craig@aland.bbn.com] Sent: Thursday, April 12, 2001 10:16 AM To: Matt Zimmerman Cc: nanog@merit.edu Subject: Re: gigabit router (was Re: Getting a "portable" /19 or /20)

In message <20010412100249.B15380@alcor.net>, Matt Zimmerman writes:

...

It _does_ show that distributed forwarding will be significantly more complex and more difficult to implement. I am not arguing that distributed forwarding is a bad thing; on the contrary, it has a logical and demonstrated performance benefit. However, it comes with a cost in overall system complexity.

Having implemented distributed forwarding once, and consulted on it a couple of times, the answer is that it adds complexity but not where you'd think.

You usually only have to generate one forwarding table (the outbound interface through which you send the packet doesn't change depending on whether you make the decision centrally or at the inbound interfaces). You can try to reduce the size of the forwarding table by excising stuff (e.g., info on VPNs that don't run through that interface) but forwarding tables are pretty small so you don't have to do that if you don't want to.

Provided you don't do something stupid and have the outbound interface re-evaluate the inbound interface's routing decision, then all you need is a way to copy the forwarding table to each interface (and if the interfaces receive the table at slightly staggered times that's OK, because they don't have to be perfectly consistent).

Now I just said "all you need" -- that's a lie :-). The major complexity is actually getting the forwarding tables out to the inbound interfaces. There are at least two issues: (a) bandwidth [a small forwarding table, times 32 or 64 ports rapidly becomes big, especially for the control data path which is typically short on bandwidth] and (b) finding a way to update the forwarding table in a way that is not too disruptive to forwarding [incremental updates are often mindbreakingly hard to do, esp. if the update changes the table size, or invalidates some entries -- you can have the forwarding table doing housecleaning instead of forwarding]. We used a bank swap -- we had an active forwarding table and an inactive one. Updates were sent to the inactive bank and when you had a fully updated bank, you send a message to the forwarding processor saying "swap banks." And, of course, the software to track which forwarding engine had which bank live and the current state of each bank's updates got a little exciting.

In short, generating the tables is easy, consistency isn't an issue, but getting the tables to the right place, in a timely manner, and without getting confused and trashing a table, is a challenge.

Thanks!

Craig