In message <20010412100249.B15380@alcor.net>, Matt Zimmerman writes:
It _does_ show that distributed forwarding will be significantly more complex and more difficult to implement. I am not arguing that distributed forwarding is a bad thing; on the contrary, it has a logical and demonstrated performance benefit. However, it comes with a cost in overall system complexity.
Having implemented distributed forwarding once, and consulted on it a couple of times, the answer is that it adds complexity but not where you'd think. You usually only have to generate one forwarding table (the outbound interface through which you send the packet doesn't change depending on whether you make the decision centrally or at the inbound interfaces). You can try to reduce the size of the forwarding table by excising stuff (e.g., info on VPNs that don't run through that interface) but forwarding tables are pretty small so you don't have to do that if you don't want to. Provided you don't do something stupid and have the outbound interface re-evaluate the inbound interface's routing decision, then all you need is a way to copy the forwarding table to each interface (and if the interfaces receive the table at slightly staggered times that's OK, because they don't have to be perfectly consistent). Now I just said "all you need" -- that's a lie :-). The major complexity is actually getting the forwarding tables out to the inbound interfaces. There are at least two issues: (a) bandwidth [a small forwarding table, times 32 or 64 ports rapidly becomes big, especially for the control data path which is typically short on bandwidth] and (b) finding a way to update the forwarding table in a way that is not too disruptive to forwarding [incremental updates are often mindbreakingly hard to do, esp. if the update changes the table size, or invalidates some entries -- you can have the forwarding table doing housecleaning instead of forwarding]. We used a bank swap -- we had an active forwarding table and an inactive one. Updates were sent to the inactive bank and when you had a fully updated bank, you send a message to the forwarding processor saying "swap banks." And, of course, the software to track which forwarding engine had which bank live and the current state of each bank's updates got a little exciting. In short, generating the tables is easy, consistency isn't an issue, but getting the tables to the right place, in a timely manner, and without getting confused and trashing a table, is a challenge. Thanks! Craig