the recent facebook engineering post on scaling memcached to 200-300K UDP requests/sec/node may be germaine here (in particular, patches to make irq handling more intelligent become very useful at the traffic levels being discussed). http://www.facebook.com/note.php?note_id=39391378919&id=9445547199&index=0 /sf On Wed, Dec 17, 2008 at 8:30 AM, Jim Shankland <nanog@shankland.org> wrote:
Chris wrote:
Hi All, Sorry if this is a repeat topic. I've done a fair bit of trawling but can't find anything concrete to base decisions on.
I'm hoping someone can offer some advice on suitable hardware and kernel tweaks for using Linux as a router running bgpd via Quagga. We do this at the moment and our box manages under the 100Mbps level very effectively. Over the next year however we expect to push about 250Mbps outbound traffic with very little inbound (50Mbps simultaneously) and I'm seeing differing suggestions of what to do in order to move up to the 1Gbps level.
As somebody else said, it's more pps than bits you need to worry about. The Intel NICs can do a full gigabit without any difficulty, if packet size is large enough. But they buckle somewhere around 300Kpps. 300K 100-byte packets is only 240 Mb/s. On the other hand, you mentioned your traffic is mostly outbound, which makes me think you might be a content provider. In that case, you'll know what your average packet size is -- and it should be a lot bigger than 100 bytes. For that type of traffic, using a Linux router up to, say, 1.5-2 Gb/s is pretty trivial. You can do more than that, too, but have to start getting a lot more careful about hardware selection, tuning, etc.
The other issue is the number of concurrent flows. The actual route table size is unimportant -- it's the size of the route cache that matters. Unfortunately, I have no figures here. But I did once convert a router from limited routes (quagga, 10K routes) to full routes (I think about 200K routes at the time), with absolutely no measurable impact. There were only a few thousand concurrent flows, and that number did not change -- and that's the one that might have made a difference.
I hope this is helpful.
Jim
-- darkuncle@{gmail.com,darkuncle.net} || 0x5537F527 http://darkuncle.net/pubkey.asc for public key