On 1/26/15 11:33 PM, Pavel Odintsov wrote:
Hello!
Looks like somebody want to build Linux soft router!) Nice idea for routing 10-30 GBps. I route about 5+ Gbps in Xeon E5-2620v2 with 4 10GE cards Intel 82599 and Debian Wheezy 3.2 (but it's really terrible kernel, everyone should use modern kernels since 3.16 because "buggy linux route cache"). My current processor load on server is about: 15%, thus I can route about 15 GE on my Linux server.
I looked into the promise and limits of this approach pretty intensively a few years back before abandoning the effort abruptly due to other constraints. Underscoring what others have said: it's all about pps, not aggregate throughput. Modern NICs can inject packets at line rate into the kernel, and distribute them across per-processor queues, etc. Payloads end up getting DMA-ed from NIC to RAM to NIC. There's really no reason you shouldn't be able to push 80 Gb/s of traffic, or more, through these boxes. As for routing protocol performance (BGP convergence time, ability to handle multiple full tables, etc.): that's just CPU and RAM. The part that's hard (as in "can't be fixed without rethinking this approach") is the per-packet routing overhead: the cost of reading the packet header, looking up the destination in the routing table, decrementing the TTL, and enqueueing the packet on the correct outbound interface. At the time, I was able to convince myself that being able to do this in 4 us, average, in the Linux kernel, was within reach. That's not really very much time: you start asking things like "will the entire routing table fit into the L2 cache?" 4 us to "think about" each packet comes out to 250Kpps per processor; with 24 processors, it's 6Mpps (assuming zero concurrency/locking overhead, which might be a little bit of an ... assumption). With 1500-byte packets, 6Mpps is 72 Gb/s of throughput -- not too shabby. But with 40-byte packets, it's less than 2 Gb/s. Which means that your Xeon ES-2620v2 will not cope well with a DDoS of 40-byte packets. That's not necessarily a reason not to use this approach, depending on your situation; but it's something to be aware of. I ended up convincing myself that OpenFlow was the right general idea: marry fast, dumb, and cheap switching hardware with fast, smart, and cheap generic CPU for the complicated stuff. My expertise, such as it ever was, is a bit stale at this point, and my figures might be a little off. But I think the general principle applies: think about the minimum number of x86 instructions, and the minimum number of main memory accesses, to inspect a packet header, do a routing table lookup, and enqueue the packet on an outbound interface. I can't see that ever getting reduced to the point where a generic server can handle 40-byte packets at line rate (for that matter, "line rate" is increasing a lot faster than "speed of generic server" these days). Jim