Re: from the academic side of the house
Tony Li writes:
On Apr 25, 2007, at 2:55 PM, Simon Leinen wrote:
Routing table lookups(*) are what's most relevant here, [...]
Actually, what's most relevant here is the ability to get end-hosts to run at rate. Packet forwarding at line rate has been demonstrated for quite awhile now.
That's true (although Steve's question was about the routers). The host bottleneck for raw 10Gb/s transfers used to be bus bandwidth. The 10GE adapters in most older land-speed record entries used the slower PCI-X, while this entry was done with PCI Express (x8) adapters. Another host issue would be interrupts and CPU load for checksum, but most modern 10GE (and also GigE!) adapters offload checksum segmentation and reassembly, as well as checksum computation and validation to the adapter if the OS/driver supports it. The adapters used in this record (Chelsio S310E) contain a full TOE (TCP Offload Engine) that can run the entire TCP state machine on the adapter, although I'm more sure whether they made use of that. Details on http://data-reservoir.adm.s.u-tokyo.ac.jp/lsr-200612-02/ -- Simon.
On Thu, 26 Apr 2007, Simon Leinen wrote:
Date: Thu, 26 Apr 2007 19:11:37 +0200 From: Simon Leinen <simon@limmat.switch.ch> Subject: Re: from the academic side of the house
[ ... ] Another host issue would be interrupts and CPU load for checksum, but most modern 10GE (and also GigE!) adapters offload checksum segmentation and reassembly, as well as checksum computation and validation to the adapter if the OS/driver supports it.
Correct, though sometimes the performance can vary wildly depending on driver and firmware. I had to turn off all the checksumming with certain driver versions for the Myrinet 10GE cards, later drivers performed OK. Also, having multi-cpu and multi-core machines helps enormously, binding a NIC/driver to a certain core, and having the receiving process on the neighbouring core can dramatically increase throughput. But per adapter/driver differences in which offloading actually increases performance sometimes still is a black art, necessitating painstaking testing...
The adapters used in this record (Chelsio S310E) contain a full TOE (TCP Offload Engine) that can run the entire TCP state machine on the adapter, although I'm more sure whether they made use of that.
I believe that they did not use it. In the past we also ran into problems and performance dips when using the accellerated Chelsio cards. Without TOE accelleration things worked much smoother. Kei's team did modify iperf such that they used mmap's to gain more of a zero-copy behaviour, depending on drivers this can also give much better performance, akin to say Myricom's claims for Myrinet... Kind regards, JP Velders
participants (2)
-
JP Velders -
Simon Leinen