On Thu, 26 Apr 2007, Simon Leinen wrote:
Date: Thu, 26 Apr 2007 19:11:37 +0200 From: Simon Leinen <simon@limmat.switch.ch> Subject: Re: from the academic side of the house
[ ... ] Another host issue would be interrupts and CPU load for checksum, but most modern 10GE (and also GigE!) adapters offload checksum segmentation and reassembly, as well as checksum computation and validation to the adapter if the OS/driver supports it.
Correct, though sometimes the performance can vary wildly depending on driver and firmware. I had to turn off all the checksumming with certain driver versions for the Myrinet 10GE cards, later drivers performed OK. Also, having multi-cpu and multi-core machines helps enormously, binding a NIC/driver to a certain core, and having the receiving process on the neighbouring core can dramatically increase throughput. But per adapter/driver differences in which offloading actually increases performance sometimes still is a black art, necessitating painstaking testing...
The adapters used in this record (Chelsio S310E) contain a full TOE (TCP Offload Engine) that can run the entire TCP state machine on the adapter, although I'm more sure whether they made use of that.
I believe that they did not use it. In the past we also ran into problems and performance dips when using the accellerated Chelsio cards. Without TOE accelleration things worked much smoother. Kei's team did modify iperf such that they used mmap's to gain more of a zero-copy behaviour, depending on drivers this can also give much better performance, akin to say Myricom's claims for Myrinet... Kind regards, JP Velders