Bonding usually involves some sort of flow-based hashing mechanism for balancing across the links. The logic behind it being that reordering TCP packets (among other things) negates the benefit of the aggregation in CPU costs and retransmissions. So it's important to keep one flow (really one direction of one flow) on one link to prevent out of order delivery. So if you want to utilize that bandwidth, you need a lot of flows. I remember numbers being thrown around that you need at least 1000 flows to balance two paths. I don't know how many you'd need for that to scale to 16 links. Obviously if this is SAN traffic, the usage patterns are going to be completely different from desktop, so ymmv. Also, this information may be out of date and/or irrelevant now, but it's something I've run into before with link aggregation. On Thu, May 19, 2011 at 1:37 PM, Jamie Bowden <jamie@photon.com> wrote:

I'm looking for people's experiences with Voltaire switches in general and the Vantage 6048 in particular.

We'd like to use a central switch and use 16 10g ports trunked via LACP to two other switches and a SAN (to clarify, the central switch would have three data channels, each one consisting of 16 trunked 10g ports, the two downstream switches would have 10gb clients hanging off of them and I'm not sure how the SAN is supposed to use 16 10g ports; they haven't felt the need to give me specs on it, but for now I'm assuming it either has a built in switch fabric that supports LACP or will have another switch dedicated to it can handle different L1 media). Assuming we can bond that many channels (which I'm not sure about, but none of the docs I've read on the IEEE protocol involved indicate a limit on the number ports that can be aggregated, and the manufacturer docs don't mention it either), how realistic is the expectation of getting a full 160gb throughput? Are the switches in question up to the task?

This is a research project we're building for a customer, so I'm trying to manage expectations, but this isn't the sort of thing I've personally ever built before and I'm hoping someone here has done something close enough and is willing to share experiences.

Thanks,

Jamie

-- ^[:wq^M