I'm looking for input on the best practices for sending large files over a long fat pipe between facilities (gigabit private circuit, ~20ms RTT).
providing you have RFC1323 type extensions enabled on a semi-decent OS, a 4MB TCP window should be more than sufficient to fill a GbE pipe over 30msec. with a modified TCP stack, that uses TCP window sizes up to 32MB, i've worked with numerous customers to achieve wire-rate GbE async replication for storage-arrays with FCIP. the modifications to TCP were mostly to adjust how it reacts to packet loss, e.g. don't "halve the window". the intent of those modifications is that it doesn't use the "greater internet" but is more suited for private connections within an enterprise customer environment. that is used in production today on many Cisco MDS 9xxx FC switch environments.
I'd like to avoid modifying TCP windows and options on end hosts where possible (I have a lot of them). I've seen products that work as "transfer stations" using "reliable UDP" to get around the windowing problem.
given you don't want to modify all your hosts, you could 'proxy' said TCP connections via 'socat' or 'netcat++'. cheers, lincoln.