I won't touch the other stuff in the message, although there are some errors. This leapt out and is related to a couple of private conversations. Marc Slemko <marcs@znep.com> writes:

...

Like maybe a lack of buffer space in terminal server?

I doubt it. Personally, aside from a gain for interactive work over a loaded low bandwidth connection, especially with some sort of priority queueing used, I find no gain from a smaller MTU on a dialup link.

Yes. The problem is not the lack of buffer space but a surplus of it. TCP will always find the choke point in a path and attempt to fill the queue directly in front of it. The congestion window will tend to stabilize on the amount of buffer space available to it at that point plus any additional propagation delay times the bandwidth at the choke point. With a tail-drop queueing "discipline", a TCP flow may have a sizeable amount of buffering available to it before a tail drop occurs; unless this is moderated by something like RED, several seconds of queueing delay should be expected. Moreover, unless, as you note, there is some sort of priority queueing, this means that a new burst of traffic will be delayed by the length of the queue, i.e., several seconds. Not so fun. Note that using small MTUs does not alter this; once TCP has discovered the amount of buffer space in front of the choke point, as long as it keeps transmitting constantly, modulo drops elsewhere, no matter what the MTU size is, the same amount of buffer space will be occupied. In fact, given that the congestion window is measured in bytes and not segments, with a smaller MTU and path MTU discovery (or fragmentation before the choke point) you will have a *longer* queueing delay thanks to the additional header bytes, assuming the queue is "slotted" with N buffers rather than just a linear array of bytes. The appropriate solution is to use RED to moderate the amount of space that any given TCP flow can occupy in a queue to a small number. This allows a flow in equilibrium to adjust to transient congestion due to other traffic through the congestion point, and also allows that other traffic through with potentially much shorter delays, without causing the choke point to become idle. Examining Floyd & Jacobson's work at ftp.ee.lbl.gov is well worthwile, btw. Sean.