The reason why we need larger buffers on some applications is because of TCP implementation detail. When TCP window grows in size (it grows exponentially) the newly created window size is bursted on to the wire at sender speed.

If sender is significantly higher speed than receiver, someone needs to store these bytes, while they are serialised at receiver speed. If we cannot store them, then the window cannot grow to accommodate the banwdith*delay product and the receiver cannot observe ideal TCP receive rate. 

If we'd change TCP sender to bandwidth estimation, and newly created window space would be serialised at estimated receiver rate then we would need dramatically less buffers. However this less aggressive TCP algorithm would be outcompeted by new reno reducing bandwidth estimation to approach zero.

Luckily almost all traffic is handled by few players, if they agree to change to well behaved TCP (or QUIC) algorithm, it doesn't matter much if the long tail is badly behaving TCP. 



On Fri, 9 Apr 2021 at 17:13, Mike Hammett <nanog@ics-il.net> wrote:
I have seen the opposite, where small buffers impacted throughput.

Then again, it was observation only, no research into why, other than superficial.

From: "Tom Beecher" <beecher@beecher.cc>
To: "Mike Hammett" <nanog@ics-il.net>
Cc: "Dmitry Sherman" <dmitry@interhost.net>, "NANOG" <nanog@nanog.org>
Sent: Friday, April 9, 2021 8:40:00 AM
Subject: Re: Trident3 vs Jericho2

If you have all the same port speed, small buffers are fine. If you have 100G and 1G ports, you'll need big buffers wherever the transition to the smaller port speed is located.

While the larger buffer there you are likely to be severely impacting application throughput.

On Fri, Apr 9, 2021 at 9:05 AM Mike Hammett <nanog@ics-il.net> wrote:
What I've observed is that it's better to have a big buffer device when you're mixing port speeds. The more dramatic the port speed differences (and the more of them), the more buffer you need.

If you have all the same port speed, small buffers are fine. If you have 100G and 1G ports, you'll need big buffers wherever the transition to the smaller port speed is located.

From: "Dmitry Sherman" <dmitry@interhost.net>
To: nanog@nanog.org
Sent: Friday, April 9, 2021 7:57:05 AM
Subject: Trident3 vs Jericho2

Once again, which is better shared buffer featurerich or fat buffer switches?
When its better to put big buffer switch? When its better to drop and retransmit instead of queueing?

Thanks.
Dmitry




--
  ++ytti