Shopify built a go app called Toxiproxy that would allow for injecting TCP oddballs into an http stream.https://github.com/Shopify/toxiproxy> Toxiproxy is a framework for simulating network conditions. It's made specifically to work in testing, CI and development environments, supporting deterministic tampering with connections, but with support for randomized chaos and customization. Toxiproxy is the tool you need to prove with tests that your application doesn't have single points of failure. We've been successfully using it in all development and test environments at Shopify since October, 2014. See our blog post on resiliency for more information.
--Pete
On Fri, Jan 17, 2025 at 2:42 PM Brandon Martin <lists.nanog@monmotha.net> wrote:On 1/17/25 14:29, William Herrin wrote:
> Well... In theory, TCP closes the segment at the end of the
> application's send() and sets the PSH flag. Likewise, on the receiving
> side the recv() returns before filling the buffer upon receipt of a
> segment with the PSH flag set.
Every segment this thing sends has PSH set which again makes me think
that they've got TCP_NODELAY set but are sending their messages
piecemeal across multiple send/write calls.
The actual high-level messages are fairly small at typically less than
100B. Most implementations end up sending the entire message in a
single TCP segment.
> In theory. In practice, it doesn't always work out that way and
> applications which depend on a short recv() meaning that was where the
> sender's send() ended tend to flake out in unexpected ways.
I don't think that's the issue in this case, but it's a useful thing to
go looking for.
--
Brandon Martin
----Peter A. DeNitto