Not to condone what cloudflare is doing, but...

An ECN connection will have different bits on various packets for the duration of the connection -- pure ACKs (ACKs not piggybacking on data) will have the ECN bits as 00b, while all other packets will have either 01b, 10b (when no congestion was experienced) or 11b (when congestion was experienced). So using the ECN bits as part of the hash would affect performance throughout the life of the connection.

On Wed, Nov 13, 2019 at 9:00 AM Matt Corallo <nanog@as397444.net> wrote:

Not ideal, sure, but if it’s only for the SYN (as you seem to indicate), splitting the flow shouldn’t have material performance degradation?

> On Nov 13, 2019, at 11:51, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
>
>
>> On 13 November 2019 17:20:18 CET, Matt Corallo <netadmin@as397444.net> wrote:
>> This sounds like a bug on Cloudflare’s end (cause trying to do anycast
>> TCP is... out of spec to say the least), not a bug in ECN/ECMP.
>
> Even without anycast, an ECMP shouldn't hash on the ECN bits. Doing so will split the flow over multiple paths; avoiding that is the whole point of doing the flow-based hashing in the first place.
>
> Anycast "only" turns a potential degradation of TCP performance into a hard failure... :)
>
> -Toke