Hello, On Wed, Nov 13, 2019 at 8:35 PM Saku Ytti <saku@ytti.fi> wrote:
On Wed, 13 Nov 2019 at 18:27, Matt Corallo <nanog@as397444.net> wrote:
This sounds like a bug on Cloudflare’s end (cause trying to do anycast TCP is... out of spec to say the least), not a bug in ECN/ECMP.
Not true. Hash result should indicate discreet flow, more importantly discreet flow should not result into two unique hash numbers. Using whole TOS byte breaks this promise and thus breaks ECMP.
Platforms allow you to configure which bytes are part of hash calculation, whole TOS byte should not be used as discreet flow SHOULD have unique ECN bits during congestion. Toke has diagnosed the problem correctly, solution is to remove TOS from ECMP hash calculation.
In fact I believe everything beyond the 5-tuple is just a bad idea to base your hash on. Here are some examples (not quite as straight forward than the TOS/ECN case here): TTL: https://mailman.nanog.org/pipermail/nanog/2018-September/096871.html IPv6 flow label: https://blog.apnic.net/2018/01/11/ipv6-flow-label-misuse-hashing/ https://pc.nanog.org/static/published/meetings/NANOG71/1531/20171003_Jaeggli... https://www.youtube.com/watch?v=b0CRjOpnT7w Lukas