(skipping up the thread some) On Fri, Mar 20, 2020 at 5:58 PM Jared Mauch <jared@puck.nether.net> wrote:
It’s the protocol 50 IPSEC VPNs. They are very sensitive to path changes and reordering as well.
If you’re tunneling more than 5 or 10Gb/s of IPSEC it’s likely going to be a bad day when you find a low speed link in the middle. Generally providers with these types of flows have both sides on the same network vs going off-net as they’re not stable on peering links that might change paths.
a bunch of times the advice given to folk in this situation is: "Add more entropy", which really for ipsec/gre/etc vpns means more endpoints. For instance, adding 3 more ips on either side for tunnel egress/ingress will make the flows (ideally) smaller and more probable to hash across different links in the intermediary network(s). This also moves the loadbalancing back behind the customer prem so ideally perhaps even the nxM flows are now balanced a little better as well. sometimes this works, sometimes it's hard to accomplish :(