On Sun, Mar 5, 2023 at 3:02 PM Alexander Huynh via NANOG <nanog@nanog.org> wrote:

On 2023-03-05 12:34:40 -0800, Dave Taht wrote:

...

I rather enjoyed doing this podcast a few weeks ago, (and enjoy this podcast a lot, generally), and it talks to what I've been up to for the past year or so on fixing bufferbloat for ISPs.

https://packetpushers.net/podcast/heavy-networking-666-improving-quality-of-...

Thank you for the link! I'll give it a listen this evening.

Pathetically, what did you think?

...

I am kind of curious as to how much XDP and EBPF now exist in the nanog universe and other applications y'all are finding for it?

We at Cloudflare use both XDP and eBPF extensively for our load balancing and DoS mitigation applications: https://www.google.com/search?q=site%3Ablog.cloudflare.com+xdp+OR+ebpf

David Tubes (of cloudflare) gave a pretty good talk at our recently concluded "understanding latency conference. Toke gave a pretty good talk on the state of bpf Kathie Nichols talked about some nifty packet analysis techniques Vodaphone opened with surprising candor about there being no demand for > 1gbit Nokia talked about L4S Stuart Cheshire of apple talked about their RPM metric Had a good panel with ookla And me, I channeled Roy Beatty from Blade Runner for all the network problems I have seen and attempted to fix in just the past few weeks, for a 3 minute monologue. https://www.youtube.com/watch?v=tAVwmUG21OY&t=6483s :) I would really like to start a "back to packet captures" movement! Have MS-clippy show up and say "Your network is being weird, would you like me to take a packet capture?", and/or daveGPT3 chime in. Anyway, last blatant plug on this list, if you want to feel some mild winds of positive change, please cue up: https://www.understandinglatency.com/recordings-2023 and pass around.

-- Alex

-- Come Heckle Mar 6-9 at: https://www.understandinglatency.com/ Dave Täht CEO, TekLibre, LLC

On 3/5/23 7:00 PM, Matt Corallo wrote:

On 3/5/23 12:34 PM, Dave Taht wrote:

...

I rather enjoyed doing this podcast a few weeks ago, (and enjoy this podcast a lot, generally), and it talks to what I've been up to for the past year or so on fixing bufferbloat for ISPs.

https://packetpushers.net/podcast/heavy-networking-666-improving-quality-of-...

I am kind of curious as to how much XDP and EBPF now exist in the nanog universe and other applications y'all are finding for it?

I while back I had to make fragments passing through Linux boxes kinda sorta reliable-ish. Sadly, Linux's fragment reassembly code has a wonderful timeout to hold onto potential packets for reassembly of 30 *seconds*, and if it has stuff its holding on to it can throw away new stuff [1].

Because frag'ing happens pretty early in receive/usually on the nic in send, rewriting packets in normal firewall rules can be annoying. Turns out this was easier to do by just slapping a "rewrite the packet to make it pretend its not a frag and turn it back on on the other side" eBPF program on tc on the outbound side and xdp on the inbound edge.

Its all a bit low-level (gotta write your own packet reading), but incredibly powerful when you need to do something dirty (or more performant, in the cloudflare case, dunno if they're doing the in-hardware XDP stuff or not).

Matt

[1] I tried to change it, the constant literally predates linux-in-git but, hey, apparently sat links are more important to support https://patchwork.kernel.org/project/netdevbpf/patch/fdcac2a0-5036-f1c8-a926...

To clarify, since it was pointed out sat links *are* important to support, I really should have said earth-mars links here. 30 seconds isn't a realistic sat link latency either these days, let alone jitter :). Honestly I'm not really sure where you could even find 30 seconds of jitter anywhere, but, hey the constant literally came from RFC 791 and we can't change that now, apparently. Matt