Re: High throughput bgp links using gentoo + stipped kernel
This is some fairly ancient hardware, so what you can get out if it will be limited. Though gige should not be impossible.
Agreed!!!
The usual tricks are to make sure netfilter is not loaded, especially the conntrack/nat based parts as that will inspect every flow for state information. Either make sure those parts are compiled out or the modules/code never loads.
If you have any iptables/netfilter rules, make sure they are 1) stateless 2) properly organized (cant just throw everything into FORWARD and expect it to be performant).
We do use a statefull iptables on our router, some forward rules... This is known to be on of our issues, not sure if having a separate iptables box would be the best and only solution for this?
You could try setting IRQ affinity so both ports run on the same core, however I'm not sure if that will help much as its still the same cache and distance to memory. On modern NICS you can do tricks like tie rx of port 1 with tx of port 2. Probably not on that generation though.
Those figures include IRQ affinity tweaks at the kernel and APIC level.
The 82571EB and 82573E is, while old, PCIe hardware, there should not be any PCI bottlenecks, even with you having to bounce off that stone age FSB that old CPU has. Not sure well that generation intel NIC silicon does linerate easily though.
But really you should get some newerish hardware with on-cpu PCIe and memory controllers (and preferably QPI). That architectural jump really upped the networking throughput of commodity hardware, probably by orders of magnitude (people were doing 40Gbps routing using standard Linux 5 years ago).
Any ideas of the setup??? Maybe as far as naming some chipset, interface? And xserver that is the best candidate. Will google.. :)
Curious about vmstat output during saturation, and kernel version too. IPv4 routing changed significantly recently and IPv6 routing performance also improved somewhat.
Will get that output during peak on monday for you guys. Newest kernel 3.6 or 7... Thank you so much for your insight, Nick.
(oops, I keep forgetting to send with my nanog identity..) On 19. mai 2013 17:48, Nick Khamis wrote:
We do use a statefull iptables on our router, some forward rules... This is known to be on of our issues, not sure if having a separate iptables box would be the best and only solution for this?
Ah, statefullness/conntrack .. once you load it you kinda lost already.. Sorry. Any gains from other tunables will likely be dwarfed by the cpu cycles spent by the kernel to track all connections. The more diverse the traffic the more it will hurt. Connection tracking is just inherently non-scalable (and fragile - by the way.) However, the cheapest and simplest is probably just to throw more modern hardware at it. A Xeon E3 (or two for redudancy ;)) is quite cheap.. The long term, scalable solution is a deeper network like you hinted at, with statefullness - if really needed at all - pushed as close to your edge and as far away from your border as possible. But.. More boxes, more to manage, more power, more stuff that can fail, more redudancies needed.. adds up. Then again if you are close to gig actual traffic already, you might want to at least think about future scalability.. <snip>
Any ideas of the setup??? Maybe as far as naming some chipset, interface? And xserver that is the best candidate. Will google.. :)
The big shift to integrated (and fast) I/O happened around 2008 IIRC, anything introduced after that is usually quite efficient at moving packets around, at least if Intel based. Even desktop i3/i5/i7 platforms can do 10gig as long as you make sure you put the network chips/cards on the cpu pcie controllers lanes. With anything new its hard to go wrong. xserver?? xserve? That is quite old..
Curious about vmstat output during saturation, and kernel version too. IPv4 routing changed significantly recently and IPv6 routing performance also improved somewhat.
Will get that output during peak on monday for you guys. Newest kernel 3.6 or 7...
Good. That is at least fairly recent and has most of the more modern networking stuff (and better defaults)
On Sun, May 19, 2013 at 11:48:17AM -0400, Nick Khamis wrote:
We do use a statefull iptables on our router, some forward rules... This is known to be on of our issues, not sure if having a separate iptables box would be the best and only solution for this?
I don't know about "only", but it'd have to come close to "best". iptables (and stateful firewalling in general) is a pretty significant CPU and memory sink. Definitely get rid of any stateful rules, preferably *all* the rules, and apply them at a separate location. We've always had BGP routing separated from firewalling, but we're currently migrating from one-giant-core-firewall to lots-of-little-firewalls because our firewalls are starting to cry a little. Nice thing is that horizontally scaling firewalls is easy -- just whack 'em on each subnet instead of running everything together. Core routing is a little harder to scale out (although as has been described already, by no means impossible). The important thing is to remove *anything* from your core routing boxes that doesn't *absolutely* have to be there -- and stateful firewall rules are *extremely* high on that list. - Matt -- When the revolution comes, they won't be able to FIND the wall. -- Brian Kantor, in the Monastery
This is what we do too: Separate firewalling and routing. We use Vyatta for both and it works. Bye, David -----Oorspronkelijk bericht----- Van: Matt Palmer [mailto:mpalmer@hezmatt.org] Verzonden: zondag 19 mei 2013 23:32 Aan: nanog@nanog.org Onderwerp: Re: High throughput bgp links using gentoo + stipped kernel On Sun, May 19, 2013 at 11:48:17AM -0400, Nick Khamis wrote:
We do use a statefull iptables on our router, some forward rules... This is known to be on of our issues, not sure if having a separate iptables box would be the best and only solution for this?
I don't know about "only", but it'd have to come close to "best". iptables (and stateful firewalling in general) is a pretty significant CPU and memory sink. Definitely get rid of any stateful rules, preferably *all* the rules, and apply them at a separate location. We've always had BGP routing separated from firewalling, but we're currently migrating from one-giant-core-firewall to lots-of-little-firewalls because our firewalls are starting to cry a little. Nice thing is that horizontally scaling firewalls is easy -- just whack 'em on each subnet instead of running everything together. Core routing is a little harder to scale out (although as has been described already, by no means impossible). The important thing is to remove *anything* from your core routing boxes that doesn't *absolutely* have to be there -- and stateful firewall rules are *extremely* high on that list. - Matt -- When the revolution comes, they won't be able to FIND the wall. -- Brian Kantor, in the Monastery
On Sun, May 19, 2013 at 11:48:17AM -0400, Nick Khamis wrote:
We do use a statefull iptables on our router, some forward rules... This is known to be on of our issues, not sure if having a separate iptables box would be the best and only solution for this?
Do you actually need stateful filtering? A lot of people seem to think that it's important, when really they're accomplishing little from it, you can block ports etc without it. And the idea of protecting hosts from strange traffic is only really significant if the hosts have very outdated TCP/IP stacks etc. And it breaks things like having multiple routers. There's an obscure NOTRACK rule you can use to cut down the number of state entries, or remote state tracking for hosts that don't need it. http://serverfault.com/questions/234560/how-to-turn-iptables-stateless although googling for NOTRACK should find other things too. Ben.
On 5/19/13 4:27 PM, Ben wrote:
Do you actually need stateful filtering? A lot of people seem to think that it's important, when really they're accomplishing little from it, you can block ports etc without it.
I believe PCI compliance requires it, other things like it probably do too. ~Seth
On Sun, 19 May 2013 16:42:23 -0700, Seth Mattinen said:
On 5/19/13 4:27 PM, Ben wrote:
Do you actually need stateful filtering? A lot of people seem to think that it's important, when really they're accomplishing little from it, you can block ports etc without it.
I believe PCI compliance requires it, other things like it probably do too.
It's the rare ISP who's border routers are in-scope for PCI compliance.
On Sun, May 19, 2013 at 04:42:23PM -0700, Seth Mattinen wrote:
On 5/19/13 4:27 PM, Ben wrote:
Do you actually need stateful filtering? A lot of people seem to think that it's important, when really they're accomplishing little from it, you can block ports etc without it.
I believe PCI compliance requires it, other things like it probably do too.
There'd be very few PCI compliant sites if PCI required stateful firewalling in core routers. - Matt
Just curious and perhaps off topic a tad but; is the stateful filtering of sessions on a router to replace a firewall? Or is there another reason to do it? I could see a benefit of creating blacklists, however, I'm struggling with what other benefits it would provide...service aware load-balancing? I'm very interested to learn what other strategies and or design considerations would be made with thinking of using filtering on a router. I'm perfectly willing to accept consolidation of services :-) On Mon, May 20, 2013 at 3:45 PM, Matt Palmer <mpalmer@hezmatt.org> wrote:
On Sun, May 19, 2013 at 04:42:23PM -0700, Seth Mattinen wrote:
On 5/19/13 4:27 PM, Ben wrote:
Do you actually need stateful filtering? A lot of people seem to think that it's important, when really they're accomplishing little from it, you can block ports etc without it.
I believe PCI compliance requires it, other things like it probably do too.
There'd be very few PCI compliant sites if PCI required stateful firewalling in core routers.
- Matt
-- Phil Fagan Denver, CO 970-480-7618
On Mon, 20 May 2013, Phil Fagan wrote:
Just curious and perhaps off topic a tad but; is the stateful filtering of sessions on a router to replace a firewall? Or is there another reason to do it? I could see a benefit of creating blacklists, however, I'm struggling with what other benefits it would provide...service aware load-balancing? I'm very interested to learn what other strategies and or design considerations would be made with thinking of using filtering on a router.
I'm perfectly willing to accept consolidation of services :-)
Stateful firewalling is also painful in environments where path asymmetry could exist, since either the routing policy would need to be designed to enforce symmetry (more complex, less reliable), or the stateful firewalling devices would need to have a way to share state information with each other to accommodate asymmetry. jms
On 5/20/13 2:45 PM, Matt Palmer wrote: > On Sun, May 19, 2013 at 04:42:23PM -0700, Seth Mattinen wrote: >> On 5/19/13 4:27 PM, Ben wrote: >>> Do you actually need stateful filtering? A lot of people seem to think >>> that it's important, when really they're accomplishing little from it, >>> you can block ports etc without it. >> I believe PCI compliance requires it, other things like it probably do too. > There'd be very few PCI compliant sites if PCI required stateful firewalling > in core routers. Putting your border router in scope for your pci environment is imho an engineering/documentation mistake. > - Matt > >
On Sun, May 19, 2013 at 11:48:17AM -0400, Nick Khamis wrote:
But really you should get some newerish hardware with on-cpu PCIe and memory controllers (and preferably QPI). That architectural jump really upped the networking throughput of commodity hardware, probably by orders of magnitude (people were doing 40Gbps routing using standard Linux 5 years ago). Any ideas of the setup??? Maybe as far as naming some chipset, interface? And xserver that is the best candidate. Will google.. :)
Base model e5 CPU is generally considered adequate, and has direct link between cache and PCI bypassing memory. http://www.intel.com/content/www/us/en/io/data-direct-i-o-faq.html Motherboard is likely to have i350 chipset for ethernet. http://www.intel.com/content/www/us/en/ethernet-controllers/ethernet-i350-se... Ben.
participants (10)
-
Andre Tomt -
Ben -
joel jaeggli -
Justin M. Streiner -
MailPlus| David Hofstee -
Matt Palmer -
Nick Khamis -
Phil Fagan -
Seth Mattinen -
Valdis.Kletnieks＠vt.edu