I get more than that with realtek nics on x86, problem is high interrupt rates even with msix, intel fixes some of those and chelsio makes it all go away... Just saying :) On 26/01/2015 23:27, Faisal Imtiaz wrote:

Hi Micah,

There is a segment in the Hardware Side of the industry that produces "Network Appliances". (Folks such as Axiomtek, Lanner Electronics, Caswell Networks, Portwell etc etc)

These appliances are commonly used as a commercial (OEM) platform for a variety of uses.. Routers, Firewalls, Specialized network applications etc.

Our internal testing ( informal), matches up with the commonly quoted PPS handling by the different product vendors who incorporate these appliances in their network product offerings.

i3/i5/i7 (x86) based network appliances will forward traffic as long as pps does not exceed 1.4million (In our testing we found the pps to be limiting factor and not the amount of traffic being moved) (will easily handle 6G to 10G of traffic

Core2duo (x86) based network appliances will forward traffic as long as pps does not exceed 600,0000 pps (will easily handle 1.5G to 2G of traffic)

Atom based (x86) network appliances will forward traffic as long as pps does not exceed 250,000 pps.

----------------------------------------

Of course, if you start to bog down the router with lots of NAT/ACL/ Bridge Rules (i.e. the CPU has to get involved in traffic management) then your actual performance will be degraded.

Regards.

Faisal Imtiaz Snappy Internet & Telecom 7266 SW 48 Street Miami, FL 33155 Tel: 305 663 5518 x 232

Help-desk: (305)663-5518 Option 2 or Email: Support@Snappytelecom.net

----- Original Message -----

...

From: "micah anderson" <micah@riseup.net> To: nanog@nanog.org Sent: Monday, January 26, 2015 5:53:54 PM Subject: scaling linux-based router hardware recommendations

Hi,

I know that specially programmed ASICs on dedicated hardware like Cisco, Juniper, etc. are going to always outperform a general purpose server running gnu/linux, *bsd... but I find the idea of trying to use proprietary, NSA-backdoored devices difficult to accept, especially when I don't have the budget for it.

I've noticed that even with a relatively modern system (supermicro with a 4 core 1265LV2 CPU, with a 9MB cache, Intel E1G44HTBLK Server adapters, and 16gig of ram, you still tend to get high percentage of time working on softirqs on all the CPUs when pps reaches somewhere around 60-70k, and the traffic approaching 600-900mbit/sec (during a DDoS, such hardware cannot typically cope).

It seems like finding hardware more optimized for very high packet per second counts would be a good thing to do. I just have no idea what is out there that could meet these goals. I'm unsure if faster CPUs, or more CPUs is really the problem, or networking cards, or just plain old fashioned tuning.

Any ideas or suggestions would be welcome! micah

And the solution to this issue is - http://routerboard.com/ or http://www.mikrotik.com/software# on x86 hardware, plus any basic layer2 switch. Don't scoff until you have tried it, the price/performance is pretty staggering if you are in the sub 20gig space. -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Mike Hammett Sent: Tuesday, 27 January 2015 2:44 p.m. To: nanog@nanog.org Subject: Re: scaling linux-based router hardware recommendations Aren't most of the new whitebox\open source platforms based on switching and not routing? I'd assume that the "cloud-scale" data centers deploying this stuff still have more traditional big iron at their cores. The small\medium sized ISP usually is left behind. They're not big enough to afford the big new hardware, but all of their user's NetFlix and porn and whatever else they do is chewing up bandwidth. For example, the small\medium ISPs are at the Nx10GigE stage now. The new hardware is expensive, the old hardware (besides being old) is likely in a huge chassis if you can get any sort of port density at all. 48 port GigE switches with a couple 10GigE can be had for $100. A minimum of 24 port 10GigE switches (except for the occasional IBM switch ) is 30x to 40x times that. Routers (BGP, MPLS, etc.) with that more than just a couple 10GigEs are even more money, I'd assume. I thought vMX was going to save the day, but it's pricing for 10 gigs of traffic (licensed by throughput and standard\advanced licenses) is really about 5x - 10x what I'd be willing to pay for it. Haven't gotten a quote from AlcaLu yet. Vyatta (last I checked, which was admittedly some time ago) doesn't have MPLS. The FreeBSD world can bring zero software cost and a stable platform, but no MPLS. Mikrotik brings most (though not all) of the features one would want... a good enough feature set, let's say... but is a non-stop flow of bugs. I don't think a week or two goes by where one of my friends doesn't submit some sort of reproducible bug to Mikrotik. They've also been "looking into" DPDK for 2.5 years now. hasn't shown up yet. I've used MT for 10 years and I'm always left wanting just a little more, but it may be the best balance between the features and performance I want and the ability to pay for it. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com ----- Original Message ----- From: "Mehmet Akcin" <mehmet@akcin.net> To: "micah anderson" <micah@riseup.net> Cc: nanog@nanog.org Sent: Monday, January 26, 2015 6:06:53 PM Subject: Re: scaling linux-based router hardware recommendations Cumulus Networks has some stuff, http://www.bigswitch.com/sites/default/files/presentations/onug-baremetal-20... Pretty decent presentation with more details you like. Mehmet

On Jan 26, 2015, at 8:53 PM, micah anderson <micah@riseup.net> wrote:

Hi,

I know that specially programmed ASICs on dedicated hardware like Cisco, Juniper, etc. are going to always outperform a general purpose server running gnu/linux, *bsd... but I find the idea of trying to use proprietary, NSA-backdoored devices difficult to accept, especially when I don't have the budget for it.

I've noticed that even with a relatively modern system (supermicro with a 4 core 1265LV2 CPU, with a 9MB cache, Intel E1G44HTBLK Server adapters, and 16gig of ram, you still tend to get high percentage of time working on softirqs on all the CPUs when pps reaches somewhere around 60-70k, and the traffic approaching 600-900mbit/sec (during a DDoS, such hardware cannot typically cope).

It seems like finding hardware more optimized for very high packet per second counts would be a good thing to do. I just have no idea what is out there that could meet these goals. I'm unsure if faster CPUs, or more CPUs is really the problem, or networking cards, or just plain old fashioned tuning.

Any ideas or suggestions would be welcome! micah

Like Mike mentioned, the feature list in RouterOS is nothing short of impressive -- problem is that pretty much everything in there is inherently buggy. That and one hell of a painful syntax-schema to work with too. On 1/27/2015 午前 10:57, Tony Wicks wrote:

And the solution to this issue is - http://routerboard.com/ or http://www.mikrotik.com/software# on x86 hardware, plus any basic layer2 switch. Don't scoff until you have tried it, the price/performance is pretty staggering if you are in the sub 20gig space.

-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Mike Hammett Sent: Tuesday, 27 January 2015 2:44 p.m. To: nanog@nanog.org Subject: Re: scaling linux-based router hardware recommendations

Aren't most of the new whitebox\open source platforms based on switching and not routing? I'd assume that the "cloud-scale" data centers deploying this stuff still have more traditional big iron at their cores.

The small\medium sized ISP usually is left behind. They're not big enough to afford the big new hardware, but all of their user's NetFlix and porn and whatever else they do is chewing up bandwidth. For example, the small\medium ISPs are at the Nx10GigE stage now. The new hardware is expensive, the old hardware (besides being old) is likely in a huge chassis if you can get any sort of port density at all.

48 port GigE switches with a couple 10GigE can be had for $100. A minimum of 24 port 10GigE switches (except for the occasional IBM switch ) is 30x to 40x times that. Routers (BGP, MPLS, etc.) with that more than just a couple 10GigEs are even more money, I'd assume.

I thought vMX was going to save the day, but it's pricing for 10 gigs of traffic (licensed by throughput and standard\advanced licenses) is really about 5x - 10x what I'd be willing to pay for it.

Haven't gotten a quote from AlcaLu yet.

Vyatta (last I checked, which was admittedly some time ago) doesn't have MPLS.

The FreeBSD world can bring zero software cost and a stable platform, but no MPLS.

Mikrotik brings most (though not all) of the features one would want... a good enough feature set, let's say... but is a non-stop flow of bugs. I don't think a week or two goes by where one of my friends doesn't submit some sort of reproducible bug to Mikrotik. They've also been "looking into" DPDK for 2.5 years now. hasn't shown up yet. I've used MT for 10 years and I'm always left wanting just a little more, but it may be the best balance between the features and performance I want and the ability to pay for it.

----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com

----- Original Message -----

From: "Mehmet Akcin" <mehmet@akcin.net> To: "micah anderson" <micah@riseup.net> Cc: nanog@nanog.org Sent: Monday, January 26, 2015 6:06:53 PM Subject: Re: scaling linux-based router hardware recommendations

Cumulus Networks has some stuff,

http://www.bigswitch.com/sites/default/files/presentations/onug-baremetal-20...

Pretty decent presentation with more details you like.

Mehmet

...

On Jan 26, 2015, at 8:53 PM, micah anderson <micah@riseup.net> wrote:

Hi,

I know that specially programmed ASICs on dedicated hardware like Cisco, Juniper, etc. are going to always outperform a general purpose server running gnu/linux, *bsd... but I find the idea of trying to use proprietary, NSA-backdoored devices difficult to accept, especially when I don't have the budget for it.

I've noticed that even with a relatively modern system (supermicro with a 4 core 1265LV2 CPU, with a 9MB cache, Intel E1G44HTBLK Server adapters, and 16gig of ram, you still tend to get high percentage of time working on softirqs on all the CPUs when pps reaches somewhere around 60-70k, and the traffic approaching 600-900mbit/sec (during a DDoS, such hardware cannot typically cope).

It seems like finding hardware more optimized for very high packet per second counts would be a good thing to do. I just have no idea what is out there that could meet these goals. I'm unsure if faster CPUs, or more CPUs is really the problem, or networking cards, or just plain old fashioned tuning.

Any ideas or suggestions would be welcome! micah

On Tue, 27 Jan 2015 11:10:54 +0900, "Paul S." said:

Like Mike mentioned, the feature list in RouterOS is nothing short of impressive -- problem is that pretty much everything in there is inherently buggy.

That and one hell of a painful syntax-schema to work with too.

Latvian grammar is.. somewhat unusual. Just be glad the development team wasn't Finnish. :) (Sorry, I couldn't resist. :)

Depends on the hardware. 30 - 45 seconds for the higher end stuff? I'm not sure how long it is on an RB750 (list price of like $40). ;-) ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com ----- Original Message ----- From: "Ken Chase" <math@sizone.org> To: nanog@nanog.org Sent: Monday, January 26, 2015 9:29:28 PM Subject: Re: scaling linux-based router hardware recommendations Hows convergence time on these mikrotik/ubiquity/etc units for a full table? /kc -- Ken Chase - math@sizone.org Toronto

On 27/01/2015, at 4:29 pm, Ken Chase <math@sizone.org> wrote:

Hows convergence time on these mikrotik/ubiquity/etc units for a full table?

For the CCR1036-12G-4S with one full table, one domestic table (NZ - ~26k entries) some peering and iBGP full convergence took about three minutes forty seconds last time I timed it from cold. I may do some new timing as they have been working hard to improve the multi core support (currently BGP still only single core however they been doing some work on efficient allocation of other tasks to cores.

/kc -- Ken Chase - math@sizone.org Toronto

Under 30sec (more like 15 to 20) on an i7 based Mikrotik for full BGP Tables. Faisal Imtiaz ----- Original Message -----

From: "Ken Chase" <math@sizone.org> To: nanog@nanog.org Sent: Monday, January 26, 2015 10:29:28 PM Subject: Re: scaling linux-based router hardware recommendations

Hows convergence time on these mikrotik/ubiquity/etc units for a full table?

/kc -- Ken Chase - math@sizone.org Toronto

Easy to make a switch when the only thing you're actually doing is teling the asic what to do (Cumulus, Ubiquiti, ... every other broadcom vendor out there...) Better yet - Atheros have finally come out with a 24*1GE + 2*10GE switch asic - only a matter of time before they challenge broadcom et al. On 27/01/2015 01:43, Mike Hammett wrote:

Aren't most of the new whitebox\open source platforms based on switching and not routing? I'd assume that the "cloud-scale" data centers deploying this stuff still have more traditional big iron at their cores.

The small\medium sized ISP usually is left behind. They're not big enough to afford the big new hardware, but all of their user's NetFlix and porn and whatever else they do is chewing up bandwidth. For example, the small\medium ISPs are at the Nx10GigE stage now. The new hardware is expensive, the old hardware (besides being old) is likely in a huge chassis if you can get any sort of port density at all.

48 port GigE switches with a couple 10GigE can be had for $100. A minimum of 24 port 10GigE switches (except for the occasional IBM switch ) is 30x to 40x times that. Routers (BGP, MPLS, etc.) with that more than just a couple 10GigEs are even more money, I'd assume.

I thought vMX was going to save the day, but it's pricing for 10 gigs of traffic (licensed by throughput and standard\advanced licenses) is really about 5x - 10x what I'd be willing to pay for it.

Haven't gotten a quote from AlcaLu yet.

Vyatta (last I checked, which was admittedly some time ago) doesn't have MPLS.

The FreeBSD world can bring zero software cost and a stable platform, but no MPLS.

Mikrotik brings most (though not all) of the features one would want... a good enough feature set, let's say... but is a non-stop flow of bugs. I don't think a week or two goes by where one of my friends doesn't submit some sort of reproducible bug to Mikrotik. They've also been "looking into" DPDK for 2.5 years now. hasn't shown up yet. I've used MT for 10 years and I'm always left wanting just a little more, but it may be the best balance between the features and performance I want and the ability to pay for it.

----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com

----- Original Message -----

From: "Mehmet Akcin" <mehmet@akcin.net> To: "micah anderson" <micah@riseup.net> Cc: nanog@nanog.org Sent: Monday, January 26, 2015 6:06:53 PM Subject: Re: scaling linux-based router hardware recommendations

Cumulus Networks has some stuff,

http://www.bigswitch.com/sites/default/files/presentations/onug-baremetal-20...

Pretty decent presentation with more details you like.

Mehmet

...

On Jan 26, 2015, at 8:53 PM, micah anderson <micah@riseup.net> wrote:

Hi,

I know that specially programmed ASICs on dedicated hardware like Cisco, Juniper, etc. are going to always outperform a general purpose server running gnu/linux, *bsd... but I find the idea of trying to use proprietary, NSA-backdoored devices difficult to accept, especially when I don't have the budget for it.

I've noticed that even with a relatively modern system (supermicro with a 4 core 1265LV2 CPU, with a 9MB cache, Intel E1G44HTBLK Server adapters, and 16gig of ram, you still tend to get high percentage of time working on softirqs on all the CPUs when pps reaches somewhere around 60-70k, and the traffic approaching 600-900mbit/sec (during a DDoS, such hardware cannot typically cope).

It seems like finding hardware more optimized for very high packet per second counts would be a good thing to do. I just have no idea what is out there that could meet these goals. I'm unsure if faster CPUs, or more CPUs is really the problem, or networking cards, or just plain old fashioned tuning.

Any ideas or suggestions would be welcome! micah

One thing to note about Ubiquiti's EdgeMax products is that they are not Intel based. They use Cavium Octeon's (at least that's what my EdgeRouter Lite has in it). Oliver ------------------------------------- Oliver Garraux Check out my blog: blog.garraux.net Follow me on Twitter: twitter.com/olivergarraux On Mon, Jan 26, 2015 at 4:18 PM, Joe Greco <jgreco@ns.sol.net> wrote:

...

I know that specially programmed ASICs on dedicated hardware like Cisco, Juniper, etc. are going to always outperform a general purpose server running gnu/linux, *bsd... but I find the idea of trying to use proprietary, NSA-backdoored devices difficult to accept, especially when I don't have the budget for it.

I've noticed that even with a relatively modern system (supermicro with a 4 core 1265LV2 CPU, with a 9MB cache, Intel E1G44HTBLK Server adapters, and 16gig of ram, you still tend to get high percentage of time working on softirqs on all the CPUs when pps reaches somewhere around 60-70k, and the traffic approaching 600-900mbit/sec (during a DDoS, such hardware cannot typically cope).

It seems like finding hardware more optimized for very high packet per second counts would be a good thing to do. I just have no idea what is out there that could meet these goals. I'm unsure if faster CPUs, or more CPUs is really the problem, or networking cards, or just plain old fashioned tuning.

10-15 years ago, we were seeing early Pentium 4 boxes capable of moving 100Kpps+ on FreeBSD. See for example http://info.iet.unipi.it/~luigi/polling/

Luigi moved on to Netmap, which looks promising for this sort of thing. https://www.usenix.org/system/files/conference/atc12/atc12-final186.pdf I was under the impression that some people have been using this for 10G routing.

Also I'll note that Ubiquiti has some remarkable low-power gear capable of 1Mpps+.

... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

I'm also in the research stage of building our own router. I'm interested in reading more if you can post links to some of this research and/or testing. David Sent from my iPad

On Jan 26, 2015, at 6:45 PM, Phil Bedard <bedard.phil@gmail.com> wrote:

Kind of unsurprisingly, the traditional network vendors are somewhat at the forefront of pushing what an x86 server can do as well. Brocade (Vyatta), Juniper, and Alcatel-Lucent all have virtualized routers using Intel's DPDK pushing 5M+ PPS at this point. They are all also tweaking what Intel is providing, and they are the ones with lots of software developers with a lot of hardware and network programming experience.

ALU claims to be able to get 160Gbps full duplex through a 2RU server with 16x10G interfaces and two 10-core latest-gen Xeon processors. Of course that's probably at 9000 byte packet sizes, but at Imix type traffic it's probably still pushing 60-70Gbps. They have a demo of lots of them in a single rack managed as a single router pushing Tbps.

A commerical offering you are going to pay for that kind of performance and the control plane software. Over time though you'll see the DPDK type enhancements make it into standard OS stacks. Other options include servers with integrated network processors or NPs on a PCI card, there is a whole rash of those type of devices out there now and coming out.

Phil

...

On 1/26/15, 22:53, "micah anderson" <micah@riseup.net> wrote:

Hi,

I know that specially programmed ASICs on dedicated hardware like Cisco, Juniper, etc. are going to always outperform a general purpose server running gnu/linux, *bsd... but I find the idea of trying to use proprietary, NSA-backdoored devices difficult to accept, especially when I don't have the budget for it.

I've noticed that even with a relatively modern system (supermicro with a 4 core 1265LV2 CPU, with a 9MB cache, Intel E1G44HTBLK Server adapters, and 16gig of ram, you still tend to get high percentage of time working on softirqs on all the CPUs when pps reaches somewhere around 60-70k, and the traffic approaching 600-900mbit/sec (during a DDoS, such hardware cannot typically cope).

It seems like finding hardware more optimized for very high packet per second counts would be a good thing to do. I just have no idea what is out there that could meet these goals. I'm unsure if faster CPUs, or more CPUs is really the problem, or networking cards, or just plain old fashioned tuning.

Any ideas or suggestions would be welcome! micah

Hello! You could try to build simple router with DPDK yourself. It's very straightforward and have good examples for simple routing. I have done some tests with PF_RING ZC (it's very similar technology to DPDK without specialization on building of network devices) while test my DDoS monitoring solution and it work perfectly. I can achieve 8 million of packets per second (10GE with 120byte packets) on very slow Intel Xeon E5 2420. You could look at this tests from PF_RING developers: http://www.ntop.org/pf_ring/pf_ring-dna-rfc-2544-benchmark/ But building router on top of PF_RING or DPDK is very challenging task because everyone want very different things (BGP, OSPF, RIP... etc.). On Tue, Jan 27, 2015 at 1:54 PM, Paul S. <contact@winterei.se> wrote:

Anyone aware of any dpdk enabled solutions in the software routing space that doesn't cost an arm and a leg?

vMX certainly does.

On 1/27/2015 午後 04:33, Pavel Odintsov wrote:

...

Hello!

Looks like somebody want to build Linux soft router!) Nice idea for routing 10-30 GBps. I route about 5+ Gbps in Xeon E5-2620v2 with 4 10GE cards Intel 82599 and Debian Wheezy 3.2 (but it's really terrible kernel, everyone should use modern kernels since 3.16 because "buggy linux route cache"). My current processor load on server is about: 15%, thus I can route about 15 GE on my Linux server.

Surely, you should deploy backup server too if master server fails.

On Tue, Jan 27, 2015 at 1:53 AM, micah anderson <micah@riseup.net> wrote:

...

Hi,

I know that specially programmed ASICs on dedicated hardware like Cisco, Juniper, etc. are going to always outperform a general purpose server running gnu/linux, *bsd... but I find the idea of trying to use proprietary, NSA-backdoored devices difficult to accept, especially when I don't have the budget for it.

I've noticed that even with a relatively modern system (supermicro with a 4 core 1265LV2 CPU, with a 9MB cache, Intel E1G44HTBLK Server adapters, and 16gig of ram, you still tend to get high percentage of time working on softirqs on all the CPUs when pps reaches somewhere around 60-70k, and the traffic approaching 600-900mbit/sec (during a DDoS, such hardware cannot typically cope).

It seems like finding hardware more optimized for very high packet per second counts would be a good thing to do. I just have no idea what is out there that could meet these goals. I'm unsure if faster CPUs, or more CPUs is really the problem, or networking cards, or just plain old fashioned tuning.

Any ideas or suggestions would be welcome! micah

-- Sincerely yours, Pavel Odintsov

There are some interesting ideas. There are tricks to getting 128K LPM routes into a Trident2 device like you mentioned. You can get the same type of devices from Cisco/Juniper for not a whole lot more, what you are really paying for is the mature control plane. https://github.com/dbarrosop/sir is a project from David Barasso at Spotify. The BGP daemon on your Internet-connected device may store all the routes in its RIB but usually doesn't need everything in the FIB. A downstream BGP controller running on a x86 server would analyze the routes more closely and use something like Netflow/Sflow (or really any criteria) to identify the routes you really care about and install those into the FIB on the device and just use the default for everything else. Metaswitch, a commercial control-plane company, had a similar idea using Openflow where the actual Internet connected device just proxied BGP connections to a controller which then went back and programmed the upstream Openflow switches. They call them "lean transit switches." The major network vendors all have the ability to run their control planes in software on x86 these days just fine. On many newer platforms they them run the software as a VM anyways. Pricing for bringing your own server is going to be cheaper, but not free. As for open source type stuff, Contrail from Juniper was made open source and has a BGP implementation, MPLS (MPLSoUDP) implementation, and a Linux kernel module to do fairly high speed packet forwarding, what they call the vRouter in Contrail. Wind River is another vendor who has incorporated the Intel DPDK stuff into a Linux distribution, but it is commercial as well. Phil On 1/27/15, 11:56, "Baldur Norddahl" <baldur.norddahl@gmail.com> wrote:

I propose the hybrid solution:

A device such as the ZTE 5960e with 24x 10G and 2x 40G will set you about USD 6000 back.

This thing can do MPLS and L3 equal cost multiple path routing. With that you can load balance across as many software routers as you need.

It also speaks BGP and can accept about 10k routes. So maybe you could consider if the full table is really worth it.

It would be possible to have your software router speak BGP with the neighbors and use next hop to direct the traffic directly to the switch. Or use proxy arp if the peer does not want to allow you to specify a different next hop than the BGP speaker. This way your software router is only moving outgoing packets. Inbound packets will never go through the computer, but will instead be delivered directly to the correct destination by hardware switching.

If you are an ISP, you will often have more inbound traffic so this very useful. Also the weak point of the software router is denial of service attacks with small packets. The attacks are likely from outside your network so your software router will not need to route it.

We need someone to code a BGP daemon, that will export the 5k most used routes to the switch. This way you can have the switch deliver the majority of the traffic directly to your peers.

If you are a service provider, much of your traffic is outbound. Put your servers or multiple routers/firewalls on the same vlan as your transit. Then add static host routes for next hop on all servers. This way you can have as many servers as you need to deliver traffic directly. You can run iBGP on all the servers, so every server knows how to route outbound by itself. MPLS would also be useful for this instead of vlan, but there is no good MPLS implementation for Linux.

Regards,

Baldur

Can be Freebsd-based? http://info.iet.unipi.it/~luigi/netmap/ 2015-01-27 14:22 GMT-02:00 Hugo Slabbert <hugo@slabnet.com>:

There is also some work in progress to improve network performance in the Linux kernel:

https://lwn.net/Articles/629155/

Preliminary, but encouraging that work is under way.

-- Hugo

On Tue 2015-Jan-27 11:33:16 +0400, Pavel Odintsov < pavel.odintsov@gmail.com> wrote:

Hello!

...

Looks like somebody want to build Linux soft router!) Nice idea for routing 10-30 GBps. I route about 5+ Gbps in Xeon E5-2620v2 with 4 10GE cards Intel 82599 and Debian Wheezy 3.2 (but it's really terrible kernel, everyone should use modern kernels since 3.16 because "buggy linux route cache"). My current processor load on server is about: 15%, thus I can route about 15 GE on my Linux server.

Surely, you should deploy backup server too if master server fails.

On Tue, Jan 27, 2015 at 1:53 AM, micah anderson <micah@riseup.net> wrote:

...

Hi,

I know that specially programmed ASICs on dedicated hardware like Cisco, Juniper, etc. are going to always outperform a general purpose server running gnu/linux, *bsd... but I find the idea of trying to use proprietary, NSA-backdoored devices difficult to accept, especially when I don't have the budget for it.

I've noticed that even with a relatively modern system (supermicro with a 4 core 1265LV2 CPU, with a 9MB cache, Intel E1G44HTBLK Server adapters, and 16gig of ram, you still tend to get high percentage of time working on softirqs on all the CPUs when pps reaches somewhere around 60-70k, and the traffic approaching 600-900mbit/sec (during a DDoS, such hardware cannot typically cope).

It seems like finding hardware more optimized for very high packet per second counts would be a good thing to do. I just have no idea what is out there that could meet these goals. I'm unsure if faster CPUs, or more CPUs is really the problem, or networking cards, or just plain old fashioned tuning.

Any ideas or suggestions would be welcome! micah

-- Sincerely yours, Pavel Odintsov

-- Hugo

-- Eduardo Schoedler

On Jan 27, 2015, at 8:31 AM, Jim Shankland <nanog@shankland.org> wrote:

My expertise, such as it ever was, is a bit stale at this point, and my figures might be a little off. But I think the general principle applies: think about the minimum number of x86 instructions, and the minimum number of main memory accesses, to inspect a packet header, do a routing table lookup, and enqueue the packet on an outbound interface. I can't see that ever getting reduced to the point where a generic server can handle 40-byte packets at line rate (for that matter, "line rate" is increasing a lot faster than "speed of generic server" these days).

Using DPDK it’s possible to do everything stated and achieve 10Gbps line rate at 64byte packets on multiple interfaces simultaneously. Add ACLs to the test setup and you can reach significant portions of 10Gbps at 64byte packets and full line rate at 128bytes. Check out Venky Venkatesan’s presentation at the last DPDK Summit for interesting information on pps/CPU cycles and some of the things that can be done to optimize forwarding in a generic processor environment. http://www.slideshare.net/jstleger/6-dpdk-summit-2014-intel-presentation-ven...

I was trying not to pitch my company on list, but the performance numbers I quoted are on the Vyatta/Brocade vRouter which is commercially available. Other vendors also also have publicly available performance numbers that are interesting.

On Jan 28, 2015, at 5:02 AM, Paul S. <contact@winterei.se> wrote:

That's the problem though.

Everyone has presentations for the most part, very few actual tools that end users can just use exist.

On 1/28/2015 午後 08:02, Robert Bays wrote:

...

...

On Jan 27, 2015, at 8:31 AM, Jim Shankland <nanog@shankland.org> wrote:

My expertise, such as it ever was, is a bit stale at this point, and my figures might be a little off. But I think the general principle applies: think about the minimum number of x86 instructions, and the minimum number of main memory accesses, to inspect a packet header, do a routing table lookup, and enqueue the packet on an outbound interface. I can't see that ever getting reduced to the point where a generic server can handle 40-byte packets at line rate (for that matter, "line rate" is increasing a lot faster than "speed of generic server" these days). Using DPDK it’s possible to do everything stated and achieve 10Gbps line rate at 64byte packets on multiple interfaces simultaneously. Add ACLs to the test setup and you can reach significant portions of 10Gbps at 64byte packets and full line rate at 128bytes.

Check out Venky Venkatesan’s presentation at the last DPDK Summit for interesting information on pps/CPU cycles and some of the things that can be done to optimize forwarding in a generic processor environment.

http://www.slideshare.net/jstleger/6-dpdk-summit-2014-intel-presentation-ven...

qnx os based router works well with powerpc, could be pushed far higher load than intel based chips Colin

...

That's the problem though.

Everyone has presentations for the most part, very few actual tools that end users can just use exist.

...

...

On Jan 27, 2015, at 8:31 AM, Jim Shankland <nanog@shankland.org> wrote:

My expertise, such as it ever was, is a bit stale at this point, and my figures might be a little off. But I think the general principle applies: think about the minimum number of x86 instructions, and the minimum number of main memory accesses, to inspect a packet header, do a routing table lookup, and enqueue the packet on an outbound interface. I can't see that ever getting reduced to the point where a generic server can handle 40-byte packets at line rate (for that matter, "line rate" is increasing a lot faster than "speed of generic server" these days). Using DPDK it’s possible to do everything stated and achieve 10Gbps

On 1/28/2015 午後 08:02, Robert Bays wrote: line rate at 64byte packets on multiple interfaces simultaneously. Add ACLs to the test setup and you can reach significant portions of 10Gbps at 64byte packets and full line rate at 128bytes.

...

Check out Venky Venkatesan’s presentation at the last DPDK Summit for

interesting information on pps/CPU cycles and some of the things that can be done to optimize forwarding in a generic processor environment.

...

http://www.slideshare.net/jstleger/6-dpdk-summit-2014-intel-presentation-ven...

...

!DSPAM:54c8de34274511264773590!

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

How is that a problem? --- Theory is when you know everything but nothing works. Practice is when everything works but no one knows why. Sometimes theory and practice are combined: nothing works and no one knows why.

-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Mark Tinka Sent: Wednesday, 28 January, 2015 07:55 To: nanog@nanog.org Subject: Re: scaling linux-based router hardware recommendations

On 28/1/15 16:45, Colin Johnston wrote:

...

qnx os based router works well with powerpc, could be pushed far higher load than intel based chips

The problem being that QNX is a 32-bit kernel.

Mark.

10g transceivers are not overly expensive if you buy compatible modules. SFP+ Direct attach cable is $16. SFP+ multimode module is $18. SFP+ singlemode LR module is $48. That is nothing compared to what vendors are asking for a "real" router. I believe there are many startups that are going for 2x 10G transit with full tables. We are one of them for sure. And then you need a cheap way to handle up to 20G bidirectional traffic, because as a startup it is not a good idea to fork over what equals to a whole year of salary to Cisco or Juniper. Even if you have that kind of money, you would want to spent it on something that will get you revenue. The obvious solution is a server (or two for redundancy) running Linux or BSD. You will be getting the Intel NIC with two SFP+ slots, so you can connect a transit connection directly to each server. This works well enough. We used a setup just like that for a year, before we upgraded to a hardware router. The weak point is that it will likely have trouble if you get hit by a real big DDoS with small packets. But back to cost of things. If I use my own company as an example, we are a FTTH provider. We use PON switches with 2x 10G ports on each switch. You can get many PON switches for the price of one router with at least 4x 10G ports (equivalent to the Linux routers). The PON switches will earn you revenue, it is what you connect your customers to. Better to get a bigger network, than spend the money on a router. The cost of SFP+/XFP and GPON C+ modules on the PON switch is only about 10% of the cost of the switch itself (again using compatible modules). A switch with 24x1G and 4x 10G can be bought for $3000. You can fill it completely with optics for $300 - again about 10%. My point is that if you are in an environment where every dollar counts, you do not need to spent a majority of your funds on optics. And neither do you need that expensive router until later in the game. Regards, Baldur On 28 January 2015 at 15:35, Charles N Wyble <charles@thefnf.org> wrote:

There is no free lunch. If you want " tools that end users can just use" then buy Cisco.

Otherwise you need to roll up your sleeves and take the pieces and put them together. Or hire people like me to do it for you.

It isn't overly complicated in my opinion. Also you'll find plenty of reasonably priced Linux or BSD integration engineers out there across the globe who are used to doing this sort of thing.

Now once you move beyond basic forwarding / high PPS processing (which seems mostly commodity now) and get into say 80gbps (40gbps full duplex) IPS , ip reputation, data loss prevention, SSL MITM, AV... well that requires some very beefy hardware. Can that be done on x86? I doubt it.

Tilera seems the way to go here. Newer FPGA boards can implement various CPU architectures on the fly. You also have CUDA. I hadn't seen chelsio, I'm very excited about that. Ill have one in my grubby little hands soon enough.

transceivers are still horribly expensive. This is a major portion of the bom cost on any build, no matter what software stack is putting packets onto them.

It isn't so simple once you move beyond the 1gbps range and want full feature set. And not in one box I think. Look at https://www.bro.org/ for interesting multi box scaling.

On January 28, 2015 7:02:34 AM CST, "Paul S." <contact@winterei.se> wrote:

...

That's the problem though.

Everyone has presentations for the most part, very few actual tools that end users can just use exist.

...

...

On Jan 27, 2015, at 8:31 AM, Jim Shankland <nanog@shankland.org> wrote:

My expertise, such as it ever was, is a bit stale at this point, and my figures might be a little off. But I think the general principle applies: think about the minimum number of x86 instructions, and the minimum number of main memory accesses, to inspect a packet header, do a routing table lookup, and enqueue the packet on an outbound interface. I can't see that ever getting reduced to the point where a generic server can handle 40-byte packets at line rate (for that matter, "line rate" is increasing a lot faster than "speed of generic server" these days). Using DPDK it’s possible to do everything stated and achieve 10Gbps

On 1/28/2015 午後 08:02, Robert Bays wrote: line rate at 64byte packets on multiple interfaces simultaneously. Add ACLs to the test setup and you can reach significant portions of 10Gbps at 64byte packets and full line rate at 128bytes.

...

Check out Venky Venkatesan’s presentation at the last DPDK Summit for

interesting information on pps/CPU cycles and some of the things that can be done to optimize forwarding in a generic processor environment.

...

http://www.slideshare.net/jstleger/6-dpdk-summit-2014-intel-presentation-ven...

...

...

!DSPAM:54c8de34274511264773590!

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

- 1x ServerU Netmap L800 box in Bridge Mode for Core Firewall protection - 2x ServerU Netmap L800 boxes as BGP router (redundant) - Several Netmap L800, L100 and iXSystems servers (iXS for everything else since ServerU are only networking-centric, not high storage high processing Xeon servers)

In this setup I am running yet another not well known but very promising technology, called Netmap.

A Netmap firewall (called netmap-ipfw) was supplied from ServerU vendor, it's a slightly modified version from what you can download from Luigi Rizzo's (netmap author) public repository with multithread capabilities based on the number of queues available in the ServerU igb(4) networking card.

What it does is, IMHO, amazing for a x86 hardware: line rate firewall on 1GbE port (1.3-1.4Mpps) and line rate firewall for 10GbE port (12-14Mpps) in a system with 8 @2.4Ghz Intel Rangeley CPU.

It's not Linux DNA. It's not PF_RING. It's not Intel DPDK.

It's netmap, it's there, available, on FreeBSD base system with a number of utilities and code for reference on Rizzos' repositories. It's there, it's available and it's amazing.

This firewall has saved my sleep several times since November, dropping up to 9Mpps amplified UDP/NTP traffic on peak DDoS attack rates.

For the BGP box, I needed trunking, Q-in-Q and vlan. And sadly right now this is not available in a netmap implementation.

It means I had to keep my BGP router in the kernel path. It's funny to say this, but Netmap usually skips kernel path completely and does its job direct on the NIC, reaching backplane and bus limits directly.

ServerU people recommended me to use Chelsio Terminator 5 40G ports. OK I only needed 10G but they convinced me not to look at the bits per second numbers but the packets per seconds number.

Honestly, I don't know how Chelsio T5 did it, even though ServerU 1GbE ports perform very good on interruption CPU usage (probably this is an Intel igb(4) / ix(4) credit) but everything I route from one 40GbE port to the other port on the same L-800 expansion card, I have very, very, very LOW interrupt rates. Sometimes I have no interrupt at all!!

I peaked routing 6Mpps on ServerU L-800 and still had CPU there,

I am also a user for FreeBSD netmap-ipfw, running kipfw fwd to, say, "fwd" http traffic to a peerapp appliance. My numbers are not line rate, I peak on 900Kpps, but still have CPU idle. I had a hard time figuring out how to use netmap-ipfw, due to lack of updated documentation, but once I got it running and set up, ecerything was very straightforward with default code, no modifications, just as available. I agree FreeBSD-netmap seems more ready, with tools, toolchains and code available wheh compared to DPDK or Linux DNA. Also in the hope for further evolvings of Netmap in the base system. Numbers are impressive indeed. -- =========== Eduardo Meyer pessoal: dudu.meyer@gmail.com profissional: ddm.farmaciap@saude.gov.br

[snip] To inject science into the discussion: http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_an_ibm... And he maintains a test setup to check for performance regressions: http://bsdrp.net/documentation/examples/freebsd_performance_regression_lab Now, this is using the in-kernel stack, not netmap/pfring/etc that uses all the batching-y, stack-shallow-y implementations that the kernel currently doesn't have. But, there are people out there doing science on it and trying very hard to kick things along. The nice thing about what has come out of the DPDK related stuff is, well, the bar is set very high now. Now it's up to the open source groups to stop messing around and do something about it. If you're interested in more of this stuff, go poke Jim at pfsense/netgate. -adrian (This and RSS work is plainly in my "stuff I do for fun" category, btw.)