What options are available for a small, low cost router that has at least four 10G ports, and can handle full BGP routes? All that I know of are the Juniper MX80, and the Brocade CER line. What does Cisco and others have that compete with these two? Any other vendors besides Juniper, Brocade, and Cisco to look at?
How much is "low cost"? Mehmet
On May 19, 2015, at 10:22, Colton Conor <colton.conor@gmail.com> wrote:
What options are available for a small, low cost router that has at least four 10G ports, and can handle full BGP routes? All that I know of are the Juniper MX80, and the Brocade CER line. What does Cisco and others have that compete with these two? Any other vendors besides Juniper, Brocade, and Cisco to look at?
As low as possible, though I am not sure how low that can be. For example, I can get a MX480 used with a 4 10G card for $16K. That would easily handle my needs, but it's overkill for what we need to do. I would love a solution under 10K, but not sure if one exists. On Tue, May 19, 2015 at 12:24 PM, Mehmet Akcin <mehmet@akcin.net> wrote:
How much is "low cost"?
Mehmet
On May 19, 2015, at 10:22, Colton Conor <colton.conor@gmail.com> wrote:
What options are available for a small, low cost router that has at least four 10G ports, and can handle full BGP routes? All that I know of are the Juniper MX80, and the Brocade CER line. What does Cisco and others have that compete with these two? Any other vendors besides Juniper, Brocade, and Cisco to look at?
On 19/May/15 19:35, Colton Conor wrote:
As low as possible, though I am not sure how low that can be.
For example, I can get a MX480 used with a 4 10G card for $16K. That would easily handle my needs, but it's overkill for what we need to do.
I would love a solution under 10K, but not sure if one exists.
If you can get an MX480 with 4x 10Gbps ports at that price, I'd take it. Might seem like too much now, but when you need to grow, having that chassis will come in very handy. The problem with boxes like the MX80, MX104 and ASR9001 is while they meet what you want now, they'll struggle because expansion is fixed (how's that for an oxymoron). That US$6,000 you'll save now will be more costly when you plan the upgrades in the future. Mark.
On May 19, 2015, at 10:22, Colton Conor <colton.conor@gmail.com> wrote:
What options are available for a small, low cost router that has at least four 10G ports, and can handle full BGP routes? All that I know of are the Juniper MX80, and the Brocade CER line. What does Cisco and others have that compete with these two? Any other vendors besides Juniper, Brocade, and Cisco to look at?
I have two ServerU L-800 boxes routing BGP and OSPF, one of those has 4x10G SFP+ port and the other box, the more interesting experience I had, has a 2x40G Chelsio expansion board. Both run FreeBSD, one of the boxes run a thing called ProApps which is a FreeBSD based system ServerU people offer to their customers, with a nice and easy GUI, but essentially FreeBSD. My experience with ServerU boxes started from security needs, for high performance firewall and IDP, and recently I started to try it as router. So far, so good. The later box I started with BSDRP and later went for a default FreeBSD system. In this system I mostly run OSPF + BFD and stateles firewall, for a very critical customer site we have at Diebold. What we do in this ServerU L-800 + Chelsio card box is: - We have BIRD doing the dirty work for OSPF + BFD- We have a trigger in BIRD wich updates Chelsio T5's Forwarding Table- We have stateless firewalling handled with cxgbetool on Chelsio directly In this particular setup, with FreeBSD+BIRD+ServerUL800+Chelsio we handle every day, 4.2Mpps on 2x40G ports mostly on Chelsio ASICS, leaving most of ServerU CPU for BIRD and other FreeBSD features such as vlan, lagg, etc. Interrupt CPU usage is very low, since it's mostly handled on Chelsio board. So far I haven't tried adding a full BGP routing table to Chelsio, but the couple dozen routes we have demand this pps rate, gracefully handled by the box. It's a 1,200 USD starting cost for a very decent router which promisses to delivery a good pps and bps rate specially when compared to Mikrotik's CCR and other Cisco/Brocade routers on this same grade. Add to it a couple hundred extra bucks to have a very decent Chelsio T5 ASICS expansion to L800 chassis and you pretty much have a system that, according to Chelsion data sheet, promisses to delivery 27 milion packets per second filtered and forwarded. Pretty much Line Rate for 10G ports. I don't know about the expected 27Mpps per port, but I can confirm 4.8Mpps peaking / 4.2Mpps avging on my rack everyday, and for the price I pay on this ServerU + FreeBSD setup I can't avoid to suggest it worths pretty much a try! http://www.serveru.us/en/netmapl800 If you buy a Chelsio card or already have it, or have it at a better price (sometimes we find very good 300.00 USD deals on chelsio T5, while their list price is ~900.00 USD) talk to 'em first, they have Chelsio front expansions by default but if you buy a Chelsio x8 PCIe card your own they need to arrange ServerU L-800 to have it perfectly fitted in their L-800 chassis, and usually it requires rear raiser replacement in their router, so talk to them first... I learned it the bad way ;] bought the chelsio card myself and found out I could not use it, since this L-800 router comes with raisers for front expansions. They were gentle enough to upgrade the raiser for free but I had to ship the box back to Florida. So talk to them...
On Tuesday, May 19, 2015, Warsaw wrote:
On May 19, 2015, at 10:22, Colton Conor <colton.conor@gmail.com <javascript:;>> wrote:
What options are available for a small, low cost router that has at least four 10G ports, and can handle full BGP routes? All that I know of are the Juniper MX80, and the Brocade CER line. What does Cisco and others have that compete with these two? Any other vendors besides Juniper, Brocade, and Cisco to look at?
I have two ServerU L-800 boxes routing BGP and OSPF, one of those has 4x10G SFP+ port and the
I'm good w/ ServerU L-800 as well running BGP with FreeBSD in a location and VyOS in a couple other. I still dont know how much traffic Mr Conor needs to forward, if it's a 10G base or just needs 10G ports. Without Chelsio ASICS I route 4Gb/s on this router and I second the suggestion for L-800 if the desired forwarding rate is around 4Gbit. I didnt know Chelsio expansions could do forwarding directly on the card. just heard about its low rate of interruption requests. Sounds like it worths further investigation thanks on that.. As for L-800 I run it for over one year now doing BGP and firewalling. Great value for a twelve hundred bucks purchase.
It's a 1,200 USD starting cost for a very decent router which promisses to delivery a good pps and bps rate specially when compared to Mikrotik's CCR and other Cisco/Brocade routers on this same grade. Add to it a couple hundred extra bucks to have a very decent Chelsio T5 ASICS expansion to L800 chassis and you pretty much have a system that, according to Chelsion data sheet, promisses to delivery 27 milion packets per second filtered and forwarded. Pretty much Line Rate for 10G ports.
I don't know about the expected 27Mpps per port, but I can confirm 4.8Mpps peaking / 4.2Mpps avging on my rack everyday, and for the price I pay on this ServerU + FreeBSD setup I can't avoid to suggest it worths pretty much a try!
http://www.serveru.us/en/netmapl800
If you buy a Chelsio card or already have it, or have it at a better price (sometimes we find very good 300.00 USD deals on chelsio T5, while their list price is ~900.00 USD) talk to 'em first, they have Chelsio front expansions by default but if you buy a Chelsio x8 PCIe card your own they need to arrange ServerU L-800 to have it perfectly fitted in their L-800 chassis, and usually it requires rear raiser replacement in their router, so talk to them first... I learned it the bad way ;] bought the chelsio card myself and found out I could not use it, since this L-800 router comes with raisers for front expansions. They were gentle enough to upgrade the raiser for free but I had to ship the box back to Florida. So talk to them...
-- =========== Eduardo Meyer pessoal: dudu.meyer@gmail.com profissional: ddm.farmaciap@saude.gov.br
If you are considering Juniper, check out the MX104. There are bundles currently that give you similar capacity to an MX80 at a significantly lower price. thanks, -Randy ----- On May 19, 2015, at 1:22 PM, Colton Conor colton.conor@gmail.com wrote:
What options are available for a small, low cost router that has at least four 10G ports, and can handle full BGP routes? All that I know of are the Juniper MX80, and the Brocade CER line. What does Cisco and others have that compete with these two? Any other vendors besides Juniper, Brocade, and Cisco to look at?
Well, Hardly low cost =D ----- Alain Hebert ahebert@pubnix.net PubNIX Inc. 50 boul. St-Charles P.O. Box 26770 Beaconsfield, Quebec H9W 6G7 Tel: 514-990-5911 http://www.pubnix.net Fax: 514-990-9443 On 05/19/15 13:31, Randy Carpenter wrote:
If you are considering Juniper, check out the MX104. There are bundles currently that give you similar capacity to an MX80 at a significantly lower price.
thanks, -Randy
----- On May 19, 2015, at 1:22 PM, Colton Conor colton.conor@gmail.com wrote:
What options are available for a small, low cost router that has at least four 10G ports, and can handle full BGP routes? All that I know of are the Juniper MX80, and the Brocade CER line. What does Cisco and others have that compete with these two? Any other vendors besides Juniper, Brocade, and Cisco to look at?
Here is what I found on Google about Cisco's options: http://www.cisco.com/c/en/us/products/routers/asr-1000-series-aggregation-se... And when it comes to Juniper, you might be able to get it done with MX40 (look at their options, there are different combinations of chassis and cards), and you can always upgrade to a MX80 later. Just not sure you can find anything low cost when you need to route 10gbps. On Tue, May 19, 2015 at 12:22 PM, Colton Conor <colton.conor@gmail.com> wrote:
What options are available for a small, low cost router that has at least four 10G ports, and can handle full BGP routes? All that I know of are the Juniper MX80, and the Brocade CER line. What does Cisco and others have that compete with these two? Any other vendors besides Juniper, Brocade, and Cisco to look at?
Oops, Cisco ASR 1k series might not cut it, you can take a look at their 9k seriers: http://www.cisco.com/c/en/us/products/routers/asr-9000-series-aggregation-se... On Tue, May 19, 2015 at 12:22 PM, Colton Conor <colton.conor@gmail.com> wrote:
What options are available for a small, low cost router that has at least four 10G ports, and can handle full BGP routes? All that I know of are the Juniper MX80, and the Brocade CER line. What does Cisco and others have that compete with these two? Any other vendors besides Juniper, Brocade, and Cisco to look at?
You could potentially do it with a Vyatta 5600 or a 6Wind Turbo router running on a generic server, but I am not sure where the cost crossover is with physical hardware especially if you go with used hardware.
Colton Conor <mailto:colton.conor@gmail.com> May 19, 2015 at 10:22 AM What options are available for a small, low cost router that has at least four 10G ports, and can handle full BGP routes? All that I know of are the Juniper MX80, and the Brocade CER line. What does Cisco and others have that compete with these two? Any other vendors besides Juniper, Brocade, and Cisco to look at?
If you want virtual 10gb ports go vmware with a cisco routing vm or juniper routing vm Colin
On 19 May 2015, at 18:40, Steve Noble <snoble@sonn.com> wrote:
You could potentially do it with a Vyatta 5600 or a 6Wind Turbo router running on a generic server, but I am not sure where the cost crossover is with physical hardware especially if you go with used hardware.
Colton Conor <mailto:colton.conor@gmail.com> May 19, 2015 at 10:22 AM What options are available for a small, low cost router that has at least four 10G ports, and can handle full BGP routes? All that I know of are the Juniper MX80, and the Brocade CER line. What does Cisco and others have that compete with these two? Any other vendors besides Juniper, Brocade, and Cisco to look at?
We are using softrouters based on Supermicro chassis, E5v3 cpu, Linux/BIRD and Intel 10G NICs. And VERY happy. On 19.05.15 20:22, Colton Conor wrote:
What options are available for a small, low cost router that has at least four 10G ports, and can handle full BGP routes? All that I know of are the Juniper MX80, and the Brocade CER line. What does Cisco and others have that compete with these two? Any other vendors besides Juniper, Brocade, and Cisco to look at?
Last config I touched: 2xIntel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz, 12 Gbit summary, <5% each core load. On 19.05.15 21:06, Piotr Iwanejko wrote:
Wiadomość napisana przez Max Tulyev <maxtul@netassist.ua> w dniu 19 maj 2015, o godz. 19:58:
We are using softrouters based on Supermicro chassis, E5v3 cpu, Linux/BIRD and Intel 10G NICs. And VERY happy.
Out of curiosity, how much traffic you pass over those softrouters?
Piotr
How much of that traffic is valid legit traffic as well :( Colin
On 19 May 2015, at 19:32, Oleg A. Arkhangelsky <sysoleg@yandex.ru> wrote:
19.05.2015, 21:26, "Max Tulyev" <maxtul@netassist.ua>:
Last config I touched: 2xIntel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz, 12 Gbit summary, <5% each core load.
And what PPS rate (in+out)?
-- wbr, Oleg.
"Anarchy is about taking complete responsibility for yourself." Alan Moore.
1.4Mpps now. On 19.05.15 21:32, Oleg A. Arkhangelsky wrote:
19.05.2015, 21:26, "Max Tulyev" <maxtul@netassist.ua>:
Last config I touched: 2xIntel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz, 12 Gbit summary, <5% each core load.
And what PPS rate (in+out)?
-- wbr, Oleg.
"Anarchy is about taking complete responsibility for yourself." Alan Moore.
The running estimate is about 3 cores per 10GIf to maintain Line-Rate forwarding. The Enterprise version of Vyatte runs around 1.5-2 cores per 10Gif (Depends on how the forwarding plane is treating traffic, if you're remarking or heavy firewall rules the interrupt forwarding cost starts to impede. -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Max Tulyev Sent: Tuesday, May 19, 2015 1:24 PM To: nanog@nanog.org Subject: Re: Low Cost 10G Router Last config I touched: 2xIntel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz, 12 Gbit summary, <5% each core load. On 19.05.15 21:06, Piotr Iwanejko wrote:
Wiadomość napisana przez Max Tulyev <maxtul@netassist.ua> w dniu 19 maj 2015, o godz. 19:58:
We are using softrouters based on Supermicro chassis, E5v3 cpu, Linux/BIRD and Intel 10G NICs. And VERY happy.
Out of curiosity, how much traffic you pass over those softrouters?
Piotr
What's the application, and what traffic levels do you anticipate. Any special features like MPLS or MPLS-TE? -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Colton Conor Sent: Tuesday, May 19, 2015 12:23 PM To: NANOG Subject: Low Cost 10G Router What options are available for a small, low cost router that has at least four 10G ports, and can handle full BGP routes? All that I know of are the Juniper MX80, and the Brocade CER line. What does Cisco and others have that compete with these two? Any other vendors besides Juniper, Brocade, and Cisco to look at?
Huawei NE40E-X1-M4 I've two of these with full routes and so far (4months) they've functioned perfectly, and the price point is... inexpensive. /rh On Tue, May 19, 2015 at 10:22 AM, Colton Conor <colton.conor@gmail.com> wrote:
What options are available for a small, low cost router that has at least four 10G ports, and can handle full BGP routes? All that I know of are the Juniper MX80, and the Brocade CER line. What does Cisco and others have that compete with these two? Any other vendors besides Juniper, Brocade, and Cisco to look at?
How much does a Huawei NE40E-X1-M4 cost Richard? On Tue, May 19, 2015 at 1:09 PM, Richard Holbo <holbor@sonss.net> wrote:
Huawei NE40E-X1-M4
I've two of these with full routes and so far (4months) they've functioned perfectly, and the price point is... inexpensive.
/rh
On Tue, May 19, 2015 at 10:22 AM, Colton Conor <colton.conor@gmail.com> wrote:
What options are available for a small, low cost router that has at least four 10G ports, and can handle full BGP routes? All that I know of are the Juniper MX80, and the Brocade CER line. What does Cisco and others have that compete with these two? Any other vendors besides Juniper, Brocade, and Cisco to look at?
How cheap is cheap and what performance numbers are you looking for? About as cheap as you can get: For about $3,000 you can build a Supermicro OEM system with an 8-core Xeon E5 V3 and 4-port 10G Intel SFP+ NIC with 8G of RAM running VyOS. The pro is that BGP convergence time will be good (better than a 7200 VXR), and number of tables likely won't be a concern since RAM is cheap. The con is that you're not doing things in hardware, so you'll have higher latency, and your PPS will be lower. I haven't tried this configuration as a full router in production, but have been using them in a few places as a firewall solution and they've handled everything I've thrown their way so far. Initially, I had these in place as "low-capital" solutions that were going to be temporary so we could start building out a new environment and collect usage data to have real world sizing data for something like an ASA cluster, but they've worked so well that we've held off on that purchase for now (given challenging budget times in higher-education). The stability of VyOS has been good, and the image-based upgrade system has worked every time without issues for the past year or two (starting from 1.0.1 to the current 1.1.5). That said documentation for VyOS is poor, so you should be ready to dig into some source code or hit the IRC channel to get things running. Having a foundation with general Linux knowledge is helpful here too. If you just need a 10G link but only commit to 2-3G then this solution might be able to work well for you. If you need closer to line-rate 10G at small packet sizes then you might start running into performance limitations due to latency. If this is the case there is the Vyatta vRouter 5600 (VyOS is based on the GPL portions of the 5400), which claims to have Intel DPDK support and can handle multi-10G at line rate; but last time I checked it was really expensive ($10,000 per core or something ridiculous like that). In terms of commercial solutions, I think 10G and BGP are two things that don't combine well for "cheap". An ASR1K might do the trick, but more likely than not you're looking at an ASR9K if you want full tables; I don't have any experience with the 1K personally so I can't speak to that. The ASR 9K is a really great platform and is what we use for BGP here, but it's pretty much the opposite of cheap. As far as the firewall stuff goes, I have a draft of VyOS as a firewall that I've been wanting to put together (still needs work): http://soucy.org/vyos/UsingVyOSasaFirewall.pdf P.S. Sorry the documentation for VyOS is so bad, what's there so far in the User Guide is basically me trying to do a first pass in hopes that others would help out and there haven't been many updates. On Tue, May 19, 2015 at 1:22 PM, Colton Conor <colton.conor@gmail.com> wrote:
What options are available for a small, low cost router that has at least four 10G ports, and can handle full BGP routes? All that I know of are the Juniper MX80, and the Brocade CER line. What does Cisco and others have that compete with these two? Any other vendors besides Juniper, Brocade, and Cisco to look at?
-- Ray Patrick Soucy Network Engineer University of Maine System T: 207-561-3526 F: 207-561-3531 MaineREN, Maine's Research and Education Network www.maineren.net
How cheap is cheap and what performance numbers are you looking for?
About as cheap as you can get:
For about $3,000 you can build a Supermicro OEM system with an 8-core Xeon E5 V3 and 4-port 10G Intel SFP+ NIC with 8G of RAM running VyOS. The pro is that BGP convergence time will be good (better than a 7200 VXR), and number of tables likely won't be a concern since RAM is cheap. The con is that you're not doing things in hardware, so you'll have higher latency, and your PPS will be lower.
What 8 core Xeon E5 v3 would that be? The 26xx's are hideously pricey, and for a router, you're probably better off with something like a Supermicro X10SRn fsvo "n" with a Xeon E5-1650v3. Board is typically around $300, 1650 is around $550, so total cost I'm guessing closer to $1500-$2000 that route. The edge you get there is the higher clock on the CPU. Only six cores and only 15M cache, but 3.5GHz. The E5-2643v3 is three times the cost for very similar performance specs. Costwise, E5 single socket is the way to go unless you *need* more. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
Chat in my nerds irc channel about 10G routers paralleling this 14:21 <b> the Xeon D-1540 has 8 cores / 16 threads, 2GHz base clock with 2.6GHz turbo, and dual 10G nics on chip 14:21 <b> 45W TDP 14:31 <b> supposedly an asrock board is coming that can be 10Gbase-T or SFP+ 14:58 <a> supermicro are shipping some SFP+ 10G E5 boards 15:00 <b> but the xeon E5 doesn't have the on die 10G nic 15:07 <a> X9DRW-7TPF+ http://www.supermicro.com/products/motherboard/xeon/c600/x9drw-7tpf_.cfm Also: 1.4Mpps per 10G link doesnt seem like the minimum packetsize one wants for handling DOS attacks, but I might be bad at math. /kc On Tue, May 19, 2015 at 03:46:16PM -0500, Joe Greco said:
How cheap is cheap and what performance numbers are you looking for?
About as cheap as you can get:
For about $3,000 you can build a Supermicro OEM system with an 8-core Xeon E5 V3 and 4-port 10G Intel SFP+ NIC with 8G of RAM running VyOS. The pro is that BGP convergence time will be good (better than a 7200 VXR), and number of tables likely won't be a concern since RAM is cheap. The con is that you're not doing things in hardware, so you'll have higher latency, and your PPS will be lower.
What 8 core Xeon E5 v3 would that be? The 26xx's are hideously pricey, and for a router, you're probably better off with something like a Supermicro X10SRn fsvo "n" with a Xeon E5-1650v3. Board is typically around $300, 1650 is around $550, so total cost I'm guessing closer to $1500-$2000 that route.
The edge you get there is the higher clock on the CPU. Only six cores and only 15M cache, but 3.5GHz. The E5-2643v3 is three times the cost for very similar performance specs. Costwise, E5 single socket is the way to go unless you *need* more.
... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
-- Ken Chase - Toronto Canada
Hello! Somebody definitely should build full feature router with DPDK/netmap/pf_ring :) I have finished detailed performance tests for all of them and could achieve wire speed forwarding (with simple packet rewrite and checksum calculation) with all of they. I.e. I could process 10GE and 14.6 mpps (64byte packets) on very cheap i7 3820 with single intel X540 NIC (total cost about $ 800) with CPU 70% load. But full BGP routing is a challenge but could be implemented with existing approaches like DXR: http://info.iet.unipi.it/~luigi/papers/20120601-dxr.pdf Cheers! On Tue, May 19, 2015 at 10:11 PM, Ken Chase <math@sizone.org> wrote:
Chat in my nerds irc channel about 10G routers paralleling this
14:21 <b> the Xeon D-1540 has 8 cores / 16 threads, 2GHz base clock with 2.6GHz turbo, and dual 10G nics on chip 14:21 <b> 45W TDP 14:31 <b> supposedly an asrock board is coming that can be 10Gbase-T or SFP+ 14:58 <a> supermicro are shipping some SFP+ 10G E5 boards 15:00 <b> but the xeon E5 doesn't have the on die 10G nic 15:07 <a> X9DRW-7TPF+
http://www.supermicro.com/products/motherboard/xeon/c600/x9drw-7tpf_.cfm
Also: 1.4Mpps per 10G link doesnt seem like the minimum packetsize one wants for handling DOS attacks, but I might be bad at math.
/kc
On Tue, May 19, 2015 at 03:46:16PM -0500, Joe Greco said:
How cheap is cheap and what performance numbers are you looking for?
About as cheap as you can get:
For about $3,000 you can build a Supermicro OEM system with an 8-core Xeon E5 V3 and 4-port 10G Intel SFP+ NIC with 8G of RAM running VyOS. The pro is that BGP convergence time will be good (better than a 7200 VXR), and number of tables likely won't be a concern since RAM is cheap. The con is that you're not doing things in hardware, so you'll have higher latency, and your PPS will be lower.
What 8 core Xeon E5 v3 would that be? The 26xx's are hideously pricey, and for a router, you're probably better off with something like a Supermicro X10SRn fsvo "n" with a Xeon E5-1650v3. Board is typically around $300, 1650 is around $550, so total cost I'm guessing closer to $1500-$2000 that route.
The edge you get there is the higher clock on the CPU. Only six cores and only 15M cache, but 3.5GHz. The E5-2643v3 is three times the cost for very similar performance specs. Costwise, E5 single socket is the way to go unless you *need* more.
... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
-- Ken Chase - Toronto Canada
-- Sincerely yours, Pavel Odintsov
On 2015-05-19 14:23, Pavel Odintsov wrote:
Hello!
Somebody definitely should build full feature router with DPDK/netmap/pf_ring :)
Netmap yes. The rest no. Why? Because netmap supports libpcap, which means everything just works. Other solutions need porting. You are going along, someone mentions a neat new libpcap based tool on NANOG and you want to try it out. If you've got DPDK/pf_ring, that means you are now having to port it. That's a fair amount of effort to just eval $COOL_NEW_TOOL.
I have finished detailed performance tests for all of them and could achieve wire speed forwarding (with simple packet rewrite and checksum calculation) with all of they.
With what features applied? DPDK with a fairly full feature set (firewall rules/dynamic routing/across a vpn tunnel/doing full l7 deep packet inspection) on straight commodity (something relatively recent gen xeon something many cores) hardware on $CERTAIN_POPULAR_RTOS seems to max out ~5gbps from what my local neighborhood network testing nerds tell me. As always, your mileage will most certainly vary of course. The nice thing about commodity boxes is that you can just deploy the same "core kit" and scale it up/down (ram/cpu/redundant psu) at your favorite vendors procurement portal (oh hey $systems_purchaser , can you order a couple extra boxes with that next set of a dozen boxes your buying with this SKU and take it out of my budget? Thx). You are still going to pay a pretty decent list price for boxes that can reasonably forward AND inspect/block/modify at anything approaching line rate over say 5gbps. Then you have things like the parallela board of course with it's FPGA. And you have CUDA cards. But staffing costs for someone who has FPGA(parallel in general)/sysadmin/netadmin skills.... well that's pricy (and you'll want a couple of those in house if you do this at any kind of scale). Or you could just contract them I suppose (say at like $700.00 per hour or so?, which is what I'd charge to be a one man FPGA coding SDN slinging band since it's sort of like catching unicorns) Course you could just have your jack of all trades in house sys/net ops person and contract coding skills as needed. Don't think this will really save you money. It won't. Buy a Juniper. Seriously. (I have a 6509 in my house along with various switches/routers/wifi/voip phones (all cisco). I'm not anti cisco by any means). But they are expensive from what I hear. You get what you pay for though. What it will get you, is a very powerful and flexible solution that lets you manage at hyperscale with a unified command/control plane. It's DEVOPS 2.0 (oooo I can fire my netadmins now like I fired my sysadmins after I gave dev full prod access? COOL!) (Yes I'm being incredibly sarcastic and don't actually believe that). :) Also look at onepk from cisco. It's kinda cool if you want SDN without having to fully build your own kit.
Hello! Yep, there are no existent open source routers yet exists. But there are a lot of capabilities for this. We could just wait some time. But DPDK _definitely_ could process 64mpps and 40GE with deep inspection and processing on enough cheap E5 2670v3 chips. Yes, definitely it's ideas about good future. They can't be used now but they have really awesome outlook. On Tue, May 19, 2015 at 11:46 PM, <charles@thefnf.org> wrote:
On 2015-05-19 14:23, Pavel Odintsov wrote:
Hello!
Somebody definitely should build full feature router with DPDK/netmap/pf_ring :)
Netmap yes. The rest no. Why? Because netmap supports libpcap, which means everything just works. Other solutions need porting. You are going along, someone mentions a neat new libpcap based tool on NANOG and you want to try it out. If you've got DPDK/pf_ring, that means you are now having to port it. That's a fair amount of effort to just eval $COOL_NEW_TOOL.
I have finished detailed performance tests for all of them and could achieve wire speed forwarding (with simple packet rewrite and checksum calculation) with all of they.
With what features applied? DPDK with a fairly full feature set (firewall rules/dynamic routing/across a vpn tunnel/doing full l7 deep packet inspection) on straight commodity (something relatively recent gen xeon something many cores) hardware on $CERTAIN_POPULAR_RTOS seems to max out ~5gbps from what my local neighborhood network testing nerds tell me.
As always, your mileage will most certainly vary of course. The nice thing about commodity boxes is that you can just deploy the same "core kit" and scale it up/down (ram/cpu/redundant psu) at your favorite vendors procurement portal (oh hey $systems_purchaser , can you order a couple extra boxes with that next set of a dozen boxes your buying with this SKU and take it out of my budget? Thx).
You are still going to pay a pretty decent list price for boxes that can reasonably forward AND inspect/block/modify at anything approaching line rate over say 5gbps. Then you have things like the parallela board of course with it's FPGA. And you have CUDA cards. But staffing costs for someone who has FPGA(parallel in general)/sysadmin/netadmin skills.... well that's pricy (and you'll want a couple of those in house if you do this at any kind of scale). Or you could just contract them I suppose (say at like $700.00 per hour or so?, which is what I'd charge to be a one man FPGA coding SDN slinging band since it's sort of like catching unicorns) Course you could just have your jack of all trades in house sys/net ops person and contract coding skills as needed.
Don't think this will really save you money. It won't.
Buy a Juniper. Seriously.
(I have a 6509 in my house along with various switches/routers/wifi/voip phones (all cisco). I'm not anti cisco by any means). But they are expensive from what I hear. You get what you pay for though.
What it will get you, is a very powerful and flexible solution that lets you manage at hyperscale with a unified command/control plane. It's DEVOPS 2.0 (oooo I can fire my netadmins now like I fired my sysadmins after I gave dev full prod access? COOL!) (Yes I'm being incredibly sarcastic and don't actually believe that). :)
Also look at onepk from cisco. It's kinda cool if you want SDN without having to fully build your own kit.
-- Sincerely yours, Pavel Odintsov
I know if is not possible to have a full routing on ex3300(low memory for it) , but i never tried to do a default router on it( with EFL licence and software above version 12) I have many bgp session with cisco 3750 switchs.. Traffic about 2gb on it... Have a peer( ebgp customer) with a acx2000( i know it have 10gb port) we send to this router a default route only... And it have 1.5gb with us and more 1gb with other link provider... Enviado via iPhone Grupo Connectoway
Em 19/05/2015, às 17:59, Pavel Odintsov <pavel.odintsov@gmail.com> escreveu:
Hello!
Yep, there are no existent open source routers yet exists. But there are a lot of capabilities for this. We could just wait some time.
But DPDK _definitely_ could process 64mpps and 40GE with deep inspection and processing on enough cheap E5 2670v3 chips.
Yes, definitely it's ideas about good future. They can't be used now but they have really awesome outlook.
On Tue, May 19, 2015 at 11:46 PM, <charles@thefnf.org> wrote:
On 2015-05-19 14:23, Pavel Odintsov wrote:
Hello!
Somebody definitely should build full feature router with DPDK/netmap/pf_ring :)
Netmap yes. The rest no. Why? Because netmap supports libpcap, which means everything just works. Other solutions need porting. You are going along, someone mentions a neat new libpcap based tool on NANOG and you want to try it out. If you've got DPDK/pf_ring, that means you are now having to port it. That's a fair amount of effort to just eval $COOL_NEW_TOOL.
I have finished detailed performance tests for all of them and could achieve wire speed forwarding (with simple packet rewrite and checksum calculation) with all of they.
With what features applied? DPDK with a fairly full feature set (firewall rules/dynamic routing/across a vpn tunnel/doing full l7 deep packet inspection) on straight commodity (something relatively recent gen xeon something many cores) hardware on $CERTAIN_POPULAR_RTOS seems to max out ~5gbps from what my local neighborhood network testing nerds tell me.
As always, your mileage will most certainly vary of course. The nice thing about commodity boxes is that you can just deploy the same "core kit" and scale it up/down (ram/cpu/redundant psu) at your favorite vendors procurement portal (oh hey $systems_purchaser , can you order a couple extra boxes with that next set of a dozen boxes your buying with this SKU and take it out of my budget? Thx).
You are still going to pay a pretty decent list price for boxes that can reasonably forward AND inspect/block/modify at anything approaching line rate over say 5gbps. Then you have things like the parallela board of course with it's FPGA. And you have CUDA cards. But staffing costs for someone who has FPGA(parallel in general)/sysadmin/netadmin skills.... well that's pricy (and you'll want a couple of those in house if you do this at any kind of scale). Or you could just contract them I suppose (say at like $700.00 per hour or so?, which is what I'd charge to be a one man FPGA coding SDN slinging band since it's sort of like catching unicorns) Course you could just have your jack of all trades in house sys/net ops person and contract coding skills as needed.
Don't think this will really save you money. It won't.
Buy a Juniper. Seriously.
(I have a 6509 in my house along with various switches/routers/wifi/voip phones (all cisco). I'm not anti cisco by any means). But they are expensive from what I hear. You get what you pay for though.
What it will get you, is a very powerful and flexible solution that lets you manage at hyperscale with a unified command/control plane. It's DEVOPS 2.0 (oooo I can fire my netadmins now like I fired my sysadmins after I gave dev full prod access? COOL!) (Yes I'm being incredibly sarcastic and don't actually believe that). :)
Also look at onepk from cisco. It's kinda cool if you want SDN without having to fully build your own kit.
-- Sincerely yours, Pavel Odintsov
... This customer had a asr1002 , but have a crash on asr router and only have this acx to up your link... Its a good test... Enviado via iPhone Grupo Connectoway
Em 19/05/2015, às 18:59, Rodrigo 1telecom <rodrigo@1telecom.com.br> escreveu:
I know if is not possible to have a full routing on ex3300(low memory for it) , but i never tried to do a default router on it( with EFL licence and software above version 12) I have many bgp session with cisco 3750 switchs.. Traffic about 2gb on it... Have a peer( ebgp customer) with a acx2000( i know it have 10gb port) we send to this router a default route only... And it have 1.5gb with us and more 1gb with other link provider... Enviado via iPhone Grupo Connectoway
Em 19/05/2015, às 17:59, Pavel Odintsov <pavel.odintsov@gmail.com> escreveu:
Hello!
Yep, there are no existent open source routers yet exists. But there are a lot of capabilities for this. We could just wait some time.
But DPDK _definitely_ could process 64mpps and 40GE with deep inspection and processing on enough cheap E5 2670v3 chips.
Yes, definitely it's ideas about good future. They can't be used now but they have really awesome outlook.
On Tue, May 19, 2015 at 11:46 PM, <charles@thefnf.org> wrote: On 2015-05-19 14:23, Pavel Odintsov wrote:
Hello!
Somebody definitely should build full feature router with DPDK/netmap/pf_ring :)
Netmap yes. The rest no. Why? Because netmap supports libpcap, which means everything just works. Other solutions need porting. You are going along, someone mentions a neat new libpcap based tool on NANOG and you want to try it out. If you've got DPDK/pf_ring, that means you are now having to port it. That's a fair amount of effort to just eval $COOL_NEW_TOOL.
I have finished detailed performance tests for all of them and could achieve wire speed forwarding (with simple packet rewrite and checksum calculation) with all of they.
With what features applied? DPDK with a fairly full feature set (firewall rules/dynamic routing/across a vpn tunnel/doing full l7 deep packet inspection) on straight commodity (something relatively recent gen xeon something many cores) hardware on $CERTAIN_POPULAR_RTOS seems to max out ~5gbps from what my local neighborhood network testing nerds tell me.
As always, your mileage will most certainly vary of course. The nice thing about commodity boxes is that you can just deploy the same "core kit" and scale it up/down (ram/cpu/redundant psu) at your favorite vendors procurement portal (oh hey $systems_purchaser , can you order a couple extra boxes with that next set of a dozen boxes your buying with this SKU and take it out of my budget? Thx).
You are still going to pay a pretty decent list price for boxes that can reasonably forward AND inspect/block/modify at anything approaching line rate over say 5gbps. Then you have things like the parallela board of course with it's FPGA. And you have CUDA cards. But staffing costs for someone who has FPGA(parallel in general)/sysadmin/netadmin skills.... well that's pricy (and you'll want a couple of those in house if you do this at any kind of scale). Or you could just contract them I suppose (say at like $700.00 per hour or so?, which is what I'd charge to be a one man FPGA coding SDN slinging band since it's sort of like catching unicorns) Course you could just have your jack of all trades in house sys/net ops person and contract coding skills as needed.
Don't think this will really save you money. It won't.
Buy a Juniper. Seriously.
(I have a 6509 in my house along with various switches/routers/wifi/voip phones (all cisco). I'm not anti cisco by any means). But they are expensive from what I hear. You get what you pay for though.
What it will get you, is a very powerful and flexible solution that lets you manage at hyperscale with a unified command/control plane. It's DEVOPS 2.0 (oooo I can fire my netadmins now like I fired my sysadmins after I gave dev full prod access? COOL!) (Yes I'm being incredibly sarcastic and don't actually believe that). :)
Also look at onepk from cisco. It's kinda cool if you want SDN without having to fully build your own kit.
-- Sincerely yours, Pavel Odintsov
On 19/May/15 23:59, Rodrigo 1telecom wrote:
I know if is not possible to have a full routing on ex3300(low memory for it) , but i never tried to do a default router on it( with EFL licence and software above version 12) I have many bgp session with cisco 3750 switchs.. Traffic about 2gb on it... Have a peer( ebgp customer) with a acx2000( i know it have 10gb port) we send to this router a default route only... And it have 1.5gb with us and more 1gb with other link provider...
If you need a full table in FIB, then you're stuffed with any switch vendor out there. But if your switch vendor is able to hold the full table in RIB, and allow you to selectively hold chosen routes in FIB, then you could get away with lots of 10Gbps-capable switches at a reasonable price. Mark.
Chat in my nerds irc channel about 10G routers paralleling this
14:21 <b> the Xeon D-1540 has 8 cores / 16 threads, 2GHz base clock with 2.6GHz turbo, and dual 10G nics on chip 14:21 <b> 45W TDP
Right, but that's a pretty lame clock.
14:31 <b> supposedly an asrock board is coming that can be 10Gbase-T or SFP+
Also the only one so far I've seen able to support multiple PCIe. The Supermicro is mini-ITX. But the AsRock has some weird power arrangement too.
14:58 <a> supermicro are shipping some SFP+ 10G E5 boards 15:00 <b> but the xeon E5 doesn't have the on die 10G nic 15:07 <a> X9DRW-7TPF+
http://www.supermicro.com/products/motherboard/xeon/c600/x9drw-7tpf_.cfm
Yes, but that's a big wattsy thing. The X10SRW comes in some 1U variants that can handle two PCIe so it'd be an interesting router platform that does not eat lots of space.
Also: 1.4Mpps per 10G link doesnt seem like the minimum packetsize one wants for handling DOS attacks, but I might be bad at math.
Always an issue. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
For about $1000 you could get a Mikrotik CCR1036-8G-2S+EM but it only has 2 SFP+ ports. http://routerboard.com/CCR1036-8G-2SplusEM Keefe On 5/19/2015 3:46 PM, Joe Greco wrote:
How cheap is cheap and what performance numbers are you looking for?
About as cheap as you can get:
For about $3,000 you can build a Supermicro OEM system with an 8-core Xeon E5 V3 and 4-port 10G Intel SFP+ NIC with 8G of RAM running VyOS. The pro is that BGP convergence time will be good (better than a 7200 VXR), and number of tables likely won't be a concern since RAM is cheap. The con is that you're not doing things in hardware, so you'll have higher latency, and your PPS will be lower. What 8 core Xeon E5 v3 would that be? The 26xx's are hideously pricey, and for a router, you're probably better off with something like a Supermicro X10SRn fsvo "n" with a Xeon E5-1650v3. Board is typically around $300, 1650 is around $550, so total cost I'm guessing closer to $1500-$2000 that route.
The edge you get there is the higher clock on the CPU. Only six cores and only 15M cache, but 3.5GHz. The E5-2643v3 is three times the cost for very similar performance specs. Costwise, E5 single socket is the way to go unless you *need* more.
... JG
I second the Mikrotik recommendation. You don’t get support like you would with Cisco but it’s a solid product. Justin Justin Wilson j2sw@mtin.net http://www.mtin.net Managed Services – xISP Solutions – Data Centers http://www.thebrotherswisp.com Podcast about xISP topics http://www.midwest-ix.com Peering – Transit – Internet Exchange
On May 19, 2015, at 3:16 PM, Keefe John <keefe-af@ethoplex.com> wrote:
For about $1000 you could get a Mikrotik CCR1036-8G-2S+EM but it only has 2 SFP+ ports.
http://routerboard.com/CCR1036-8G-2SplusEM
Keefe
On 5/19/2015 3:46 PM, Joe Greco wrote:
How cheap is cheap and what performance numbers are you looking for?
About as cheap as you can get:
For about $3,000 you can build a Supermicro OEM system with an 8-core Xeon E5 V3 and 4-port 10G Intel SFP+ NIC with 8G of RAM running VyOS. The pro is that BGP convergence time will be good (better than a 7200 VXR), and number of tables likely won't be a concern since RAM is cheap. The con is that you're not doing things in hardware, so you'll have higher latency, and your PPS will be lower. What 8 core Xeon E5 v3 would that be? The 26xx's are hideously pricey, and for a router, you're probably better off with something like a Supermicro X10SRn fsvo "n" with a Xeon E5-1650v3. Board is typically around $300, 1650 is around $550, so total cost I'm guessing closer to $1500-$2000 that route.
The edge you get there is the higher clock on the CPU. Only six cores and only 15M cache, but 3.5GHz. The E5-2643v3 is three times the cost for very similar performance specs. Costwise, E5 single socket is the way to go unless you *need* more.
... JG
I've seen serious, unusual performance bottlenecks in Mikrotik CCR, in some cases not even achieving a gigabit speeds on 10G interfaces. Performance drops more rapidly then Cisco with smaller packet sizes. -mel beckman
On May 19, 2015, at 12:28 PM, Justin Wilson - MTIN <lists@mtin.net> wrote:
I second the Mikrotik recommendation. You don’t get support like you would with Cisco but it’s a solid product.
Justin
Justin Wilson j2sw@mtin.net http://www.mtin.net Managed Services – xISP Solutions – Data Centers http://www.thebrotherswisp.com Podcast about xISP topics http://www.midwest-ix.com Peering – Transit – Internet Exchange
On May 19, 2015, at 3:16 PM, Keefe John <keefe-af@ethoplex.com> wrote:
For about $1000 you could get a Mikrotik CCR1036-8G-2S+EM but it only has 2 SFP+ ports.
http://routerboard.com/CCR1036-8G-2SplusEM
Keefe
On 5/19/2015 3:46 PM, Joe Greco wrote:
How cheap is cheap and what performance numbers are you looking for?
About as cheap as you can get:
For about $3,000 you can build a Supermicro OEM system with an 8-core Xeon E5 V3 and 4-port 10G Intel SFP+ NIC with 8G of RAM running VyOS. The pro is that BGP convergence time will be good (better than a 7200 VXR), and number of tables likely won't be a concern since RAM is cheap. The con is that you're not doing things in hardware, so you'll have higher latency, and your PPS will be lower. What 8 core Xeon E5 v3 would that be? The 26xx's are hideously pricey, and for a router, you're probably better off with something like a Supermicro X10SRn fsvo "n" with a Xeon E5-1650v3. Board is typically around $300, 1650 is around $550, so total cost I'm guessing closer to $1500-$2000 that route.
The edge you get there is the higher clock on the CPU. Only six cores and only 15M cache, but 3.5GHz. The E5-2643v3 is three times the cost for very similar performance specs. Costwise, E5 single socket is the way to go unless you *need* more.
... JG
What about L3 switches? You could receive full BGP table with Linux BOX with ExaBGP, parse it and feed to L3 switch. On Tue, May 19, 2015 at 10:44 PM, Mel Beckman <mel@beckman.org> wrote:
I've seen serious, unusual performance bottlenecks in Mikrotik CCR, in some cases not even achieving a gigabit speeds on 10G interfaces. Performance drops more rapidly then Cisco with smaller packet sizes.
-mel beckman
On May 19, 2015, at 12:28 PM, Justin Wilson - MTIN <lists@mtin.net> wrote:
I second the Mikrotik recommendation. You don’t get support like you would with Cisco but it’s a solid product.
Justin
Justin Wilson j2sw@mtin.net http://www.mtin.net Managed Services – xISP Solutions – Data Centers http://www.thebrotherswisp.com Podcast about xISP topics http://www.midwest-ix.com Peering – Transit – Internet Exchange
On May 19, 2015, at 3:16 PM, Keefe John <keefe-af@ethoplex.com> wrote:
For about $1000 you could get a Mikrotik CCR1036-8G-2S+EM but it only has 2 SFP+ ports.
http://routerboard.com/CCR1036-8G-2SplusEM
Keefe
On 5/19/2015 3:46 PM, Joe Greco wrote:
How cheap is cheap and what performance numbers are you looking for?
About as cheap as you can get:
For about $3,000 you can build a Supermicro OEM system with an 8-core Xeon E5 V3 and 4-port 10G Intel SFP+ NIC with 8G of RAM running VyOS. The pro is that BGP convergence time will be good (better than a 7200 VXR), and number of tables likely won't be a concern since RAM is cheap. The con is that you're not doing things in hardware, so you'll have higher latency, and your PPS will be lower. What 8 core Xeon E5 v3 would that be? The 26xx's are hideously pricey, and for a router, you're probably better off with something like a Supermicro X10SRn fsvo "n" with a Xeon E5-1650v3. Board is typically around $300, 1650 is around $550, so total cost I'm guessing closer to $1500-$2000 that route.
The edge you get there is the higher clock on the CPU. Only six cores and only 15M cache, but 3.5GHz. The E5-2643v3 is three times the cost for very similar performance specs. Costwise, E5 single socket is the way to go unless you *need* more.
... JG
-- Sincerely yours, Pavel Odintsov
I do use L3 switches for BGP at some locations (Cisco 3750) and they perform great. The problem is no instrumentation (e.g. Sflow, netflow). -mel via cell
On May 19, 2015, at 12:55 PM, Pavel Odintsov <pavel.odintsov@gmail.com> wrote:
What about L3 switches? You could receive full BGP table with Linux BOX with ExaBGP, parse it and feed to L3 switch.
On Tue, May 19, 2015 at 10:44 PM, Mel Beckman <mel@beckman.org> wrote: I've seen serious, unusual performance bottlenecks in Mikrotik CCR, in some cases not even achieving a gigabit speeds on 10G interfaces. Performance drops more rapidly then Cisco with smaller packet sizes.
-mel beckman
On May 19, 2015, at 12:28 PM, Justin Wilson - MTIN <lists@mtin.net> wrote:
I second the Mikrotik recommendation. You don’t get support like you would with Cisco but it’s a solid product.
Justin
Justin Wilson j2sw@mtin.net http://www.mtin.net Managed Services – xISP Solutions – Data Centers http://www.thebrotherswisp.com Podcast about xISP topics http://www.midwest-ix.com Peering – Transit – Internet Exchange
On May 19, 2015, at 3:16 PM, Keefe John <keefe-af@ethoplex.com> wrote:
For about $1000 you could get a Mikrotik CCR1036-8G-2S+EM but it only has 2 SFP+ ports.
http://routerboard.com/CCR1036-8G-2SplusEM
Keefe
On 5/19/2015 3:46 PM, Joe Greco wrote:
How cheap is cheap and what performance numbers are you looking for?
About as cheap as you can get:
For about $3,000 you can build a Supermicro OEM system with an 8-core Xeon E5 V3 and 4-port 10G Intel SFP+ NIC with 8G of RAM running VyOS. The pro is that BGP convergence time will be good (better than a 7200 VXR), and number of tables likely won't be a concern since RAM is cheap. The con is that you're not doing things in hardware, so you'll have higher latency, and your PPS will be lower. What 8 core Xeon E5 v3 would that be? The 26xx's are hideously pricey, and for a router, you're probably better off with something like a Supermicro X10SRn fsvo "n" with a Xeon E5-1650v3. Board is typically around $300, 1650 is around $550, so total cost I'm guessing closer to $1500-$2000 that route.
The edge you get there is the higher clock on the CPU. Only six cores and only 15M cache, but 3.5GHz. The E5-2643v3 is three times the cost for very similar performance specs. Costwise, E5 single socket is the way to go unless you *need* more.
... JG
-- Sincerely yours, Pavel Odintsov
I've seen serious, unusual performance bottlenecks in Mikrotik CCR, in some cases not even achieving a gigabit speeds on 10G interfaces. Performance drops more rapidly then Cisco with smaller packet sizes.
-mel beckman
Folks often forget that Mikrotik ROS can also run on x86 machines..... Size your favorite hardware (server) or network appliance with appropriate ports, add MT ROS on a CF card, and you are good to go. We use i7 based network appliance with dual 10g cards (you can use a quad 10g card, such as those made by hotlav). with a 2gig of ram, you can easily do multiple (4-5 or more full bgp peers), and i7 are good for approx 1.2mill pps. Best of luck. Faisal Imtiaz Snappy Internet & Telecom
So this new $1295 Mikrotik CCR1036-8G-2S+EM has a 36 core Tilera CPU with 16GB of ram. Each core is running at 1.2Ghz? I assume that Mikrotik is multicore in software, so why does this box not outperform these intel boxes that everyone is recommending? Is it just a limitation of ports? On Tue, May 19, 2015 at 6:03 PM, Faisal Imtiaz <faisal@snappytelecom.net> wrote:
I've seen serious, unusual performance bottlenecks in Mikrotik CCR, in some cases not even achieving a gigabit speeds on 10G interfaces. Performance drops more rapidly then Cisco with smaller packet sizes.
-mel beckman
Folks often forget that Mikrotik ROS can also run on x86 machines.....
Size your favorite hardware (server) or network appliance with appropriate ports, add MT ROS on a CF card, and you are good to go.
We use i7 based network appliance with dual 10g cards (you can use a quad 10g card, such as those made by hotlav).
with a 2gig of ram, you can easily do multiple (4-5 or more full bgp peers), and i7 are good for approx 1.2mill pps.
Best of luck.
Faisal Imtiaz Snappy Internet & Telecom
The BGP daemon on the CCR routers is not multi-threaded; it only will use one core. Josh On Tue, May 19, 2015 at 10:06 PM, Colton Conor <colton.conor@gmail.com> wrote:
So this new $1295 Mikrotik CCR1036-8G-2S+EM has a 36 core Tilera CPU with 16GB of ram. Each core is running at 1.2Ghz? I assume that Mikrotik is multicore in software, so why does this box not outperform these intel boxes that everyone is recommending? Is it just a limitation of ports?
On Tue, May 19, 2015 at 6:03 PM, Faisal Imtiaz <faisal@snappytelecom.net> wrote:
I've seen serious, unusual performance bottlenecks in Mikrotik CCR, in some cases not even achieving a gigabit speeds on 10G interfaces.
Performance
drops more rapidly then Cisco with smaller packet sizes.
-mel beckman
Folks often forget that Mikrotik ROS can also run on x86 machines.....
Size your favorite hardware (server) or network appliance with appropriate ports, add MT ROS on a CF card, and you are good to go.
We use i7 based network appliance with dual 10g cards (you can use a quad 10g card, such as those made by hotlav).
with a 2gig of ram, you can easily do multiple (4-5 or more full bgp peers), and i7 are good for approx 1.2mill pps.
Best of luck.
Faisal Imtiaz Snappy Internet & Telecom
So are the rest of the processes in Mikrotik OS multi threaded? I would hope so to take advantage of 36 cores! What is up with all of these network vendors not supporting more than one core in their OS? I just don't get it. On Tue, May 19, 2015 at 9:49 PM, Josh Baird <joshbaird@gmail.com> wrote:
The BGP daemon on the CCR routers is not multi-threaded; it only will use one core.
Josh
On Tue, May 19, 2015 at 10:06 PM, Colton Conor <colton.conor@gmail.com> wrote:
So this new $1295 Mikrotik CCR1036-8G-2S+EM has a 36 core Tilera CPU with 16GB of ram. Each core is running at 1.2Ghz? I assume that Mikrotik is multicore in software, so why does this box not outperform these intel boxes that everyone is recommending? Is it just a limitation of ports?
On Tue, May 19, 2015 at 6:03 PM, Faisal Imtiaz <faisal@snappytelecom.net> wrote:
I've seen serious, unusual performance bottlenecks in Mikrotik CCR, in some cases not even achieving a gigabit speeds on 10G interfaces.
Performance
drops more rapidly then Cisco with smaller packet sizes.
-mel beckman
Folks often forget that Mikrotik ROS can also run on x86 machines.....
Size your favorite hardware (server) or network appliance with appropriate ports, add MT ROS on a CF card, and you are good to go.
We use i7 based network appliance with dual 10g cards (you can use a quad 10g card, such as those made by hotlav).
with a 2gig of ram, you can easily do multiple (4-5 or more full bgp peers), and i7 are good for approx 1.2mill pps.
Best of luck.
Faisal Imtiaz Snappy Internet & Telecom
Since you are considering multiple options, I'd build a decision matrix. You can put down all the requirements, score each option, and then normalize it to give each a final score. After that you can calculate some other things such as throughput per dollar, etc. http://asq.org/learn-about-quality/decision-making-tools/overview/decision-m... Regarding the Mikrotik, there's a difference between Multithreading and Multiprocessing. On Wed, May 20, 2015 at 11:44 AM, Colton Conor <colton.conor@gmail.com> wrote:
So are the rest of the processes in Mikrotik OS multi threaded? I would hope so to take advantage of 36 cores!
What is up with all of these network vendors not supporting more than one core in their OS? I just don't get it.
On Tue, May 19, 2015 at 9:49 PM, Josh Baird <joshbaird@gmail.com> wrote:
The BGP daemon on the CCR routers is not multi-threaded; it only will use one core.
Josh
On Tue, May 19, 2015 at 10:06 PM, Colton Conor <colton.conor@gmail.com> wrote:
So this new $1295 Mikrotik CCR1036-8G-2S+EM has a 36 core Tilera CPU with 16GB of ram. Each core is running at 1.2Ghz? I assume that Mikrotik is multicore in software, so why does this box not outperform these intel boxes that everyone is recommending? Is it just a limitation of ports?
On Tue, May 19, 2015 at 6:03 PM, Faisal Imtiaz < faisal@snappytelecom.net> wrote:
I've seen serious, unusual performance bottlenecks in Mikrotik CCR,
in
some
cases not even achieving a gigabit speeds on 10G interfaces. Performance drops more rapidly then Cisco with smaller packet sizes.
-mel beckman
Folks often forget that Mikrotik ROS can also run on x86 machines.....
Size your favorite hardware (server) or network appliance with appropriate ports, add MT ROS on a CF card, and you are good to go.
We use i7 based network appliance with dual 10g cards (you can use a quad 10g card, such as those made by hotlav).
with a 2gig of ram, you can easily do multiple (4-5 or more full bgp peers), and i7 are good for approx 1.2mill pps.
Best of luck.
Faisal Imtiaz Snappy Internet & Telecom
On 20 May 2015 at 17:44, Colton Conor <colton.conor@gmail.com> wrote:
So are the rest of the processes in Mikrotik OS multi threaded? I would hope so to take advantage of 36 cores!
The forthcoming new major software release from Mikrotik apparently will have multi-threaded BGP - it is targetted at their (also forthcoming) 72 core 8x10GE router, the CCR1072 I would treat this as speculation until you can order it though - it's been "promised" for 18 months now. Aled
There will *not* be multi-threaded BGP in RouterOS. I was going to refer you to the post I made last night, but due to the unique way the e-mail list is setup, I replied directly to Colton instead of the list. I resent it again to the list. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "Aled Morris" <aledm@qix.co.uk> To: "Colton Conor" <colton.conor@gmail.com> Cc: "North American Network Operators Group" <nanog@nanog.org> Sent: Wednesday, May 20, 2015 11:59:04 AM Subject: Re: Low Cost 10G Router On 20 May 2015 at 17:44, Colton Conor <colton.conor@gmail.com> wrote:
So are the rest of the processes in Mikrotik OS multi threaded? I would hope so to take advantage of 36 cores!
The forthcoming new major software release from Mikrotik apparently will have multi-threaded BGP - it is targetted at their (also forthcoming) 72 core 8x10GE router, the CCR1072 I would treat this as speculation until you can order it though - it's been "promised" for 18 months now. Aled
Well, the cores on a many-core CPU aren't going to have the "torque" that a Xeon would. They're also still working on the software. It has gotten a ton better over the life of the CCRs thus far. BGP is still atrocious on the CCRs, but that's because the route update process isn't multithreaded. It won't be multithreaded in the next major version either, but they will have done some programming voodoo (all programming is voodoo to me) to reign in the poor performance issues with full tables. https://youtu.be/ihZiAC-Rox8?t=37m8s ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "Colton Conor" <colton.conor@gmail.com> To: "Faisal Imtiaz" <faisal@snappytelecom.net> Cc: "North American Network Operators Group" <nanog@nanog.org> Sent: Tuesday, May 19, 2015 9:06:26 PM Subject: Re: Low Cost 10G Router So this new $1295 Mikrotik CCR1036-8G-2S+EM has a 36 core Tilera CPU with 16GB of ram. Each core is running at 1.2Ghz? I assume that Mikrotik is multicore in software, so why does this box not outperform these intel boxes that everyone is recommending? Is it just a limitation of ports? On Tue, May 19, 2015 at 6:03 PM, Faisal Imtiaz <faisal@snappytelecom.net> wrote:
I've seen serious, unusual performance bottlenecks in Mikrotik CCR, in some cases not even achieving a gigabit speeds on 10G interfaces. Performance drops more rapidly then Cisco with smaller packet sizes.
-mel beckman
Folks often forget that Mikrotik ROS can also run on x86 machines.....
Size your favorite hardware (server) or network appliance with appropriate ports, add MT ROS on a CF card, and you are good to go.
We use i7 based network appliance with dual 10g cards (you can use a quad 10g card, such as those made by hotlav).
with a 2gig of ram, you can easily do multiple (4-5 or more full bgp peers), and i7 are good for approx 1.2mill pps.
Best of luck.
Faisal Imtiaz Snappy Internet & Telecom
On Wed, May 20, 2015 at 2:07 PM, Mike Hammett <nanog@ics-il.net> wrote:
Well, the cores on a many-core CPU aren't going to have the "torque" that a Xeon would. They're also still working on the software. It has gotten a ton better over the life of the CCRs thus far. BGP is still atrocious on the CCRs, but that's because the route update process isn't multithreaded. It won't be multithreaded in the next major version either, but they will have done some programming voodoo (all programming is voodoo to me) to reign in the poor performance issues with full tables.
I honestly don't know why most people gets impressed by the number of Tylera cores on CCR and think it's a good thing. Your "torque" point makes much sense to me. A few cores with decent clock and Xeon or Rangeley "torque" is just better. Adding that much weak tylera cores with low clock only results in much more context switching, much more CPU Affinity needs. Multithreading the relevant grained bit of code will also lead to more context switching, but for threads now instead of processes. As I understand the architecture of those solutions, I don't see why a bgp daemon mono threaded is a problem. Ok, multithreaded would give a better full routing convergence. But once the routing table is loaded it does not matter how many threads the bgp process will use. The dirty work on Linux (RouterOS kernel for that matter) will be done on the forward information table, on the packet forwarding code and specially on softirq (interrupt requests). This is where the bottleneck seems to be, IMHO. Linux is not good at multithreaded packet forwarding and not good specially at handling interrupt requests on multi-queue NICs. So, RouterOS is not good as well. Therefore that "several dozens" cheap and weak tylera cores powering CCR boxes is absolutely not friendly for Linux core and RouterOS itself. I'm better served off with a smaller amount of cores with better clock and better "torque" as Mr Hammett mentioned (I liked the expression usage yes) and that's why a Linux or a BSD box with a couple Xeon CPUs will perform better than CCR. Sometimes as someone mentioned a couple i7 cores will outperform a CCR box as well. More torque, yeah. Less context switching and time sharing wasted. However this horizontal scalar number of tylera cores on the CCR is good for marketing. After all "you are buying a 36 CPU box" paying "a couple hundred bucks". Impressive, hum? Well not for me.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest Internet Exchange http://www.midwest-ix.com
----- Original Message -----
From: "Colton Conor" <colton.conor@gmail.com> To: "Faisal Imtiaz" <faisal@snappytelecom.net> Cc: "North American Network Operators Group" <nanog@nanog.org> Sent: Tuesday, May 19, 2015 9:06:26 PM Subject: Re: Low Cost 10G Router
So this new $1295 Mikrotik CCR1036-8G-2S+EM has a 36 core Tilera CPU with 16GB of ram. Each core is running at 1.2Ghz? I assume that Mikrotik is multicore in software, so why does this box not outperform these intel boxes that everyone is recommending? Is it just a limitation of ports?
On Tue, May 19, 2015 at 6:03 PM, Faisal Imtiaz <faisal@snappytelecom.net> wrote:
I've seen serious, unusual performance bottlenecks in Mikrotik CCR, in some cases not even achieving a gigabit speeds on 10G interfaces.
Performance
drops more rapidly then Cisco with smaller packet sizes.
-mel beckman
Folks often forget that Mikrotik ROS can also run on x86 machines.....
Size your favorite hardware (server) or network appliance with appropriate ports, add MT ROS on a CF card, and you are good to go.
We use i7 based network appliance with dual 10g cards (you can use a quad 10g card, such as those made by hotlav).
with a 2gig of ram, you can easily do multiple (4-5 or more full bgp peers), and i7 are good for approx 1.2mill pps.
Best of luck.
Faisal Imtiaz Snappy Internet & Telecom
Well said Eddie, It would be worth pointing out that on CCR's each port also has a core dedicated to it, a benefit of such a design is that each port is able to handle a much higher PPS rate, and if there is a DDOS attack on one port, it will not bring down the rest of the ports / router etc. (disclaimer, if the router is setup properly, without all traffic going thru the CPU etc etc). Faisal Imtiaz Snappy Internet & Telecom ----- Original Message -----
From: "Eddie Tardist" <edtardist@gmail.com> To: "North American Network Operators Group" <nanog@nanog.org> Sent: Wednesday, May 20, 2015 6:34:11 PM Subject: Re: Low Cost 10G Router
On Wed, May 20, 2015 at 2:07 PM, Mike Hammett <nanog@ics-il.net> wrote:
Well, the cores on a many-core CPU aren't going to have the "torque" that a Xeon would. They're also still working on the software. It has gotten a ton better over the life of the CCRs thus far. BGP is still atrocious on the CCRs, but that's because the route update process isn't multithreaded. It won't be multithreaded in the next major version either, but they will have done some programming voodoo (all programming is voodoo to me) to reign in the poor performance issues with full tables.
I honestly don't know why most people gets impressed by the number of Tylera cores on CCR and think it's a good thing. Your "torque" point makes much sense to me. A few cores with decent clock and Xeon or Rangeley "torque" is just better. Adding that much weak tylera cores with low clock only results in much more context switching, much more CPU Affinity needs.
Multithreading the relevant grained bit of code will also lead to more context switching, but for threads now instead of processes.
As I understand the architecture of those solutions, I don't see why a bgp daemon mono threaded is a problem. Ok, multithreaded would give a better full routing convergence. But once the routing table is loaded it does not matter how many threads the bgp process will use. The dirty work on Linux (RouterOS kernel for that matter) will be done on the forward information table, on the packet forwarding code and specially on softirq (interrupt requests). This is where the bottleneck seems to be, IMHO. Linux is not good at multithreaded packet forwarding and not good specially at handling interrupt requests on multi-queue NICs. So, RouterOS is not good as well.
Therefore that "several dozens" cheap and weak tylera cores powering CCR boxes is absolutely not friendly for Linux core and RouterOS itself.
I'm better served off with a smaller amount of cores with better clock and better "torque" as Mr Hammett mentioned (I liked the expression usage yes) and that's why a Linux or a BSD box with a couple Xeon CPUs will perform better than CCR. Sometimes as someone mentioned a couple i7 cores will outperform a CCR box as well. More torque, yeah. Less context switching and time sharing wasted.
However this horizontal scalar number of tylera cores on the CCR is good for marketing. After all "you are buying a 36 CPU box" paying "a couple hundred bucks". Impressive, hum? Well not for me.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest Internet Exchange http://www.midwest-ix.com
----- Original Message -----
From: "Colton Conor" <colton.conor@gmail.com> To: "Faisal Imtiaz" <faisal@snappytelecom.net> Cc: "North American Network Operators Group" <nanog@nanog.org> Sent: Tuesday, May 19, 2015 9:06:26 PM Subject: Re: Low Cost 10G Router
So this new $1295 Mikrotik CCR1036-8G-2S+EM has a 36 core Tilera CPU with 16GB of ram. Each core is running at 1.2Ghz? I assume that Mikrotik is multicore in software, so why does this box not outperform these intel boxes that everyone is recommending? Is it just a limitation of ports?
On Tue, May 19, 2015 at 6:03 PM, Faisal Imtiaz <faisal@snappytelecom.net> wrote:
I've seen serious, unusual performance bottlenecks in Mikrotik CCR, in some cases not even achieving a gigabit speeds on 10G interfaces.
Performance
drops more rapidly then Cisco with smaller packet sizes.
-mel beckman
Folks often forget that Mikrotik ROS can also run on x86 machines.....
Size your favorite hardware (server) or network appliance with appropriate ports, add MT ROS on a CF card, and you are good to go.
We use i7 based network appliance with dual 10g cards (you can use a quad 10g card, such as those made by hotlav).
with a 2gig of ram, you can easily do multiple (4-5 or more full bgp peers), and i7 are good for approx 1.2mill pps.
Best of luck.
Faisal Imtiaz Snappy Internet & Telecom
> BGP is still atrocious on the CCRs, but that's because the route > update process isn't multithreaded. I recently took a close look at this, and that the update process is single-threaded is not the major problem so long as churn is not too great. The problem is that due to a deeper problem the entire forwarding table needs to be recalculated for *each* update. This means that even with the usual background noise in the DFZ the daemon is constantly updating everything. There are other bugs as well such as not supporting recursive next hop (e.g. via OSPF) lookup for IPv6 which means that if you have any iBGP sessions and more than one internal path you're out of luck with no obvious workaround. The stock answer from Mikrotik is that "everything will be fixed in the next major release of the OS". When that happens, and how long it takes to shake out the inevitable new bugs is an open question. Personally I give it at least a year before we would even try to use these seriously for BGP. Until then, it's FreeBSD and BIRD. Best, -w -- William Waites <wwaites@tardis.ed.ac.uk> | School of Informatics http://tardis.ed.ac.uk/~wwaites/ | University of Edinburgh https://hubs.net.uk/ | HUBS AS60241 The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
2015-05-19 16:16 GMT-03:00 Keefe John <keefe-af@ethoplex.com>:
For about $1000 you could get a Mikrotik CCR1036-8G-2S+EM but it only has 2 SFP+ ports.
Run away from Mikrotik, especially if you want to run BGP. -- Eduardo Schoedler
Microtik CCR have a huge issues in case of DDOS: http://forum.mikrotik.com/viewtopic.php?t=92728 On Tue, May 19, 2015 at 10:37 PM, Eduardo Schoedler <listas@esds.com.br> wrote:
2015-05-19 16:16 GMT-03:00 Keefe John <keefe-af@ethoplex.com>:
For about $1000 you could get a Mikrotik CCR1036-8G-2S+EM but it only has 2 SFP+ ports.
Run away from Mikrotik, especially if you want to run BGP.
-- Eduardo Schoedler
-- Sincerely yours, Pavel Odintsov
You're right I dropped down to the v2 for pricing reasons: - Supermicro SuperServer 5017R-MTRF - 4x SATA - 8x DDR3 - 400W Redundant - Eight-Core Intel Xeon Processor E5-2640 v2 2.00GHz 20MB Cache (95W) - 4 x SAMSUNG 2GB PC3-12800 DDR3-160 - 2 x 500GB SATA 6.0Gb/s 7200RPM - 3.5" - Western Digital RE4 WD5003ABYZ - Supermicro System Cabinet Front Bezel CSE-PTFB-813B with Lock and Filter (Black) - No Windows Operating System (Hardware Warranty Only, No Software Support) - Three Year Warranty with Advanced Parts Replacement FWIW I used Sourcecode as the system builder. They've been great to work with. On Tue, May 19, 2015 at 4:46 PM, Joe Greco <jgreco@ns.sol.net> wrote:
How cheap is cheap and what performance numbers are you looking for?
About as cheap as you can get:
For about $3,000 you can build a Supermicro OEM system with an 8-core Xeon E5 V3 and 4-port 10G Intel SFP+ NIC with 8G of RAM running VyOS. The pro is that BGP convergence time will be good (better than a 7200 VXR), and number of tables likely won't be a concern since RAM is cheap. The con is that you're not doing things in hardware, so you'll have higher latency, and your PPS will be lower.
What 8 core Xeon E5 v3 would that be? The 26xx's are hideously pricey, and for a router, you're probably better off with something like a Supermicro X10SRn fsvo "n" with a Xeon E5-1650v3. Board is typically around $300, 1650 is around $550, so total cost I'm guessing closer to $1500-$2000 that route.
The edge you get there is the higher clock on the CPU. Only six cores and only 15M cache, but 3.5GHz. The E5-2643v3 is three times the cost for very similar performance specs. Costwise, E5 single socket is the way to go unless you *need* more.
... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
-- Ray Patrick Soucy Network Engineer University of Maine System T: 207-561-3526 F: 207-561-3531 MaineREN, Maine's Research and Education Network www.maineren.net
P.S I went through HotLava Systems for the Intel-based SFP+ NICs to add to those, http://hotlavasystems.com/ (not trying to plug; these are just hard to find) On Wed, May 20, 2015 at 9:08 AM, Ray Soucy <rps@maine.edu> wrote:
You're right I dropped down to the v2 for pricing reasons:
- Supermicro SuperServer 5017R-MTRF - 4x SATA - 8x DDR3 - 400W Redundant - Eight-Core Intel Xeon Processor E5-2640 v2 2.00GHz 20MB Cache (95W) - 4 x SAMSUNG 2GB PC3-12800 DDR3-160 - 2 x 500GB SATA 6.0Gb/s 7200RPM - 3.5" - Western Digital RE4 WD5003ABYZ - Supermicro System Cabinet Front Bezel CSE-PTFB-813B with Lock and Filter (Black) - No Windows Operating System (Hardware Warranty Only, No Software Support) - Three Year Warranty with Advanced Parts Replacement
FWIW I used Sourcecode as the system builder. They've been great to work with.
On Tue, May 19, 2015 at 4:46 PM, Joe Greco <jgreco@ns.sol.net> wrote:
How cheap is cheap and what performance numbers are you looking for?
About as cheap as you can get:
For about $3,000 you can build a Supermicro OEM system with an 8-core Xeon E5 V3 and 4-port 10G Intel SFP+ NIC with 8G of RAM running VyOS. The pro is that BGP convergence time will be good (better than a 7200 VXR), and number of tables likely won't be a concern since RAM is cheap. The con is that you're not doing things in hardware, so you'll have higher latency, and your PPS will be lower.
What 8 core Xeon E5 v3 would that be? The 26xx's are hideously pricey, and for a router, you're probably better off with something like a Supermicro X10SRn fsvo "n" with a Xeon E5-1650v3. Board is typically around $300, 1650 is around $550, so total cost I'm guessing closer to $1500-$2000 that route.
The edge you get there is the higher clock on the CPU. Only six cores and only 15M cache, but 3.5GHz. The E5-2643v3 is three times the cost for very similar performance specs. Costwise, E5 single socket is the way to go unless you *need* more.
... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
-- Ray Patrick Soucy Network Engineer University of Maine System
T: 207-561-3526 F: 207-561-3531
MaineREN, Maine's Research and Education Network www.maineren.net
-- Ray Patrick Soucy Network Engineer University of Maine System T: 207-561-3526 F: 207-561-3531 MaineREN, Maine's Research and Education Network www.maineren.net
Hello! Ray, I could suggest switch from multi physical CPU configuration to single. Like Intel Xeon E5-1650/1660/1680 or even Xeon E3 platforms. Because multi processor systems need really huge amount of knowledge for NUMA configuration and PCI-E devices assignment for each NUMA. Secondly, I could vote many times for Supermicro! :) Dell or HP are really ugly systems for soft routers. CPU frequency tuning, PCM debugging are real nightmare on this systems. Please beware of they! Supermicro is very clear and do not block useful functions of platform. On Wed, May 20, 2015 at 4:08 PM, Ray Soucy <rps@maine.edu> wrote:
You're right I dropped down to the v2 for pricing reasons:
- Supermicro SuperServer 5017R-MTRF - 4x SATA - 8x DDR3 - 400W Redundant - Eight-Core Intel Xeon Processor E5-2640 v2 2.00GHz 20MB Cache (95W) - 4 x SAMSUNG 2GB PC3-12800 DDR3-160 - 2 x 500GB SATA 6.0Gb/s 7200RPM - 3.5" - Western Digital RE4 WD5003ABYZ - Supermicro System Cabinet Front Bezel CSE-PTFB-813B with Lock and Filter (Black) - No Windows Operating System (Hardware Warranty Only, No Software Support) - Three Year Warranty with Advanced Parts Replacement
FWIW I used Sourcecode as the system builder. They've been great to work with.
On Tue, May 19, 2015 at 4:46 PM, Joe Greco <jgreco@ns.sol.net> wrote:
How cheap is cheap and what performance numbers are you looking for?
About as cheap as you can get:
For about $3,000 you can build a Supermicro OEM system with an 8-core Xeon E5 V3 and 4-port 10G Intel SFP+ NIC with 8G of RAM running VyOS. The pro is that BGP convergence time will be good (better than a 7200 VXR), and number of tables likely won't be a concern since RAM is cheap. The con is that you're not doing things in hardware, so you'll have higher latency, and your PPS will be lower.
What 8 core Xeon E5 v3 would that be? The 26xx's are hideously pricey, and for a router, you're probably better off with something like a Supermicro X10SRn fsvo "n" with a Xeon E5-1650v3. Board is typically around $300, 1650 is around $550, so total cost I'm guessing closer to $1500-$2000 that route.
The edge you get there is the higher clock on the CPU. Only six cores and only 15M cache, but 3.5GHz. The E5-2643v3 is three times the cost for very similar performance specs. Costwise, E5 single socket is the way to go unless you *need* more.
... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
-- Ray Patrick Soucy Network Engineer University of Maine System
T: 207-561-3526 F: 207-561-3531
MaineREN, Maine's Research and Education Network www.maineren.net
-- Sincerely yours, Pavel Odintsov
On 2015-05-20 08:17, Pavel Odintsov wrote:
Hello!
Ray, I could suggest switch from multi physical CPU configuration to single. Like Intel Xeon E5-1650/1660/1680 or even Xeon E3 platforms. Because multi processor systems need really huge amount of knowledge for NUMA configuration and PCI-E devices assignment for each NUMA.
Not really. Well that's opinion I suppose. It didn't seem like that steep of a learning curve. Just need to play with taskset and do some reading. If you are just starting out and experimenting, then sure a single CPU system would probably be the way to go.
Secondly, I could vote many times for Supermicro! :) Dell or HP are really ugly systems for soft routers. CPU frequency tuning, PCM debugging are real nightmare on this systems.
And why is that any different on a supermicro system? Isn't it all the same hardware? I personally would recommend buying from Dell or HP, as they things like 4hr turn around times (at least in the major urban centers, usually it's about an hour). I don't know how good Supermicro purchase/procurement system is. Dell has some neat things for asset management, support etc. HP probably has the same. Please beware of they!
Supermicro is very clear and do not block useful functions of platform.
What don't they "block"? What vendors block things, and what things do they block?
On 19/May/15 20:46, Ray Soucy wrote:
An ASR1K might do the trick, but more likely than not you're looking at an ASR9K if you want full tables; I don't have any experience with the 1K personally so I can't speak to that. The ASR 9K is a really great platform and is what we use for BGP here, but it's pretty much the opposite of cheap.
The ASR1000 is a very good box, but I tend to prefer them for low-speed services, which are generally non-Ethernet in nature, e.g., downstream customers coming in via SDH. They do support 10Gbps ports, but that is a 1-port SPA; and the most you can have in today's SIP's (carrier cards) would be 4x 1-port SPA's. So not very dense. Their forwarding planes start at 2.5Gbps (fixed) all the way to 200Gbps (13-slot chassis). But you're more likely to run out of high-speed ports before you stress a 200Gbps forwarding plane on that chassis. So if the applications are purely Ethernet, I'd not consider the ASR1000. But if there is a mix-and-match for Ethernet and non-Ethernet ports, it's the perfect box. That and the MX104. Mark.
For the lists benefit, there is a 6 X 10GBE option for the ASR1000 series it seems. No idea on pricing though. http://www.cisco.com/c/en/us/products/collateral/application-networking-serv... Cheers, Mark On Wed, May 20, 2015 at 3:59 PM, Mark Tinka <mark.tinka@seacom.mu> wrote:
On 19/May/15 20:46, Ray Soucy wrote:
An ASR1K might do the trick, but more likely than not you're looking at an ASR9K if you want full tables; I don't have any experience with the 1K personally so I can't speak to that. The ASR 9K is a really great platform and is what we use for BGP here, but it's pretty much the opposite of cheap.
The ASR1000 is a very good box, but I tend to prefer them for low-speed services, which are generally non-Ethernet in nature, e.g., downstream customers coming in via SDH.
They do support 10Gbps ports, but that is a 1-port SPA; and the most you can have in today's SIP's (carrier cards) would be 4x 1-port SPA's. So not very dense.
Their forwarding planes start at 2.5Gbps (fixed) all the way to 200Gbps (13-slot chassis). But you're more likely to run out of high-speed ports before you stress a 200Gbps forwarding plane on that chassis.
So if the applications are purely Ethernet, I'd not consider the ASR1000. But if there is a mix-and-match for Ethernet and non-Ethernet ports, it's the perfect box. That and the MX104.
Mark.
-- Regards, Mark L. Tees
ASR1K (XE) has great BGP implementation, go for it if you are OK with density/throughput. Regards, Jeff
On May 19, 2015, at 11:35 PM, Mark Tees <marktees@gmail.com> wrote:
For the lists benefit, there is a 6 X 10GBE option for the ASR1000 series it seems. No idea on pricing though.
http://www.cisco.com/c/en/us/products/collateral/application-networking-serv...
Cheers,
Mark
On Wed, May 20, 2015 at 3:59 PM, Mark Tinka <mark.tinka@seacom.mu> wrote:
On 19/May/15 20:46, Ray Soucy wrote:
An ASR1K might do the trick, but more likely than not you're looking at an ASR9K if you want full tables; I don't have any experience with the 1K personally so I can't speak to that. The ASR 9K is a really great platform and is what we use for BGP here, but it's pretty much the opposite of cheap.
The ASR1000 is a very good box, but I tend to prefer them for low-speed services, which are generally non-Ethernet in nature, e.g., downstream customers coming in via SDH.
They do support 10Gbps ports, but that is a 1-port SPA; and the most you can have in today's SIP's (carrier cards) would be 4x 1-port SPA's. So not very dense.
Their forwarding planes start at 2.5Gbps (fixed) all the way to 200Gbps (13-slot chassis). But you're more likely to run out of high-speed ports before you stress a 200Gbps forwarding plane on that chassis.
So if the applications are purely Ethernet, I'd not consider the ASR1000. But if there is a mix-and-match for Ethernet and non-Ethernet ports, it's the perfect box. That and the MX104.
Mark.
-- Regards,
Mark L. Tees
You can save a ton if you drop the requirement for full routes. Ask for a simple default route and then calculate your most used routes offline and upload that daily to the switch. I believe if you have just a few thousand routes, your outbound will be nearly the same as with full routes. Your inbound will be exactly the same, as even the smallest device can announce your prefixes. PS. ZTE has a ZXR 8900e switch with 8x 10g with 1 million routes for less than 10k USD. A ZTE 59e switch with 4x 10g with 30k routes is about 3k USD. Regards, Baldur
I haven't tried myself but some of the stuff Cumulus Linux is doing is pretty amazing, not certain quagga can or should handle full bgp table but you could probably get a Penguin 10gbe for less than 8k. On Tue, May 19, 2015, 10:25 AM Colton Conor <colton.conor@gmail.com> wrote:
What options are available for a small, low cost router that has at least four 10G ports, and can handle full BGP routes? All that I know of are the Juniper MX80, and the Brocade CER line. What does Cisco and others have that compete with these two? Any other vendors besides Juniper, Brocade, and Cisco to look at?
On 20/05/2015 14:32, Cody Grosskopf wrote:
I haven't tried myself but some of the stuff Cumulus Linux is doing is pretty amazing, not certain quagga can or should handle full bgp table but you could probably get a Penguin 10gbe for less than 8k.
quagga (or whatever RIB manager you want, e.g. bird) isn't the issue. The issue is that these switches have limited hardware FIB capacity and if you attempt to put a full table on them, they won't accept it. Nick
We could cut full BGP and select only important prefixes with ExaBGP. On Wed, May 20, 2015 at 4:41 PM, Nick Hilliard <nick@foobar.org> wrote:
On 20/05/2015 14:32, Cody Grosskopf wrote:
I haven't tried myself but some of the stuff Cumulus Linux is doing is pretty amazing, not certain quagga can or should handle full bgp table but you could probably get a Penguin 10gbe for less than 8k.
quagga (or whatever RIB manager you want, e.g. bird) isn't the issue. The issue is that these switches have limited hardware FIB capacity and if you attempt to put a full table on them, they won't accept it.
Nick
-- Sincerely yours, Pavel Odintsov
Yes, right! But ExaBGP could receive full BGP table, drop some rules and reflect they to Quagga which could load FIB on the Cumulus. On Wed, May 20, 2015 at 4:53 PM, Nick Hilliard <nick@foobar.org> wrote:
On 20/05/2015 14:46, Pavel Odintsov wrote:
We could cut full BGP and select only important prefixes with ExaBGP.
exabgp is rib mgmt only and doesn't program the fib. you will need quagga / bird / etc for this.
Nick
-- Sincerely yours, Pavel Odintsov
On 20/05/2015 14:56, Pavel Odintsov wrote:
Yes, right! But ExaBGP could receive full BGP table, drop some rules and reflect they to Quagga which could load FIB on the Cumulus.
or you could not bother with exabgp and do your route filtering on quagga. Nothing wrong with exabgp, btw. Great product. It's just the wrong tool for the job here. Nick
Yes, you could do filtering with Quagga. But Quagga is pretty old tool without multiple dynamic features. But with ExaBGP you could do really any significant route table transformations with Python in few lines of code. But it's definitely add additional point of failure/bug. On Wed, May 20, 2015 at 4:57 PM, Nick Hilliard <nick@foobar.org> wrote:
On 20/05/2015 14:56, Pavel Odintsov wrote:
Yes, right! But ExaBGP could receive full BGP table, drop some rules and reflect they to Quagga which could load FIB on the Cumulus.
or you could not bother with exabgp and do your route filtering on quagga.
Nothing wrong with exabgp, btw. Great product. It's just the wrong tool for the job here.
Nick
-- Sincerely yours, Pavel Odintsov
On 20 May 2015 at 15:00, Pavel Odintsov <pavel.odintsov@gmail.com> wrote:
Yes, you could do filtering with Quagga. But Quagga is pretty old tool without multiple dynamic features. But with ExaBGP you could do really any significant route table transformations with Python in few lines of code. But it's definitely add additional point of failure/bug.
Couldn't your back-end scripts running under ExaBGP also manage the FIB, using standard Unix tools/APIs? Managing the FIB is basically just "route add" and "route delete" right? Aled
On 20/05/2015 15:25, Aled Morris wrote:
Couldn't your back-end scripts running under ExaBGP also manage the FIB, using standard Unix tools/APIs?
Managing the FIB is basically just "route add" and "route delete" right?
Yes, you could probably do this. No, you probably wouldn't want to do this. Pls see the netlink interface modules in bird and quagga to understand why. Nick
Hello! Yes, we could run route add / route del when we got any announce from external world with ExaBGP directly. I have implemented custom custom Firewall (netmap-ipfw) management tool which implement in similar manner. But I'm working with BGP flow spec. It's so complex, standard BGP is much times simpler. And I could share my ExaBGP configuration and hook scripts. ExaBGP config: https://github.com/FastVPSEestiOu/fastnetmon/blob/master/src/scripts/exabgp_... Hook script which put all announces to Redis Queue: https://github.com/FastVPSEestiOu/fastnetmon/blob/master/src/scripts/exabgp_... But full BGP route table is enough big and need external processing. But yes, with some Python code is possible to implement route server with ExaBGP. On Wed, May 20, 2015 at 5:25 PM, Aled Morris <aledm@qix.co.uk> wrote:
On 20 May 2015 at 15:00, Pavel Odintsov <pavel.odintsov@gmail.com> wrote:
Yes, you could do filtering with Quagga. But Quagga is pretty old tool without multiple dynamic features. But with ExaBGP you could do really any significant route table transformations with Python in few lines of code. But it's definitely add additional point of failure/bug.
Couldn't your back-end scripts running under ExaBGP also manage the FIB, using standard Unix tools/APIs?
Managing the FIB is basically just "route add" and "route delete" right?
Aled
-- Sincerely yours, Pavel Odintsov
So, from the sounds of it most are saying for low cost, the way to go would be a software router, which I was trying to avoid. To answer the bandwidth question, we would have three 10G ports with three different carriers and at max push 10Gbps of total traffic to start. I think this leaves me with hardware routers that can support full BGP tables. So, who actually sells full bgp routers. So far on my list I have: Juniper MX Series Brocade MLXe or CER Cisco ASR 9K Huawei NE40E-X1-M4 ZTE, not sure which model? ALU 7750 Besides the above, am I missing anyone else that makes a true carrier grade hardware router? On Wed, May 20, 2015 at 9:54 AM, Pavel Odintsov <pavel.odintsov@gmail.com> wrote:
Hello!
Yes, we could run route add / route del when we got any announce from external world with ExaBGP directly. I have implemented custom custom Firewall (netmap-ipfw) management tool which implement in similar manner. But I'm working with BGP flow spec. It's so complex, standard BGP is much times simpler.
And I could share my ExaBGP configuration and hook scripts.
ExaBGP config: https://github.com/FastVPSEestiOu/fastnetmon/blob/master/src/scripts/exabgp_...
Hook script which put all announces to Redis Queue:
https://github.com/FastVPSEestiOu/fastnetmon/blob/master/src/scripts/exabgp_...
But full BGP route table is enough big and need external processing.
But yes, with some Python code is possible to implement route server with ExaBGP.
On Wed, May 20, 2015 at 5:25 PM, Aled Morris <aledm@qix.co.uk> wrote:
On 20 May 2015 at 15:00, Pavel Odintsov <pavel.odintsov@gmail.com> wrote:
Yes, you could do filtering with Quagga. But Quagga is pretty old tool without multiple dynamic features. But with ExaBGP you could do really any significant route table transformations with Python in few lines of code. But it's definitely add additional point of failure/bug.
Couldn't your back-end scripts running under ExaBGP also manage the FIB, using standard Unix tools/APIs?
Managing the FIB is basically just "route add" and "route delete" right?
Aled
-- Sincerely yours, Pavel Odintsov
good, cheap, built by someone else.... pick 2 On Wed, May 20, 2015 at 9:42 AM, Colton Conor <colton.conor@gmail.com> wrote:
So, from the sounds of it most are saying for low cost, the way to go would be a software router, which I was trying to avoid. To answer the bandwidth question, we would have three 10G ports with three different carriers and at max push 10Gbps of total traffic to start.
I think this leaves me with hardware routers that can support full BGP tables. So, who actually sells full bgp routers. So far on my list I have: Juniper MX Series Brocade MLXe or CER Cisco ASR 9K Huawei NE40E-X1-M4 ZTE, not sure which model? ALU 7750
Besides the above, am I missing anyone else that makes a true carrier grade hardware router?
On Wed, May 20, 2015 at 9:54 AM, Pavel Odintsov <pavel.odintsov@gmail.com> wrote:
Hello!
Yes, we could run route add / route del when we got any announce from external world with ExaBGP directly. I have implemented custom custom Firewall (netmap-ipfw) management tool which implement in similar manner. But I'm working with BGP flow spec. It's so complex, standard BGP is much times simpler.
And I could share my ExaBGP configuration and hook scripts.
ExaBGP config: https://github.com/FastVPSEestiOu/fastnetmon/blob/master/src/scripts/exabgp_...
Hook script which put all announces to Redis Queue:
https://github.com/FastVPSEestiOu/fastnetmon/blob/master/src/scripts/exabgp_...
But full BGP route table is enough big and need external processing.
But yes, with some Python code is possible to implement route server with ExaBGP.
On Wed, May 20, 2015 at 5:25 PM, Aled Morris <aledm@qix.co.uk> wrote:
On 20 May 2015 at 15:00, Pavel Odintsov <pavel.odintsov@gmail.com> wrote:
Yes, you could do filtering with Quagga. But Quagga is pretty old tool without multiple dynamic features. But with ExaBGP you could do really any significant route table transformations with Python in few lines of code. But it's definitely add additional point of failure/bug.
Couldn't your back-end scripts running under ExaBGP also manage the FIB, using standard Unix tools/APIs?
Managing the FIB is basically just "route add" and "route delete" right?
Aled
-- Sincerely yours, Pavel Odintsov
As mentioned by others on the list, a properly configured ASR1004 and up can do this. --Blake Colton Conor wrote on 5/20/2015 11:42 AM:
So, from the sounds of it most are saying for low cost, the way to go would be a software router, which I was trying to avoid. To answer the bandwidth question, we would have three 10G ports with three different carriers and at max push 10Gbps of total traffic to start.
I think this leaves me with hardware routers that can support full BGP tables. So, who actually sells full bgp routers. So far on my list I have: Juniper MX Series Brocade MLXe or CER Cisco ASR 9K Huawei NE40E-X1-M4 ZTE, not sure which model? ALU 7750
Besides the above, am I missing anyone else that makes a true carrier grade hardware router?
On Wed, May 20, 2015 at 9:54 AM, Pavel Odintsov <pavel.odintsov@gmail.com> wrote:
Hello!
Yes, we could run route add / route del when we got any announce from external world with ExaBGP directly. I have implemented custom custom Firewall (netmap-ipfw) management tool which implement in similar manner. But I'm working with BGP flow spec. It's so complex, standard BGP is much times simpler.
And I could share my ExaBGP configuration and hook scripts.
ExaBGP config: https://github.com/FastVPSEestiOu/fastnetmon/blob/master/src/scripts/exabgp_...
Hook script which put all announces to Redis Queue:
https://github.com/FastVPSEestiOu/fastnetmon/blob/master/src/scripts/exabgp_...
But full BGP route table is enough big and need external processing.
But yes, with some Python code is possible to implement route server with ExaBGP.
On Wed, May 20, 2015 at 5:25 PM, Aled Morris <aledm@qix.co.uk> wrote:
On 20 May 2015 at 15:00, Pavel Odintsov <pavel.odintsov@gmail.com> wrote:
Yes, you could do filtering with Quagga. But Quagga is pretty old tool without multiple dynamic features. But with ExaBGP you could do really any significant route table transformations with Python in few lines of code. But it's definitely add additional point of failure/bug.
Couldn't your back-end scripts running under ExaBGP also manage the FIB, using standard Unix tools/APIs?
Managing the FIB is basically just "route add" and "route delete" right?
Aled
-- Sincerely yours, Pavel Odintsov
Add Alcatel-Lucent 7750? I have no experience but this list seems to love them. On Wed, May 20, 2015, 9:44 AM Colton Conor <colton.conor@gmail.com> wrote:
So, from the sounds of it most are saying for low cost, the way to go would be a software router, which I was trying to avoid. To answer the bandwidth question, we would have three 10G ports with three different carriers and at max push 10Gbps of total traffic to start.
I think this leaves me with hardware routers that can support full BGP tables. So, who actually sells full bgp routers. So far on my list I have: Juniper MX Series Brocade MLXe or CER Cisco ASR 9K Huawei NE40E-X1-M4 ZTE, not sure which model? ALU 7750
Besides the above, am I missing anyone else that makes a true carrier grade hardware router?
On Wed, May 20, 2015 at 9:54 AM, Pavel Odintsov <pavel.odintsov@gmail.com> wrote:
Hello!
Yes, we could run route add / route del when we got any announce from external world with ExaBGP directly. I have implemented custom custom Firewall (netmap-ipfw) management tool which implement in similar manner. But I'm working with BGP flow spec. It's so complex, standard BGP is much times simpler.
And I could share my ExaBGP configuration and hook scripts.
ExaBGP config:
https://github.com/FastVPSEestiOu/fastnetmon/blob/master/src/scripts/exabgp_...
Hook script which put all announces to Redis Queue:
https://github.com/FastVPSEestiOu/fastnetmon/blob/master/src/scripts/exabgp_...
But full BGP route table is enough big and need external processing.
But yes, with some Python code is possible to implement route server with ExaBGP.
On Wed, May 20, 2015 at 5:25 PM, Aled Morris <aledm@qix.co.uk> wrote:
On 20 May 2015 at 15:00, Pavel Odintsov <pavel.odintsov@gmail.com> wrote:
Yes, you could do filtering with Quagga. But Quagga is pretty old tool without multiple dynamic features. But with ExaBGP you could do really any significant route table transformations with Python in few lines of code. But it's definitely add additional point of failure/bug.
Couldn't your back-end scripts running under ExaBGP also manage the
FIB,
using standard Unix tools/APIs?
Managing the FIB is basically just "route add" and "route delete" right?
Aled
-- Sincerely yours, Pavel Odintsov
Yep, thats what I meant be ALU 7750 :) On Wed, May 20, 2015 at 12:17 PM, Cody Grosskopf <codygrosskopf@gmail.com> wrote:
Add Alcatel-Lucent 7750? I have no experience but this list seems to love them.
On Wed, May 20, 2015, 9:44 AM Colton Conor <colton.conor@gmail.com> wrote:
So, from the sounds of it most are saying for low cost, the way to go would be a software router, which I was trying to avoid. To answer the bandwidth question, we would have three 10G ports with three different carriers and at max push 10Gbps of total traffic to start.
I think this leaves me with hardware routers that can support full BGP tables. So, who actually sells full bgp routers. So far on my list I have: Juniper MX Series Brocade MLXe or CER Cisco ASR 9K Huawei NE40E-X1-M4 ZTE, not sure which model? ALU 7750
Besides the above, am I missing anyone else that makes a true carrier grade hardware router?
On Wed, May 20, 2015 at 9:54 AM, Pavel Odintsov <pavel.odintsov@gmail.com
wrote:
Hello!
Yes, we could run route add / route del when we got any announce from external world with ExaBGP directly. I have implemented custom custom Firewall (netmap-ipfw) management tool which implement in similar manner. But I'm working with BGP flow spec. It's so complex, standard BGP is much times simpler.
And I could share my ExaBGP configuration and hook scripts.
ExaBGP config:
https://github.com/FastVPSEestiOu/fastnetmon/blob/master/src/scripts/exabgp_...
Hook script which put all announces to Redis Queue:
https://github.com/FastVPSEestiOu/fastnetmon/blob/master/src/scripts/exabgp_...
But full BGP route table is enough big and need external processing.
But yes, with some Python code is possible to implement route server with ExaBGP.
On Wed, May 20, 2015 at 5:25 PM, Aled Morris <aledm@qix.co.uk> wrote:
On 20 May 2015 at 15:00, Pavel Odintsov <pavel.odintsov@gmail.com> wrote:
Yes, you could do filtering with Quagga. But Quagga is pretty old
tool
without multiple dynamic features. But with ExaBGP you could do really any significant route table transformations with Python in few lines of code. But it's definitely add additional point of failure/bug.
Couldn't your back-end scripts running under ExaBGP also manage the FIB, using standard Unix tools/APIs?
Managing the FIB is basically just "route add" and "route delete" right?
Aled
-- Sincerely yours, Pavel Odintsov
ZTE M6000-3S. It is what we use. Works well for us. Just remember to get a memory upgrade to 8 GB memory or you will run out of RIB space. Regards Baldur Den 20/05/2015 18.43 skrev "Colton Conor" <colton.conor@gmail.com>:
So, from the sounds of it most are saying for low cost, the way to go would be a software router, which I was trying to avoid. To answer the bandwidth question, we would have three 10G ports with three different carriers and at max push 10Gbps of total traffic to start.
I think this leaves me with hardware routers that can support full BGP tables. So, who actually sells full bgp routers. So far on my list I have: Juniper MX Series Brocade MLXe or CER Cisco ASR 9K Huawei NE40E-X1-M4 ZTE, not sure which model? ALU 7750
Besides the above, am I missing anyone else that makes a true carrier grade hardware router?
On Wed, May 20, 2015 at 9:54 AM, Pavel Odintsov <pavel.odintsov@gmail.com> wrote:
Hello!
Yes, we could run route add / route del when we got any announce from external world with ExaBGP directly. I have implemented custom custom Firewall (netmap-ipfw) management tool which implement in similar manner. But I'm working with BGP flow spec. It's so complex, standard BGP is much times simpler.
And I could share my ExaBGP configuration and hook scripts.
ExaBGP config:
https://github.com/FastVPSEestiOu/fastnetmon/blob/master/src/scripts/exabgp_...
Hook script which put all announces to Redis Queue:
https://github.com/FastVPSEestiOu/fastnetmon/blob/master/src/scripts/exabgp_...
But full BGP route table is enough big and need external processing.
But yes, with some Python code is possible to implement route server with ExaBGP.
On Wed, May 20, 2015 at 5:25 PM, Aled Morris <aledm@qix.co.uk> wrote:
On 20 May 2015 at 15:00, Pavel Odintsov <pavel.odintsov@gmail.com> wrote:
Yes, you could do filtering with Quagga. But Quagga is pretty old tool without multiple dynamic features. But with ExaBGP you could do really any significant route table transformations with Python in few lines of code. But it's definitely add additional point of failure/bug.
Couldn't your back-end scripts running under ExaBGP also manage the
FIB,
using standard Unix tools/APIs?
Managing the FIB is basically just "route add" and "route delete" right?
Aled
-- Sincerely yours, Pavel Odintsov
Well, in my experience, which is limited to small iron mostly. Juniper MX104 Do not forget to get a second RE (Routine Engine) for software upgrade, and be prepare to accept to pay a "license" to use the 10Gbps ports on top of buying the IO cards. (1 license per 2 ports). Don't forget to set aside some times to port your configuration into it, if you are used to Cisco/Brocade style config. And that I'm too stupid to figure out a way to make 'test policy' do the same thing as "show ip bgp route-map XYZ" CER2K (latest revision) Has plenty of RAM for 6 full routing table (and maybe more) and 1.5M RIB compared to the ~524k from the first gen. ( Got burned on those ) MLX Juniper MX104 where cheaper for about the same platform using MLX products. Cisco I don't know about the licensing for the ASR but I mostly deal with second hand devices. They are not flashy but do the job. Huawei, ZTE I didn't touch those and mostly won't beside looking into some security concern some people are having. PS: With almost 130k prefixes polluting the routing table you could use a software route server and feed an auto-summary of the full route into a router/switch that can handle the RIB/FIB. I have yet to test Bird but I heard good things about using it for that function. ( By pollution, I mean, it was a test made on 6 peers where I found ~130k prefixes where using the same path as their larger subnet, I have to put up more time on that bench thou ) ----- Alain Hebert ahebert@pubnix.net PubNIX Inc. 50 boul. St-Charles P.O. Box 26770 Beaconsfield, Quebec H9W 6G7 Tel: 514-990-5911 http://www.pubnix.net Fax: 514-990-9443 On 05/20/15 12:42, Colton Conor wrote:
So, from the sounds of it most are saying for low cost, the way to go would be a software router, which I was trying to avoid. To answer the bandwidth question, we would have three 10G ports with three different carriers and at max push 10Gbps of total traffic to start.
I think this leaves me with hardware routers that can support full BGP tables. So, who actually sells full bgp routers. So far on my list I have: Juniper MX Series Brocade MLXe or CER Cisco ASR 9K Huawei NE40E-X1-M4 ZTE, not sure which model? ALU 7750
Besides the above, am I missing anyone else that makes a true carrier grade hardware router?
On Wed, May 20, 2015 at 9:54 AM, Pavel Odintsov <pavel.odintsov@gmail.com> wrote:
Hello!
Yes, we could run route add / route del when we got any announce from external world with ExaBGP directly. I have implemented custom custom Firewall (netmap-ipfw) management tool which implement in similar manner. But I'm working with BGP flow spec. It's so complex, standard BGP is much times simpler.
And I could share my ExaBGP configuration and hook scripts.
ExaBGP config: https://github.com/FastVPSEestiOu/fastnetmon/blob/master/src/scripts/exabgp_...
Hook script which put all announces to Redis Queue:
https://github.com/FastVPSEestiOu/fastnetmon/blob/master/src/scripts/exabgp_...
But full BGP route table is enough big and need external processing.
But yes, with some Python code is possible to implement route server with ExaBGP.
On Wed, May 20, 2015 at 5:25 PM, Aled Morris <aledm@qix.co.uk> wrote:
On 20 May 2015 at 15:00, Pavel Odintsov <pavel.odintsov@gmail.com> wrote:
Yes, you could do filtering with Quagga. But Quagga is pretty old tool without multiple dynamic features. But with ExaBGP you could do really any significant route table transformations with Python in few lines of code. But it's definitely add additional point of failure/bug.
Couldn't your back-end scripts running under ExaBGP also manage the FIB, using standard Unix tools/APIs?
Managing the FIB is basically just "route add" and "route delete" right?
Aled
-- Sincerely yours, Pavel Odintsov
On 20/May/15 19:44, Alain Hebert wrote:
Cisco
I don't know about the licensing for the ASR but I mostly deal with second hand devices.
They are not flashy but do the job.
If you are not trying to enable any IOS XR PIE's that need licenses (like video monitoring or optical monitoring), the only license worries you'll have on the ASR9001 is the ASR9001-S. The ASR9001-S is a 60Gbps version of the ASR9001 (50% capacity). You can upgrade the ASR9001-S to the ASR9001 with a software license. Mark.
On Wed, May 20, 2015 at 1:42 PM, Colton Conor <colton.conor@gmail.com> wrote:
So, from the sounds of it most are saying for low cost, the way to go would be a software router, which I was trying to avoid. To answer the bandwidth question, we would have three 10G ports with three different carriers and at max push 10Gbps of total traffic to start.
I think this leaves me with hardware routers that can support full BGP tables. So, who actually sells full bgp routers. So far on my list I have: Juniper MX Series Brocade MLXe or CER Cisco ASR 9K Huawei NE40E-X1-M4 ZTE, not sure which model? ALU 7750
Besides the above, am I missing anyone else that makes a true carrier grade hardware router?
right now I'm pushing 11G/s 1.2Mpps, ServerU L-800 + Chelsio T580-CR, see below although you can ssh in, it's definitely not a software router since it's essentially T5 ASICS hardware pushing the packets % sudo rate -i cxgbe0 -R -b => Currently 11.08 Gbps/1199.50 kpps, Average: 11.08 Gbps/1199.50 kpps => Currently 11.13 Gbps/1206.68 kpps, Average: 11.10 Gbps/1203.08 kpps => Currently 11.11 Gbps/1202.70 kpps, Average: 11.10 Gbps/1202.95 kpps => Currently 11.13 Gbps/1206.54 kpps, Average: 11.11 Gbps/1203.85 kpps => Currently 11.24 Gbps/1207.24 kpps, Average: 11.12 Gbps/1204.53 kpps => Currently 11.12 Gbps/1208.79 kpps, Average: 11.12 Gbps/1205.24 kpps => Currently 11.22 Gbps/1208.03 kpps, Average: 11.12 Gbps/1205.63 kpps => Currently 11.12 Gbps/1207.79 kpps, Average: 11.12 Gbps/1205.90 kpps => Currently 11.23 Gbps/1207.76 kpps, Average: 11.12 Gbps/1206.11 kpps => Currently 11.24 Gbps/1207.46 kpps, Average: 11.12 Gbps/1206.24 kpps => Currently 11.32 Gbps/1207.82 kpps, Average: 11.12 Gbps/1206.39 kpps => Currently 11.03 Gbps/1207.04 kpps, Average: 11.12 Gbps/1206.44 kpps btw this is a 40G QSFP SR4 port it's a thousand dollar card on top of a thousand dollar router + a penny for their x8 raiser card you won't find anything like that below 3k USD for your 10G routing low cost needs, I'm guessing
2015-05-20 20:54 GMT-03:00 BPNoC Group <bpnoc.lists@gmail.com>:
right now I'm pushing 11G/s 1.2Mpps, ServerU L-800 + Chelsio T580-CR, see below although you can ssh in, it's definitely not a software router since it's essentially T5 ASICS hardware pushing the packets
% sudo rate -i cxgbe0 -R -b => Currently 11.08 Gbps/1199.50 kpps, Average: 11.08 Gbps/1199.50 kpps => Currently 11.13 Gbps/1206.68 kpps, Average: 11.10 Gbps/1203.08 kpps => Currently 11.11 Gbps/1202.70 kpps, Average: 11.10 Gbps/1202.95 kpps => Currently 11.13 Gbps/1206.54 kpps, Average: 11.11 Gbps/1203.85 kpps => Currently 11.24 Gbps/1207.24 kpps, Average: 11.12 Gbps/1204.53 kpps => Currently 11.12 Gbps/1208.79 kpps, Average: 11.12 Gbps/1205.24 kpps => Currently 11.22 Gbps/1208.03 kpps, Average: 11.12 Gbps/1205.63 kpps => Currently 11.12 Gbps/1207.79 kpps, Average: 11.12 Gbps/1205.90 kpps => Currently 11.23 Gbps/1207.76 kpps, Average: 11.12 Gbps/1206.11 kpps => Currently 11.24 Gbps/1207.46 kpps, Average: 11.12 Gbps/1206.24 kpps => Currently 11.32 Gbps/1207.82 kpps, Average: 11.12 Gbps/1206.39 kpps => Currently 11.03 Gbps/1207.04 kpps, Average: 11.12 Gbps/1206.44 kpps
How much routes in the FIB? Thanks. -- Eduardo Schoedler
On Wed, May 20, 2015 at 9:16 PM, Eduardo Schoedler <listas@esds.com.br> wrote:
2015-05-20 20:54 GMT-03:00 BPNoC Group <bpnoc.lists@gmail.com>:
right now I'm pushing 11G/s 1.2Mpps, ServerU L-800 + Chelsio T580-CR, see below although you can ssh in, it's definitely not a software router since it's essentially T5 ASICS hardware pushing the packets
% sudo rate -i cxgbe0 -R -b => Currently 11.08 Gbps/1199.50 kpps, Average: 11.08 Gbps/1199.50 kpps => Currently 11.13 Gbps/1206.68 kpps, Average: 11.10 Gbps/1203.08 kpps => Currently 11.11 Gbps/1202.70 kpps, Average: 11.10 Gbps/1202.95 kpps => Currently 11.13 Gbps/1206.54 kpps, Average: 11.11 Gbps/1203.85 kpps => Currently 11.24 Gbps/1207.24 kpps, Average: 11.12 Gbps/1204.53 kpps => Currently 11.12 Gbps/1208.79 kpps, Average: 11.12 Gbps/1205.24 kpps => Currently 11.22 Gbps/1208.03 kpps, Average: 11.12 Gbps/1205.63 kpps => Currently 11.12 Gbps/1207.79 kpps, Average: 11.12 Gbps/1205.90 kpps => Currently 11.23 Gbps/1207.76 kpps, Average: 11.12 Gbps/1206.11 kpps => Currently 11.24 Gbps/1207.46 kpps, Average: 11.12 Gbps/1206.24 kpps => Currently 11.32 Gbps/1207.82 kpps, Average: 11.12 Gbps/1206.39 kpps => Currently 11.03 Gbps/1207.04 kpps, Average: 11.12 Gbps/1206.44 kpps
How much routes in the FIB?
Thanks.
actually it makes no difference, the relevant route entries are stored in the T5 chip cxgbetool tells me I have 532447 entries right now for fib 0 anyway, I have a similar number of entries (a couple more due to pinned ipv6 not triggered to the card), but other than management port for ssh, snmp, webgui and netflow, only 180kpps for a trunked copper dmz segment is actually forwarded at fib. everything else is done on the card
-- Eduardo Schoedler
Hello Pavel, Using ExaBGP as an SDN already has been done (and in a very large scale). But I would agree with Nick; It is not something I would recommend to everyone. Once more to echo Nick, to add/remove route/fw entries on Linux please do use netlink. The lastest ExaBGP master has some start of code to implement NetLink in python but I recently found a python module for it: https://github.com/svinota/pyroute2 Before ExaBGP can become a route server, I must complete a number of pieces (like the CLI which I am currently coding). I have spoken with the IX community about making ExaBGP a RR/RS and the idea was not badly received, but no one offered to help so it is on the back burner. Thomas On 20 May 2015, at 15:54, Pavel Odintsov wrote:
Hello!
Yes, we could run route add / route del when we got any announce from external world with ExaBGP directly. I have implemented custom custom Firewall (netmap-ipfw) management tool which implement in similar manner. But I'm working with BGP flow spec. It's so complex, standard BGP is much times simpler.
And I could share my ExaBGP configuration and hook scripts.
ExaBGP config: https://github.com/FastVPSEestiOu/fastnetmon/blob/master/src/scripts/exabgp_...
Hook script which put all announces to Redis Queue: https://github.com/FastVPSEestiOu/fastnetmon/blob/master/src/scripts/exabgp_...
But full BGP route table is enough big and need external processing.
But yes, with some Python code is possible to implement route server with ExaBGP.
On Wed, May 20, 2015 at 5:25 PM, Aled Morris <aledm@qix.co.uk> wrote:
On 20 May 2015 at 15:00, Pavel Odintsov <pavel.odintsov@gmail.com> wrote:
Yes, you could do filtering with Quagga. But Quagga is pretty old tool without multiple dynamic features. But with ExaBGP you could do really any significant route table transformations with Python in few lines of code. But it's definitely add additional point of failure/bug.
Couldn't your back-end scripts running under ExaBGP also manage the FIB, using standard Unix tools/APIs?
Managing the FIB is basically just "route add" and "route delete" right?
Aled
-- Sincerely yours, Pavel Odintsov
I have tried Cumulus. It's awesome! :) You definitely could run Quagga, Bird or even ExaBGP https://github.com/Exa-Networks/exabgp and build full feature router from 10GE switch. On Wed, May 20, 2015 at 4:32 PM, Cody Grosskopf <codygrosskopf@gmail.com> wrote:
I haven't tried myself but some of the stuff Cumulus Linux is doing is pretty amazing, not certain quagga can or should handle full bgp table but you could probably get a Penguin 10gbe for less than 8k.
On Tue, May 19, 2015, 10:25 AM Colton Conor <colton.conor@gmail.com> wrote:
What options are available for a small, low cost router that has at least four 10G ports, and can handle full BGP routes? All that I know of are the Juniper MX80, and the Brocade CER line. What does Cisco and others have that compete with these two? Any other vendors besides Juniper, Brocade, and Cisco to look at?
-- Sincerely yours, Pavel Odintsov
participants (40)
-
Alain Hebert -
Aled Morris -
Baldur Norddahl -
Blake Dunlap -
Blake Hudson -
BPNoC Group -
charles＠thefnf.org -
Cody Grosskopf -
Colin Johnston -
Colton Conor -
Eddie Tardist -
Eduardo Meyer -
Eduardo Schoedler -
Faisal Imtiaz -
Jameson, Daniel -
Jeff Tantsura -
Joe Greco -
Josh Baird -
Justin Wilson - MTIN -
Keefe John -
Ken Chase -
Mark Tees -
Mark Tinka -
Max Tulyev -
Mehmet Akcin -
Mel Beckman -
Mike Hammett -
Nick Hilliard -
Oleg A. Arkhangelsky -
Pavel Odintsov -
Piotr Iwanejko -
Rafael Possamai -
Randy Carpenter -
Ray Soucy -
Richard Holbo -
Rodrigo 1telecom -
Steve Noble -
Thomas Mangin -
Warsaw LATAM Operations Group -
William Waites