10GE router resource
Hi everybody! I find myself in the market for some 10GE routers. As I don't buy these everyday, I was wondering if any of you guys had any good resources for evaluating different vendors and models. I'm mainly thinking about non-vendor resources as the vendorspeak sites are not that hard to find. Also I'd love to hear recommendatios for "budget" 10GE routers. The "budget" router would be used to hook up client networks through one 10GE interface and connect to different transit providers through two 10GE interfaces. - Zed ____________________________________________________________________________________ Never miss a thing. Make Yahoo your home page. http://www.yahoo.com/r/hs
On Monday 24 March 2008, user user wrote:
Hi everybody!
Hello.
Also I'd love to hear recommendatios for "budget" 10GE routers. The "budget" router would be used to hook up client networks through one 10GE interface and connect to different transit providers through two 10GE interfaces.
Today, from Cisco, the smallest router you'll get a 10Gbps Ethernet port on is the Cisco ASR1000 series. Mind you, though, FCS for this box isn't until about May. Also, this box is oversubscribed as the current switch fabric is 10Gbps. From Juniper, the smallest M-series box you'll get the same port on is the M120 platform. You could also look at smaller switches from both vendors, but if you plan on taking full BGP feeds from your upstream providers, this might be an issue. Cheers, Mark.
From: Mark Tinka <mtinka@globaltransit.net> Date: Mon, 24 Mar 2008 19:12:57 +0800 Sender: owner-nanog@merit.edu
On Monday 24 March 2008, user user wrote:
Hi everybody!
Hello.
Also I'd love to hear recommendatios for "budget" 10GE routers. The "budget" router would be used to hook up client networks through one 10GE interface and connect to different transit providers through two 10GE interfaces.
Today, from Cisco, the smallest router you'll get a 10Gbps Ethernet port on is the Cisco ASR1000 series. Mind you, though, FCS for this box isn't until about May. Also, this box is oversubscribed as the current switch fabric is 10Gbps.
From Juniper, the smallest M-series box you'll get the same port on is the M120 platform.
You could also look at smaller switches from both vendors, but if you plan on taking full BGP feeds from your upstream providers, this might be an issue.
Depending on how the box will be used, Foundry is probably the cheapest, followed by Force10. Since yo will be connecting to two transit providers, you probably need the full routing table, but if you don't need full routes, the new Juniper EX8200 looks like an option. It is limited to about 12K routes in the FIB. It's not shipping at this time and I don't know when FSR is scheduled. Note that F10 does not do MPLS and neither F10 or Foundry has the software stability of either C or J, so you will need to look closely at exactly the features needed. -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: oberman@es.net Phone: +1 510 486-8634 Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751
Also I'd love to hear recommendatios for "budget" 10GE routers. The "budget" router would be used to hook up client networks through one 10GE interface and connect to different transit providers through two 10GE interfaces.
If you don't need BGP-ish power, David Newman just published his test of 10GigE switches today in Network World. He was focusing mostly on switching in the enterprise, but he has a variety of other performance metrics and results which may be helpful: http://www.networkworld.com/reviews/2008/032408-switch-test.html?t51hb jms -- Joel M Snyder, 1404 East Lind Road, Tucson, AZ, 85719 Senior Partner, Opus One Phone: +1 520 324 0494 jms@Opus1.COM http://www.opus1.com/jms
Joel Snyder wrote:
Also I'd love to hear recommendatios for "budget" 10GE routers. The "budget" router would be used to hook up client networks through one 10GE interface and connect to different transit providers through two 10GE interfaces.
If you don't need BGP-ish power, David Newman just published his test of 10GigE switches today in Network World. He was focusing mostly on switching in the enterprise, but he has a variety of other performance metrics and results which may be helpful:
http://www.networkworld.com/reviews/2008/032408-switch-test.html?t51hb
The author's specifications eliminated Cisco's 4900M from the competition. That not unexpected though since it was a evaluation of access switches w/ 10G uplinks. The 4900M has 8 on-board 10G interfaces and expansion modules that can carry 8 more (not oversubscribed) or 16 (oversubscribed). It has has GigE support via TwinGig modules in the expansion module bays. It also has a 320Gbps backplane and can handle up to 200k v4 routes. It's an impressive little switch if you need 10G aggregation. It can't handle a full table of course but it still has a lot of use. No MPLS options. It's based on the 4500's Sup 6-E. http://www.cisco.com/en/US/products/ps9310/index.html The base unit starts at $16k. Justin
On Mar 24, 2008, at 10:23 AM, user user wrote:
Hi everybody!
I find myself in the market for some 10GE routers. As I don't buy these everyday, I was wondering if any of you guys had any good resources for evaluating different vendors and models. I'm mainly thinking about non-vendor resources as the vendorspeak sites are not that hard to find.
Also I'd love to hear recommendatios for "budget" 10GE routers. The "budget" router would be used to hook up client networks through one 10GE interface and connect to different transit providers through two 10GE interfaces.
- Zed
Hiya, When it comes to budget, force10 are good. I wouldn't be able to confirm if they're worth performance-wise. I'd strongly suggest Foundry, I'm a big fan of their kits, price-wise and performance-wise, provided you do not need rocket-science features. MLX/XMR models will surely do the trick perfectly. When it comes to router purchasing habits, we all tend to get religious... Bottom line is that most of the 'regular' vendors (namely Cisco, Juniper, Foundry, Force10, Extreme, Riverstone) implement pretty much the same set of features, which are all IETF/IEEE normalized, meaning if you don't need proprietary features (and you'll wish you don't), any router will be fine, the only difference will come from: - the chassis being non-blocking or not (i.e. backplane design) - the price per port - the operating OS - the feeling you'll get with the salesperson, and the reputation of their Support Teams. - vendor specific features such as Flow Sampling To make it simple, most vendors have an IOS like OS, except Juniper which has a really clever and elegant OS, but are very pricey. Foundry and Force10 have the cheapest price per port Cisco does only Netflow, Foundry & Force10 only SFlow (which is a true standard) and I think Juniper does JFlow Cisco's kits are packed with proprietary protocols (HSRP and GLBP instead of VRRP, their own ethernet trunking, EIGRP as their own and yet extremely efficient IGP, TCL scriptable CLI...) , some of them are really good, some are crappy, but I suggest you'd stick with IEEE/IETF protocol to avoid future trouble. One thing: RSTP/802-1w is very (very, very, very) not often interoperable between vendors who all have their own interpretation of the norm and can quickly turn into a nightmare. I'd strongly suggest try&buys if (R)STP interoperability is required, but I'm a little paranoid :) Greg VILLAIN Independant Network & Telco Architecture Consultant
Greg has laid out a great bit of information and I would like to add just one possibility to the list of budget 10GE routers: Vyatta. According to a recent press release from that company ( http://www.vyatta.com/about/pressreleases.php?id=51) they offer a product that is "2 to 3X higher performance at a cost savings of more than 75 percent" when compared to Cisco's 7200. Unfortunately I have not had the opportunity to test or use the Vyatta routers yet; I have however successfully used other open-source Linux based routers in the past with great success. If you are looking for a truly budget 10GE router, they may be worth adding to the list and looking into. On Tue, Mar 25, 2008 at 10:36 AM, Greg VILLAIN <nanog@grrrrreg.net> wrote:
On Mar 24, 2008, at 10:23 AM, user user wrote:
Hi everybody!
I find myself in the market for some 10GE routers. As I don't buy these everyday, I was wondering if any of you guys had any good resources for evaluating different vendors and models. I'm mainly thinking about non-vendor resources as the vendorspeak sites are not that hard to find.
Also I'd love to hear recommendatios for "budget" 10GE routers. The "budget" router would be used to hook up client networks through one 10GE interface and connect to different transit providers through two 10GE interfaces.
- Zed
Hiya,
When it comes to budget, force10 are good. I wouldn't be able to confirm if they're worth performance-wise. I'd strongly suggest Foundry, I'm a big fan of their kits, price-wise and performance-wise, provided you do not need rocket-science features. MLX/XMR models will surely do the trick perfectly.
When it comes to router purchasing habits, we all tend to get religious... Bottom line is that most of the 'regular' vendors (namely Cisco, Juniper, Foundry, Force10, Extreme, Riverstone) implement pretty much the same set of features, which are all IETF/IEEE normalized, meaning if you don't need proprietary features (and you'll wish you don't), any router will be fine, the only difference will come from: - the chassis being non-blocking or not (i.e. backplane design) - the price per port - the operating OS - the feeling you'll get with the salesperson, and the reputation of their Support Teams. - vendor specific features such as Flow Sampling To make it simple, most vendors have an IOS like OS, except Juniper which has a really clever and elegant OS, but are very pricey. Foundry and Force10 have the cheapest price per port Cisco does only Netflow, Foundry & Force10 only SFlow (which is a true standard) and I think Juniper does JFlow Cisco's kits are packed with proprietary protocols (HSRP and GLBP instead of VRRP, their own ethernet trunking, EIGRP as their own and yet extremely efficient IGP, TCL scriptable CLI...) , some of them are really good, some are crappy, but I suggest you'd stick with IEEE/IETF protocol to avoid future trouble.
One thing: RSTP/802-1w is very (very, very, very) not often interoperable between vendors who all have their own interpretation of the norm and can quickly turn into a nightmare. I'd strongly suggest try&buys if (R)STP interoperability is required, but I'm a little paranoid :)
Greg VILLAIN Independant Network & Telco Architecture Consultant
-- "Those who do not create the future they want must endure the future they get." ~Draper L. Kaufman, Jr. --
Chris Grundemann wrote:
Greg has laid out a great bit of information and I would like to add just one possibility to the list of budget 10GE routers: Vyatta. According to a recent press release from that company (http://www.vyatta.com/about/pressreleases.php?id=51) they offer a product that is "2 to 3X higher performance at a cost savings of more than 75 percent" when compared to Cisco's 7200. Unfortunately I have not had the opportunity to test or use the Vyatta routers yet; I have however successfully used other open-source Linux based routers in the past with great success. If you are looking for a truly budget 10GE router, they may be worth adding to the list and looking into.
Whether you can actually do 10Gb/s reasonably on a linux or freebsd soft-switched router platform is going to depend a lot on your actual pps rate. 800K pps which is 10Gb/s / 1500 bytes is feasible, but 19M pps which is 10Gb/s / 64 bytes is not. Susceptibility to dos traffic at relatively low bit, but high pps rates is a general issue with soft-switched platforms. and needs to be accounted for in model deployments.
On Tue, Mar 25, 2008 at 10:36 AM, Greg VILLAIN <nanog@grrrrreg.net <mailto:nanog@grrrrreg.net>> wrote:
On Mar 24, 2008, at 10:23 AM, user user wrote: > > Hi everybody! > > I find myself in the market for some 10GE routers. As > I don't buy these everyday, I was wondering if any of > you guys had any good resources for evaluating > different vendors and models. I'm mainly thinking > about non-vendor resources as the vendorspeak sites > are not that hard to find. > > Also I'd love to hear recommendatios for "budget" 10GE > routers. The "budget" router would be used to hook up > client networks through one 10GE interface and connect > to different transit providers through two 10GE > interfaces. > > - Zed
Hiya,
When it comes to budget, force10 are good. I wouldn't be able to confirm if they're worth performance-wise. I'd strongly suggest Foundry, I'm a big fan of their kits, price-wise and performance-wise, provided you do not need rocket-science features. MLX/XMR models will surely do the trick perfectly.
When it comes to router purchasing habits, we all tend to get religious... Bottom line is that most of the 'regular' vendors (namely Cisco, Juniper, Foundry, Force10, Extreme, Riverstone) implement pretty much the same set of features, which are all IETF/IEEE normalized, meaning if you don't need proprietary features (and you'll wish you don't), any router will be fine, the only difference will come from: - the chassis being non-blocking or not (i.e. backplane design) - the price per port - the operating OS - the feeling you'll get with the salesperson, and the reputation of their Support Teams. - vendor specific features such as Flow Sampling To make it simple, most vendors have an IOS like OS, except Juniper which has a really clever and elegant OS, but are very pricey. Foundry and Force10 have the cheapest price per port Cisco does only Netflow, Foundry & Force10 only SFlow (which is a true standard) and I think Juniper does JFlow Cisco's kits are packed with proprietary protocols (HSRP and GLBP instead of VRRP, their own ethernet trunking, EIGRP as their own and yet extremely efficient IGP, TCL scriptable CLI...) , some of them are really good, some are crappy, but I suggest you'd stick with IEEE/IETF protocol to avoid future trouble.
One thing: RSTP/802-1w is very (very, very, very) not often interoperable between vendors who all have their own interpretation of the norm and can quickly turn into a nightmare. I'd strongly suggest try&buys if (R)STP interoperability is required, but I'm a little paranoid :)
Greg VILLAIN Independant Network & Telco Architecture Consultant
-- "Those who do not create the future they want must endure the future they get." ~Draper L. Kaufman, Jr. --
Hi Chris Could you share your opensouce 10G info for me? For the past 8 months, I have problem to use the 10G in linux system. I have to continuous to upgrade the hardwares... my existing system is using the new CPU now, 4G memory, 1 x 10G card plus several 1G NICs. Intel 2 Ext CPU X9650 @ 3.00GHz All CPU is in 100% used when it is in 4G totally (download + upload). thank you so much --- Chris Grundemann <cgrundemann@gmail.com> wrote:
Greg has laid out a great bit of information and I would like to add just one possibility to the list of budget 10GE routers: Vyatta. According to a recent press release from that company ( http://www.vyatta.com/about/pressreleases.php?id=51) they offer a product that is "2 to 3X higher performance at a cost savings of more than 75 percent" when compared to Cisco's 7200. Unfortunately I have not had the opportunity to test or use the Vyatta routers yet; I have however successfully used other open-source Linux based routers in the past with great success. If you are looking for a truly budget 10GE router, they may be worth adding to the list and looking into.
On Tue, Mar 25, 2008 at 10:36 AM, Greg VILLAIN <nanog@grrrrreg.net> wrote:
On Mar 24, 2008, at 10:23 AM, user user wrote:
Hi everybody!
I find myself in the market for some 10GE
I don't buy these everyday, I was wondering if any of you guys had any good resources for evaluating different vendors and models. I'm mainly
about non-vendor resources as the vendorspeak sites are not that hard to find.
Also I'd love to hear recommendatios for "budget" 10GE routers. The "budget" router would be used to hook up client networks through one 10GE interface and connect to different transit providers through two 10GE interfaces.
- Zed
Hiya,
When it comes to budget, force10 are good. I wouldn't be able to confirm if they're worth performance-wise. I'd strongly suggest Foundry, I'm a big fan of
and performance-wise, provided you do not need rocket-science features. MLX/XMR models will surely do the trick perfectly.
When it comes to router purchasing habits, we all tend to get religious... Bottom line is that most of the 'regular' vendors (namely Cisco, Juniper, Foundry, Force10, Extreme, Riverstone) implement pretty much the same set of features, which are all IETF/IEEE normalized, meaning if you don't need proprietary features (and you'll wish you don't), any router will be fine, the only difference will come from: - the chassis being non-blocking or not (i.e. backplane design) - the price per port - the operating OS - the feeling you'll get with the salesperson, and
their Support Teams. - vendor specific features such as Flow Sampling To make it simple, most vendors have an IOS like OS, except Juniper which has a really clever and elegant OS, but are very pricey. Foundry and Force10 have the cheapest price per
routers. As thinking their kits, price-wise the reputation of port
Cisco does only Netflow, Foundry & Force10 only SFlow (which is a true standard) and I think Juniper does JFlow Cisco's kits are packed with proprietary protocols (HSRP and GLBP instead of VRRP, their own ethernet trunking, EIGRP as their own and yet extremely efficient IGP, TCL scriptable CLI...) , some of them are really good, some are crappy, but I suggest you'd stick with IEEE/IETF protocol to avoid future trouble.
One thing: RSTP/802-1w is very (very, very, very) not often interoperable between vendors who all have their own interpretation of the norm and can quickly turn into a nightmare. I'd strongly suggest try&buys if (R)STP interoperability is required, but I'm a little paranoid :)
Greg VILLAIN Independant Network & Telco Architecture Consultant
-- "Those who do not create the future they want must endure the future they get." ~Draper L. Kaufman, Jr. --
____________________________________________________________________________________ Looking for last minute shopping deals? Find them fast with Yahoo! Search. http://tools.search.yahoo.com/newsearch/category.php?category=shopping
On Tue, Mar 25, 2008 at 1:59 PM, Chris Grundemann <cgrundemann@gmail.com> wrote:
Greg has laid out a great bit of information and I would like to add just one possibility to the list of budget 10GE routers: Vyatta. According to a recent press release from that company (http://www.vyatta.com/about/pressreleases.php?id=51) they offer a product that is "2 to 3X higher performance at a cost savings of more than 75 percent" when compared to Cisco's 7200.
"Vyatta operates at Layer 3 wire speed across three Gigabit Ethernet ports in full mesh when forwarding 512-byte frames or higher." 3x1 GE << 1x10 GE Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
On Tue, Mar 25, 2008 at 1:56 PM, William Herrin <herrin-nanog@dirtside.com> wrote:
On Tue, Mar 25, 2008 at 1:59 PM, Chris Grundemann <cgrundemann@gmail.com> wrote:
Greg has laid out a great bit of information and I would like to add just one possibility to the list of budget 10GE routers: Vyatta. According to a recent press release from that company (http://www.vyatta.com/about/pressreleases.php?id=51) they offer a product that is "2 to 3X higher performance at a cost savings of more than 75 percent" when compared to Cisco's 7200.
"Vyatta operates at Layer 3 wire speed across three Gigabit Ethernet ports in full mesh when forwarding 512-byte frames or higher."
3x1 GE << 1x10 GE
It appears that I put my foot in my mouth. I have read several claims that the Vyatta software is scalable to 10G, most notably here: http://www.networkworld.com/news/2008/031708-vyatta-open-source-router.html. Upon further investigation, I have been unable to substantiate that claim. My experience is similar to those who have posted here, pps is the limiting factor - usually somewhere between 500-800K. Apparently I was over eager to believe that more had been achieved. To Ann's question on resources; I have only used Linux routers with 1G ports but have surpassed 10G total throughput (up+ down) using various dual proc set ups, most often Intel Xeon in Dell servers. A gentlemen by the name of Martin Pels wrote a good paper on the subject early last year that can be found here: http://docs.rodecker.nl/10-GE_Routing_on_Linux.pdf. He hit a wall at 700K pps and was using two dual core Intel Xeon 64bit 2.33GHz CPUs and 2GB of RAM in a Dell PowerEdge 1950. ~Chris
Regards, Bill Herrin
-- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
On Tue, Mar 25, 2008, Chris Grundemann wrote:
To Ann's question on resources; I have only used Linux routers with 1G ports but have surpassed 10G total throughput (up+ down) using various dual proc set ups, most often Intel Xeon in Dell servers. A gentlemen by the name of Martin Pels wrote a good paper on the subject early last year that can be found here: http://docs.rodecker.nl/10-GE_Routing_on_Linux.pdf. He hit a wall at 700K pps and was using two dual core Intel Xeon 64bit 2.33GHz CPUs and 2GB of RAM in a Dell PowerEdge 1950.
Mike Tancsa did some benchmarking in late 2006: http://www.tancsa.com/blast.html I think things are slightly faster now but not because of a massive change in software architecture. Adrian
Very interesting study I had not seen, and a bummer. That really puts a cramp in my advocation of our CARP+pf load balancers/firewalls/gateways. Than again, what's a PIX box capable of? I also had to switch to OpenBSD as there was a fatal crash with the bridge device in FreeBSD when used with my paticular OpenVPN/CARP/pf combination. AFAIK pf/forwarding only takes place on one core and wouldn't take advantage of the other 3 cores, correct? -Patrick ----- Original Message ----- From: "Adrian Chadd" <adrian@creative.net.au> To: "Chris Grundemann" <cgrundemann@gmail.com> Cc: "William Herrin" <herrin-nanog@dirtside.com>, nanog@nanog.org Sent: Tuesday, March 25, 2008 6:02:03 PM (GMT-0800) America/Los_Angeles Subject: Re: 10GE router resource On Tue, Mar 25, 2008, Chris Grundemann wrote:
To Ann's question on resources; I have only used Linux routers with 1G ports but have surpassed 10G total throughput (up+ down) using various dual proc set ups, most often Intel Xeon in Dell servers. A gentlemen by the name of Martin Pels wrote a good paper on the subject early last year that can be found here: http://docs.rodecker.nl/10-GE_Routing_on_Linux.pdf. He hit a wall at 700K pps and was using two dual core Intel Xeon 64bit 2.33GHz CPUs and 2GB of RAM in a Dell PowerEdge 1950.
Mike Tancsa did some benchmarking in late 2006: http://www.tancsa.com/blast.html I think things are slightly faster now but not because of a massive change in software architecture. Adrian
On Tue, Mar 25, 2008 at 6:15 PM, Patrick Clochesy <patrick@chegg.com> wrote:
Very interesting study I had not seen, and a bummer. That really puts a cramp in my advocation of our CARP+pf load balancers/firewalls/gateways. Than again, what's a PIX box capable of?
I'd rather tweak a whitebox than pay through the nose for a PIX.
I also had to switch to OpenBSD as there was a fatal crash with the bridge device in FreeBSD when used with my paticular OpenVPN/CARP/pf combination.
AFAIK pf/forwarding only takes place on one core and wouldn't take advantage of the other 3 cores, correct?
Correct. There has been some great speed and efficiency improvements in pf and other networking parts of OpenBSD; though from anecdotal evidence, 10GbE is not ready for 'primetime' (for certain definitions of 'primetime'). actually I'll just skip making an ass out of myself and hope henning@ chimes in, since I believe he reads NANOG as well. aaron.glenn
Aaron Glenn wrote:
On Tue, Mar 25, 2008 at 6:15 PM, Patrick Clochesy <patrick@chegg.com> wrote:
Very interesting study I had not seen, and a bummer. That really puts a cramp in my advocation of our CARP+pf load balancers/firewalls/gateways. Than again, what's a PIX box capable of?
I'd rather tweak a whitebox than pay through the nose for a PIX.
Curious if you or others have tried Solaris 10 or OpenSolaris, they claim that they are approaching wire speed 10G with the right card (possibly their own, which is about $995 list). --Patrick
On Tue, Mar 25, 2008, Patrick Giagnocavo wrote:
Aaron Glenn wrote:
On Tue, Mar 25, 2008 at 6:15 PM, Patrick Clochesy <patrick@chegg.com> wrote:
Very interesting study I had not seen, and a bummer. That really puts a cramp in my advocation of our CARP+pf load balancers/firewalls/gateways. Than again, what's a PIX box capable of?
I'd rather tweak a whitebox than pay through the nose for a PIX.
Curious if you or others have tried Solaris 10 or OpenSolaris, they claim that they are approaching wire speed 10G with the right card (possibly their own, which is about $995 list).
FreeBSD is doing wirespeed 10GE. Oh wait, do you mean forwarding, or just TCP? :) Adrian
On Tue, 25 Mar 2008, Aaron Glenn wrote:
On Tue, Mar 25, 2008 at 6:15 PM, Patrick Clochesy <patrick@chegg.com> wrote:
Very interesting study I had not seen, and a bummer. That really puts a cramp in my advocation of our CARP+pf load balancers/firewalls/gateways. Than again, what's a PIX box capable of?
I'd rather tweak a whitebox than pay through the nose for a PIX.
I also had to switch to OpenBSD as there was a fatal crash with the bridge device in FreeBSD when used with my paticular OpenVPN/CARP/pf combination.
AFAIK pf/forwarding only takes place on one core and wouldn't take advantage of the other 3 cores, correct?
Correct. There has been some great speed and efficiency improvements in pf and other networking parts of OpenBSD; though from anecdotal evidence, 10GbE is not ready for 'primetime' (for certain definitions of 'primetime').
Anybody who does any sort of home-brew routing NEEDS to read this post: http://lists.freebsd.org/pipermail/freebsd-current/2008-January/082469.html Quote: --- Forwarding (routing between multiple interfaces) and filtering (ipfw) IIRC with quad Intel e1000 NIC: Dual Intel Xeon 2.8GHz: 240Kpps 12k L1 cache Single Intel Xeon 2.8GHz: 380Kpps 12k L1 cache Core 2 Duo 1.8Ghz: 420kpps 12k L1 cache Single Pentium-M 1.8GHz: 550Kpps 32k L1 cache Dual AMD opteron 2GHz: 890Kpps 64k L1 cache Single AMD opteron 2GHz: 970Kpps 64k L1 cache All these hosts had 255 vlan interfaces with about 3000 routes and about 30000 firewall rules, with a good spread of packets between the interfaces with polling and fastforwarding. I struggled to generate enough packets to load the AMD routers. --- Quite interesting data, no? Especially when you can now get 3GHz opterons with 128k of L1 cache? How sweet is a sub-$1k router that can do multiple gig-e's at 1.5mpps? Sounds like a dynamite platform for high-end datacenter CPEs that are soft on dynamic routing...and even the open-source dynamic routing is reasonably solid these days... Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 ---
How sweet is a sub-$1k router that can do multiple gig-e's at 1.5mpps? Sounds like a dynamite platform for high-end datacenter CPEs that are soft on dynamic routing...and even the open-source dynamic routing is reasonably solid these days...
I can't believe I am about to ask this on a public mailing list, but.. Has anyone tested this in even a remotely production environment, while running any sort of MPLS LDP as a LSR?
On Wed, Mar 26, 2008 at 1:16 AM, Alex Rubenstein <alex@corp.nac.net> wrote:
How sweet is a sub-$1k router that can do multiple gig-e's at 1.5mpps? Sounds like a dynamite platform for high-end datacenter CPEs that are soft on dynamic routing...and even the open-source dynamic routing is reasonably solid these days...
I can't believe I am about to ask this on a public mailing list, but..
Has anyone tested this in even a remotely production environment, while running any sort of MPLS LDP as a LSR?
bahahaah! oh, sorry... also, how does all-small-packets performance and reasonable ACL behaviour work? (reasonable for dos things let's keep under 1k acl lines) What about IDB-type numbers? is this a 10-interfaces at line-rate or 10k interfaces at line-rate (line-rate on say ... 8 10G interfaces)? Scaling a routing platform in software for high bandwidth services is difficult... or seems to be at least. -Chris
On Tuesday 25 March 2008, Aaron Glenn wrote:
On Tue, Mar 25, 2008 at 6:15 PM, Patrick Clochesy <patrick@chegg.com> wrote:
Very interesting study I had not seen, and a bummer. That really puts a cramp in my advocation of our CARP+pf load balancers/firewalls/gateways. Than again, what's a PIX box capable of?
I'd rather tweak a whitebox than pay through the nose for a PIX.
But aren't PIXen whiteboxes internally? I know the PIX-like LocalDirector that was donated to us makes a very nice nBox deployment for us. Lots of these sorts of boxes are internally whiteboxes (I'm using that term loosely to mean an Intel-based box that could potentially run something like a Linux or *BSD). The second-hand Content Engine 565 I got on eBay that had a fried power supply was just a Cisco-labeled IBM eServer xSeries 305, and was loaded with WindowsXP when I got it. It's running CentOS 5 now, with a new IBM power supply in the box. The two earlier Content Engines and two even earlier Cache Engines I got second-hand are likewise custom Intel hardware; PIII 800's, to be precise. Now, they DO use ECC RAM, which most whiteboxes won't have. But otherwise they are customized whiteboxes, and you're paying for the software and support. But cisco is not alone in this. Nomadix gateways, to use one example, are built on custom embedded x86 systems. What I'm waiting on is someone to take a system like a Xilinx ML410 dev board and use the FPGA to do hardware-accelerated forwarding/filtering. See http://www.lynuxworks.com/board-support/xilinx/ml410.php for info on the board. As to PIXen performance, see the charts in http://en.wikipedia.org/wiki/Cisco_PIX -- Lamar Owen Chief Information Officer Pisgah Astronomical Research Institute 1 PARI Drive Rosman, NC 28772 (828)862-5554 www.pari.edu
The PIX are EoS. Yes, they were white boxes when Cisco bought out the original company. The ASA's, however, are not white boxes. That said, it is notable that Cisco is now running their latest announced hardware, primarily the Nexus 7000's and ASR's, run a Linux kernel and IOS on top of that. That doesn't make them white boxes either though. Fred Reimer, CISSP, CCNP, CQS-VPN, CQS-ISS Senior Network Engineer Coleman Technologies, Inc. 954-298-1697 -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Lamar Owen Sent: Wednesday, March 26, 2008 12:20 PM To: nanog@nanog.org Subject: Re: 10GE router resource On Tuesday 25 March 2008, Aaron Glenn wrote:
On Tue, Mar 25, 2008 at 6:15 PM, Patrick Clochesy <patrick@chegg.com> wrote:
Very interesting study I had not seen, and a bummer. That really puts a cramp in my advocation of our CARP+pf load balancers/firewalls/gateways. Than again, what's a PIX box capable of?
I'd rather tweak a whitebox than pay through the nose for a PIX.
But aren't PIXen whiteboxes internally? I know the PIX-like LocalDirector that was donated to us makes a very nice nBox deployment for us. Lots of these sorts of boxes are internally whiteboxes (I'm using that term loosely to mean an Intel-based box that could potentially run something like a Linux or *BSD). The second-hand Content Engine 565 I got on eBay that had a fried power supply was just a Cisco-labeled IBM eServer xSeries 305, and was loaded with WindowsXP when I got it. It's running CentOS 5 now, with a new IBM power supply in the box. The two earlier Content Engines and two even earlier Cache Engines I got second-hand are likewise custom Intel hardware; PIII 800's, to be precise. Now, they DO use ECC RAM, which most whiteboxes won't have. But otherwise they are customized whiteboxes, and you're paying for the software and support. But cisco is not alone in this. Nomadix gateways, to use one example, are built on custom embedded x86 systems. What I'm waiting on is someone to take a system like a Xilinx ML410 dev board and use the FPGA to do hardware-accelerated forwarding/filtering. See http://www.lynuxworks.com/board-support/xilinx/ml410.php for info on the board. As to PIXen performance, see the charts in http://en.wikipedia.org/wiki/Cisco_PIX -- Lamar Owen Chief Information Officer Pisgah Astronomical Research Institute 1 PARI Drive Rosman, NC 28772 (828)862-5554 www.pari.edu
That said, it is notable that Cisco is now running their latest announced hardware, primarily the Nexus 7000's and ASR's, run a Linux kernel and IOS on top of that.
Moore's Law may have helped software packet forwarding rates but there's still 2 to 3 orders of magnitude performance difference between hardware & software. just to be clear about a few things: in the case of Nexus 7K the control-plane runs atop of Linux, data-plane runs entirely in custom packet forwarding ASICs distributed on the I/O (linecard) modules. N7K never drops to "software forwarding". the first forwarding engine in N7K does 60M PPS with all features enabled. i.e. you could be performing ACLs on port, VLAN & routed on both ingress & egress, doing netflow, policing, QoS, whatever - its still 60M PPS. you'll see that pps numbers scale upwards as the product progresses through its roadmap. Cisco doesn't make any secret of N7K running atop of Linux, the reality is that it doesn't have to be Linux, it could be any SMP/multi-threaded capable POSIX-compliant kernel, it just so happens that Linux makes sense for a variety of reasons. Also, perhaps pedantic but just to be absolutely clear: N7K doesn't run on IOS, it runs on NX-OS. ASR is slightly different, it can perform packet processing in software (IOSd) however that is really only meant for things that don't make sense to implement in what is now called the QuantumFlow programmable processor. e.g. if you needed your AppleTalk or Vines running at millions of packets/second, then i'd argue you have bigger problems. :) cheers, lincoln.
Yes, when I said IOS runs on top of Linux I was specifically referring to the ASR, not both the ASR and the Nexus 7K. Both platforms were just announced, and Cisco has decade long (at least) plans for their life cycle, particularly given how much was invested in their development. The ASR can punt packets to the RP, but it has complete separation between the control and data plane in my understanding. Fred Reimer, CISSP, CCNP, CQS-VPN, CQS-ISS Senior Network Engineer Coleman Technologies, Inc. 954-298-1697 -----Original Message----- From: Lincoln Dale [mailto:ltd@interlink.com.au] Sent: Thursday, March 27, 2008 2:46 AM To: Fred Reimer; 'Lamar Owen'; nanog@nanog.org Subject: RE: 10GE router resource
That said, it is notable that Cisco is now running their latest announced hardware, primarily the Nexus 7000's and ASR's, run a Linux kernel and IOS on top of that.
Moore's Law may have helped software packet forwarding rates but there's still 2 to 3 orders of magnitude performance difference between hardware & software. just to be clear about a few things: in the case of Nexus 7K the control-plane runs atop of Linux, data-plane runs entirely in custom packet forwarding ASICs distributed on the I/O (linecard) modules. N7K never drops to "software forwarding". the first forwarding engine in N7K does 60M PPS with all features enabled. i.e. you could be performing ACLs on port, VLAN & routed on both ingress & egress, doing netflow, policing, QoS, whatever - its still 60M PPS. you'll see that pps numbers scale upwards as the product progresses through its roadmap. Cisco doesn't make any secret of N7K running atop of Linux, the reality is that it doesn't have to be Linux, it could be any SMP/multi-threaded capable POSIX-compliant kernel, it just so happens that Linux makes sense for a variety of reasons. Also, perhaps pedantic but just to be absolutely clear: N7K doesn't run on IOS, it runs on NX-OS. ASR is slightly different, it can perform packet processing in software (IOSd) however that is really only meant for things that don't make sense to implement in what is now called the QuantumFlow programmable processor. e.g. if you needed your AppleTalk or Vines running at millions of packets/second, then i'd argue you have bigger problems. :) cheers, lincoln.
* Aaron Glenn <aaron.glenn@gmail.com> [2008-03-26 03:14]:
On Tue, Mar 25, 2008 at 6:15 PM, Patrick Clochesy <patrick@chegg.com> wrote:
Very interesting study I had not seen, and a bummer. That really puts a cramp in my advocation of our CARP+pf load balancers/firewalls/gateways. Than again, what's a PIX box capable of?
I'd rather tweak a whitebox than pay through the nose for a PIX.
I also had to switch to OpenBSD as there was a fatal crash with the bridge device in FreeBSD when used with my paticular OpenVPN/CARP/pf combination.
AFAIK pf/forwarding only takes place on one core and wouldn't take advantage of the other 3 cores, correct?
Correct. There has been some great speed and efficiency improvements in pf and other networking parts of OpenBSD; though from anecdotal evidence, 10GbE is not ready for 'primetime' (for certain definitions of 'primetime').
actually I'll just skip making an ass out of myself and hope henning@ chimes in, since I believe he reads NANOG as well.
occasionally. as with all other OSes constructed benchmarks would show 10GE to work at wirespeed with reasonable hardware. I would not use it (yet) if I truly need 10 GBit/s forwarding rate, and that goes for any OS. -- Henning Brauer, hb@bsws.de, henning@openbsd.org BS Web Services, http://bsws.de Full-Service ISP - Secure Hosting, Mail and DNS Services Dedicated Servers, Rootservers, Application Hosting - Hamburg & Amsterdam
On Tue, Mar 25, 2008, Patrick Clochesy wrote:
Very interesting study I had not seen, and a bummer. That really puts a cramp in my advocation of our CARP+pf load balancers/firewalls/gateways. Than again, what's a PIX box capable of?
Well, you get what you pay for. If you're willing to blow $10k on a firewall, maybe you'll be willing to blow $10k on a *BSD developer to work on improving forwarding performance. It'd only take ten or so people to make donations or sponsor work of that size for the benefits to appear.
I also had to switch to OpenBSD as there was a fatal crash with the bridge device in FreeBSD when used with my paticular OpenVPN/CARP/pf combination.
Did you log a bug? :)
AFAIK pf/forwarding only takes place on one core and wouldn't take advantage of the other 3 cores, correct?
Uhm, its not quite that simple. ithreads on FreeBSD at least will run on one CPU at a time (unless you're running some hacked up russian-driven intel gige driver, which runs multiple ithreads for the device to improve performance under certain circumstances!) and these classes of cards and busses wouldn't benefit from >1 core contending for one card/bus. If you're running >1 card then you may find the ithreads run on different CPUs, each doing lookups and forwarding, but I haven't sat down and looked at that sort of forwarding performance under FreeBSD. My focus at the moment is "tcp proxy on a stick" throughput with one interfaces and >1 core doing userland processing. Adrian
To answer your question, the 5580 ASA (PIX is EoS if you didn’t know) is capable of 10G “HTTP” traffic and 20G “jumbo frame” packets. However, 64-byte packet rate is “limited” to 4,000,000pps. And yes, you will pay for that performance. You get a lot more than just a packet filter with the ASA though. Fred Reimer, CISSP, CCNP, CQS-VPN, CQS-ISS Senior Network Engineer Coleman Technologies, Inc. 954-298-1697 From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Patrick Clochesy Sent: Tuesday, March 25, 2008 9:16 PM To: Adrian Chadd Cc: nanog@nanog.org Subject: Re: 10GE router resource Very interesting study I had not seen, and a bummer. That really puts a cramp in my advocation of our CARP+pf load balancers/firewalls/gateways. Than again, what's a PIX box capable of? I also had to switch to OpenBSD as there was a fatal crash with the bridge device in FreeBSD when used with my paticular OpenVPN/CARP/pf combination. AFAIK pf/forwarding only takes place on one core and wouldn't take advantage of the other 3 cores, correct? -Patrick ----- Original Message ----- From: "Adrian Chadd" <adrian@creative.net.au> To: "Chris Grundemann" <cgrundemann@gmail.com> Cc: "William Herrin" <herrin-nanog@dirtside.com>, nanog@nanog.org Sent: Tuesday, March 25, 2008 6:02:03 PM (GMT-0800) America/Los_Angeles Subject: Re: 10GE router resource On Tue, Mar 25, 2008, Chris Grundemann wrote:
To Ann's question on resources; I have only used Linux routers with 1G ports but have surpassed 10G total throughput (up+ down) using various dual proc set ups, most often Intel Xeon in Dell servers. A gentlemen by the name of Martin Pels wrote a good paper on the subject early last year that can be found here: http://docs.rodecker.nl/10-GE_Routing_on_Linux.pdf. He hit a wall at 700K pps and was using two dual core Intel Xeon 64bit 2.33GHz CPUs and 2GB of RAM in a Dell PowerEdge 1950.
Mike Tancsa did some benchmarking in late 2006: http://www.tancsa.com/blast.html I think things are slightly faster now but not because of a massive change in software architecture. Adrian
Patrick Clochesy wrote:
Very interesting study I had not seen, and a bummer. That really puts a cramp in my advocation of our CARP+pf load balancers/firewalls/gateways. Than again, what's a PIX box capable of?
I also had to switch to OpenBSD as there was a fatal crash with the bridge device in FreeBSD when used with my paticular OpenVPN/CARP/pf combination.
AFAIK pf/forwarding only takes place on one core and wouldn't take advantage of the other 3 cores, correct?
-Patrick
http://pf4freebsd.love2party.net/pflock/ is worth a quick read. 7.0 already supports some SMP networking but when the pflock changes are done you'll likely see some pretty serious performance from those devices. Regards, Chris
* Patrick Clochesy <patrick@chegg.com> [2008-03-26 02:26]:
I also had to switch to OpenBSD
congrats
AFAIK pf/forwarding only takes place on one core and wouldn't take advantage of the other 3 cores, correct?
for the moment, yes. -- Henning Brauer, hb@bsws.de, henning@openbsd.org BS Web Services, http://bsws.de Full-Service ISP - Secure Hosting, Mail and DNS Services Dedicated Servers, Rootservers, Application Hosting - Hamburg & Amsterdam
http://docs.rodecker.nl/10-GE_Routing_on_Linux.pdf. He hit a wall at 700K pps and was using two dual core Intel Xeon 64bit 2.33GHz CPUs and 2GB of RAM in a Dell PowerEdge 1950.
Unless I am misreading this, he did not hit a wall. What he did was test a design that was scalable to multiple cores and show that the two core version could not go beyond 700k pps. The next logical question is how much more can you push with larger numbers of cores. The key thing is to use a recent Linux kernel that can share interrupts among multiple cores and to run it on a CPU using MSI interrupts. Since this was written up in January of 2007, There are people who use Linux for load balancing who also are working on finding how well it can cope with 10G of traffic and they have some anecdotal evidence of 800k pps. --Michael Dillon
On Mar 26, 2008, at 11:57 AM, <michael.dillon@bt.com> wrote:
http://docs.rodecker.nl/10-GE_Routing_on_Linux.pdf. He hit a wall at 700K pps and was using two dual core Intel Xeon 64bit 2.33GHz CPUs and 2GB of RAM in a Dell PowerEdge 1950.
Unless I am misreading this, he did not hit a wall. What he did was test a design that was scalable to multiple cores and show that the two core version could not go beyond 700k pps. The next logical question is how much more can you push with larger numbers of cores. The key thing is to use a recent Linux kernel that can share interrupts among multiple cores and to run it on a CPU using MSI interrupts. Since this was written up in January of 2007,
There are people who use Linux for load balancing who also are working on finding how well it can cope with 10G of traffic and they have some anecdotal evidence of 800k pps.
--Michael Dillon
If I just may share my opinion on this whole Software Router debate. Even if it is technically feasible to route traffic over a server, I would not hesitate to sound old-fashioned and state that it is not a server's main role, i.e. what it is designed for. Mainly, I would assume that you'd get the same Network I/O issues with small packets that Disk I/O you would notice in a strictly systems/ server environment. Most of all, Routing Equipment manufacturers offer more than a physical routing chassis, they offer Hardware and Software support and that I say, is essential - if you want open source in your routing devices, I'd suggest you pick Juniper, their OS is BSDdey - you'll love it, plus they will provide you with support, which good or bad, will be better than none in times where you'll be stuck with an undocumented memory leak of your favorite open source software routers. It is not about making it work, it is about having it work -all the time-, even if it is more costly, even if YOU have failed troubleshooting a crash, SOMEONE will be forced to help you, by contract. Risk assessment folks, risk assessment... Greg VILLAIN Independant Network & Telco Architecture Consultant Greg VILLAIN Independant Network & Telco Architecture Consultant +33 6 87 48 66 14
On 26/03/2008, at 10:23 PM, Greg VILLAIN wrote:
It is not about making it work, it is about having it work -all the time-,
Hey, that sounds fantastic. Can you let me know where I can get one of these platforms that works -all the time-? Because the ones I have now crash occasionally, which is inconvenient. And untidy. Surely the thing that needs to work all the time is the network, not any individual forwarding element within the network. Design so that a software or hardware fault in a commodity-OS router doesn't take down the whole network, then you can leave any serious outages until next business day. That's how we're meant to do things. Right? Considering that all major vendors are using open-source OS's as the embedded microkernel of choice and running their "operating system" as an application (anyone have any ACE blades in 6500's?), I'm not convinced by any FUD that says open source OS's aren't suitable for routers. All we're really talking about here is the depth of the abstractions that implement the features we need; whether they're on dedicated silicon, custom- designed hardware, or a PC doesn't matter at all if they all meet customer-stated requirements for performance and reliability. - mark -- Mark Newton Email: newton@internode.com.au (W) Network Engineer Email: newton@atdot.dotat.org (H) Internode Systems Pty Ltd Desk: +61-8-82282999 "Network Man" - Anagram of "Mark Newton" Mobile: +61-416-202-223
Even if it is technically feasible to route traffic over a server,
A computer running Linux is not a server. If it's job is to route and forward packets, then it is a router. And the fact that people are pushing the envelope at 10G is because Linux routers are already known to do a great job at 1G and below.
It is not about making it work, it is about having it work -all the time-,
You can't buy "...having it work -all the time-". The only way to achieve that is by building in proper redundancy, probably N+2 or more, and by paying attention to all the deployment and operational nits that equipment vendors will not do for you. Companies like Google have demonstrated that this can be very effective with cheap components failing all the time.
even if it is more costly, even if YOU have failed troubleshooting a crash, SOMEONE will be forced to help you, by contract.
Contracts never force anyone to help you. They only force the vendor to pay a penalty when they fail to help you. When miracles are needed, you had better design it into the system yourself, not rely on a vendor to provide them.
Risk assessment folks, risk assessment...
Indeed! If you want to cover your behind, then buy the big brand names and forget about the consequences. If you want to operate a network service that functions all the time, then make sure you control and mitigate all the risk elements. Don't assume that a big brand name will do it for you. In particular, big brand names focus their effort on the majority customers so if you want to do something a little bit different, you will find that the big brand name won't be able to help you until next year, and that is assuming that their priorities don't shift your request right off the list. --Michael Dillon
michael.dillon@bt.com wrote:
Even if it is technically feasible to route traffic over a server,
A computer running Linux is not a server. If it's job is to route and forward packets, then it is a router. And the fact that people are pushing the envelope at 10G is because Linux routers are already known to do a great job at 1G and below.
Just because you can do something doesn't mean that you should. http://katcampbell.files.wordpress.com/2007/07/overload.jpg http://www.gpsa.co.za/Jokes/OVERLOAD.JPG http://englishrussia.com/images/overload.jpg http://www.tensionnot.com/images/images/Automobiles587.jpg Justin
On Wed, Mar 26, 2008, Justin Shore wrote:
A computer running Linux is not a server. If it's job is to route and forward packets, then it is a router. And the fact that people are pushing the envelope at 10G is because Linux routers are already known to do a great job at 1G and below.
Just because you can do something doesn't mean that you should.
Hands up those of you running Cat6500's in service provider environments. Adrian
* Justin Shore
Just because you can do something doesn't mean that you should.
* Adrian Chadd
Hands up those of you running Cat6500's in service provider environments.
*hand* Actually, not quite yet, but I'm considering purchasing a pair of Cat6500's (with Sup720 PFC3CXL) for a new colo I'm setting up, bundling them together with VSS. They'll terminate a few transit links and links to other colos, in addition to functioning as distribution/access switches for the data center itself. Are you saying that there's something about the Cat6500's that makes them unsuitable for such usage? I'd sure like to hear more about that before I go ahead and buy them, if so. Up until now I've been using whiteboxes with Linux and Quagga for the layer 3 services. This setup has served us well, but it seems we're starting to approach a performance limit at around 1 Gbps routed traffic so we need new gear anyway - a good time to start doing routing in hardware, I thought. Regards -- Tore Anderson
At 04:33 AM 3/27/2008, Tore Anderson wrote:
Hands up those of you running Cat6500's in service provider environments.
*hand*
Actually, not quite yet, but I'm considering purchasing a pair of Cat6500's (with Sup720 PFC3CXL) for a new colo I'm setting up, bundling them together with VSS. They'll terminate a few transit links and links to other colos, in addition to functioning as distribution/access switches for the data center itself.
Are you saying that there's something about the Cat6500's that makes them unsuitable for such usage? I'd sure like to hear more about that before I go ahead and buy them, if so.
Cisco wants you to pay 4 times as much for the 7600 which is the same platform except the cards are vertical instead of horizontal. (If you have a NEBS chassis, then that's not even a differentiator.) Oh, there is also a ROM/PROM/Flash chip in the chassis which tells IOS that you are on a Catalyst and not a 7600 so the newer 7600 IOS code supposedly won't work. This is the "code split" which they did about a year? ago. The Catalyst works great as a core router, but Cisco says that's the job for a 7600, not a 6500. I don't know if there are any other differences, FlexWAN card support? But for most of us, the 6500 works great and does everything we need. That's what the OP was referring to I believe. -Robert Tellurian Networks - Global Hosting Solutions Since 1995 http://www.tellurian.com | 888-TELLURIAN | 973-300-9211 "Well done is better than well said." - Benjamin Franklin
Robert Boyle wrote:
At 04:33 AM 3/27/2008, Tore Anderson wrote:
Hands up those of you running Cat6500's in service provider environments.
*hand*
Actually, not quite yet, but I'm considering purchasing a pair of Cat6500's (with Sup720 PFC3CXL) for a new colo I'm setting up, bundling them together with VSS. They'll terminate a few transit links and links to other colos, in addition to functioning as distribution/access switches for the data center itself.
Are you saying that there's something about the Cat6500's that makes them unsuitable for such usage? I'd sure like to hear more about that before I go ahead and buy them, if so.
Cisco wants you to pay 4 times as much for the 7600 which is the same platform except the cards are vertical instead of horizontal. (If you have a NEBS chassis, then that's not even a differentiator.) Oh, there is also a ROM/PROM/Flash chip in the chassis which tells IOS that you are on a Catalyst and not a 7600 so the newer 7600 IOS code supposedly won't work. This is the "code split" which they did about a year? ago. The Catalyst works great as a core router, but Cisco says that's the job for a 7600, not a 6500. I don't know if there are any other differences, FlexWAN card support? But for most of us, the 6500 works great and does everything we need. That's what the OP was referring to I believe.
6708 linecards aren't supported in 7600s (though I think that was supposed to change in SRC, which I'm not running yet because its reported to be buggy as hell). Support for the ES linecards is only found in the 7600 chassis too which has certain implications (some critical) if you're doing MPLS. One thing that the 6500 can't do and never will be able to do is CALEA. If you're a SP and have to have support for CALEA as required by law (and can't get it closer to the edge) then you should put some more thought into the 7600s or some other solution. Yesterday Gert posted an interesting take to the C-NSP list on the Business Unit split that formally separated the 6500s from the 7600s. http://puck.nether.net/pipermail/cisco-nsp/2008-March/049082.html I'm not advocating one over the other but I am advocating a thorough examination of one's needs, wants and requirements before buying one or the other. We (and every other US SP) are required by law to support CALEA for all broadband subs. Since we couldn't do it on our edge we were forced to do it in the core. That required us to run SR and SR made us buy 7600s. Others may not have that need. Justin
Hi IMHO, people don't want to do this way when they have the budget. just Oracle VS Mysql I admit open source is good. but just for employer --- Justin Shore <justin@justinshore.com> wrote:
michael.dillon@bt.com wrote:
Even if it is technically feasible to route
traffic over a
server,
A computer running Linux is not a server. If it's job is to route and forward packets, then it is a router. And the fact that people are pushing the envelope at 10G is because Linux routers are already known to do a great job at 1G and below.
Just because you can do something doesn't mean that you should.
http://katcampbell.files.wordpress.com/2007/07/overload.jpg
http://www.gpsa.co.za/Jokes/OVERLOAD.JPG http://englishrussia.com/images/overload.jpg
http://www.tensionnot.com/images/images/Automobiles587.jpg
Justin
____________________________________________________________________________________ Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ
Greg VILLAIN wrote:
It is not about making it work, it is about having it work -all the time-, even if it is more costly, even if YOU have failed troubleshooting a crash, SOMEONE will be forced to help you, by contract. Risk assessment folks, risk assessment...
Greg VILLAIN Independant Network & Telco Architecture Consultant
In my experience this is almost entirely the opposite. They're only forced to help you if you're about to buy more gear from them and you're holding that over their head. There have been times where I've been the one forced to find the solution and present it to the vendor. They had no "on the books" solution to what was, in the end, truly a fairly simple problem. Moral of the story, you don't always get what you pay for. Regards, Chris
On Wed, Mar 26, 2008, michael.dillon@bt.com wrote:
http://docs.rodecker.nl/10-GE_Routing_on_Linux.pdf. He hit a wall at 700K pps and was using two dual core Intel Xeon 64bit 2.33GHz CPUs and 2GB of RAM in a Dell PowerEdge 1950.
Unless I am misreading this, he did not hit a wall. What he did was test a design that was scalable to multiple cores and show that the two core version could not go beyond 700k pps. The next logical question is how much more can you push with larger numbers of cores. The key thing is to use a recent Linux kernel that can share interrupts among multiple cores and to run it on a CPU using MSI interrupts. Since this was written up in January of 2007,
There are people who use Linux for load balancing who also are working on finding how well it can cope with 10G of traffic and they have some anecdotal evidence of 800k pps.
I didn't think the hardware quite worked like that :) The paper doesn't cover -why- he hit a limit on a single core and why two cores are any faster. He didn't do any benchmarking, no oprofile traces, etc. What would be much more interesting is to see where its running out of steam, and why more L1 cache helps. The AMD/Intel difference could be due to how the memory systems operate/differ, but its all conjecture from me at this point. I haven't looked into it in depth. Just a random datapoint, some FreeBSD related people working on commercial systems have noted they were able to achieve 1mil pps on intel gige hardware. Its just not in open source. :) Adrian
Is there a multiport card out there on to which some of the forwarding responsibilities can be offloaded? Perhaps the CPU doesn't need to see every packet that arrives on the machine. -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Adrian Chadd Sent: Wednesday, March 26, 2008 09:30 To: michael.dillon@bt.com Cc: nanog@nanog.org Subject: Re: 10GE router resource On Wed, Mar 26, 2008, michael.dillon@bt.com wrote:
http://docs.rodecker.nl/10-GE_Routing_on_Linux.pdf. He hit a wall at 700K pps and was using two dual core Intel Xeon 64bit 2.33GHz CPUs and 2GB of RAM in a Dell PowerEdge 1950.
Unless I am misreading this, he did not hit a wall. What he did was test a design that was scalable to multiple cores and show that the two core version could not go beyond 700k pps. The next logical question is how much more can you push with larger numbers of cores. The key thing is to use a recent Linux kernel that can share interrupts among multiple cores and to run it on a CPU using MSI interrupts. Since this was written up in January of 2007,
There are people who use Linux for load balancing who also are working on finding how well it can cope with 10G of traffic and they have some anecdotal evidence of 800k pps.
I didn't think the hardware quite worked like that :) The paper doesn't cover -why- he hit a limit on a single core and why two cores are any faster. He didn't do any benchmarking, no oprofile traces, etc. What would be much more interesting is to see where its running out of steam, and why more L1 cache helps. The AMD/Intel difference could be due to how the memory systems operate/differ, but its all conjecture from me at this point. I haven't looked into it in depth. Just a random datapoint, some FreeBSD related people working on commercial systems have noted they were able to achieve 1mil pps on intel gige hardware. Its just not in open source. :) Adrian -- Scanned for viruses and dangerous content at http://www.oneunified.net and is believed to be clean. -- Scanned for viruses and dangerous content at http://www.oneunified.net and is believed to be clean.
Is there a multiport card out there on to which some of the forwarding responsibilities can be offloaded? Perhaps the CPU doesn't need to see every packet that arrives on the machine.
Am I the only person who has heard of Google? It didn't take me long to find this wiki page http://www.bro-ids.org/wiki/index.php/ClusterFrontends for an Opensource Intrusion Detection System that lists various 10G cards for Linux and a couple of FPGA cards so that you can roll your own ASICs. Anyway, this one http://www.lewiz.com/talon3220.html has two ports and claims to reach 8.8 Gbps with 1500 byte packets. People rolling their own router are not the only ones who want to do 10G on Linux. --Michael Dillon
michael.dillon@bt.com writes:
People rolling their own router are not the only ones who want to do 10G on Linux.
speaking of which, has anybody run "xorp" in production? it looks as much like JunOS as quagga/zebra looks like IOS. if "click" works on current hardware and if the xorp/click integration is good, this could be a great science fair project for smaller network operators who need big PPS. -- Paul Vixie
Paul Vixie wrote:
michael.dillon@bt.com writes:
People rolling their own router are not the only ones who want to do 10G on Linux.
speaking of which, has anybody run "xorp" in production? it looks as much like JunOS as quagga/zebra looks like IOS. if "click" works on current hardware and if the xorp/click integration is good, this could be a great science fair project for smaller network operators who need big PPS.
Vyatta is built on top of xorp. You can download the bootable iso from their site and take a low-commitment look: http://www.vyatta.com/download/index.php --Peter
Actually the latest version of Vyatta uses Quagga. If anyone is interested in discussing the differences in running the two in production networks feel free to contact me off list. In full disclosure, I work for Vyatta. Cheers, Robert. Peter Wohlers wrote:
Vyatta is built on top of xorp. You can download the bootable iso from their site and take a low-commitment look: http://www.vyatta.com/download/index.php
--Peter
Actually, soon this will no longer be true. Vyatta's new platform, Glendale, will be moving to Quagga. Quagga is much more stable, and slow-moving compared to Xorp, which makes me slightly more comfortable (less breakage between versions). There are some major features lacking inside of the platform. For example, it lacks the ability to do BFD, BGP over IPSec, Multicast, etc... This major lack of features makes this a hard to deploy piece of software. I am sure with enough customers Vyatta would be able to catch up to Cisco. Also, from a viewpoint of hardware, x86 is a fairly decent platform. I can stuff 40 (4x10GigE multiplex with a switch) 1 GigE ports in it. Though, the way that Linux works, it cannot handle high packet rates. If you are planning on handling large flows with mostly large packets, you are alright for the most part. Just be warned. Peter Wohlers wrote:
Paul Vixie wrote:
michael.dillon@bt.com writes:
People rolling their own router are not the only ones who want to do 10G on Linux.
speaking of which, has anybody run "xorp" in production? it looks as much like JunOS as quagga/zebra looks like IOS. if "click" works on current hardware and if the xorp/click integration is good, this could be a great science fair project for smaller network operators who need big PPS.
Vyatta is built on top of xorp. You can download the bootable iso from their site and take a low-commitment look: http://www.vyatta.com/download/index.php
--Peter
-- +1.925.202.9485 Sargun Dhillon deCarta sdhillon@decarta.com www.decarta.com
On Wed, Mar 26, 2008 at 4:26 PM, Sargun Dhillon <sdhillon@decarta.com> wrote:
from a viewpoint of hardware, x86 is a fairly decent platform. I can stuff 40 (4x10GigE multiplex with a switch) 1 GigE ports in it. Though, the way that Linux works, it cannot handle high packet rates.
Correction: The way DRAM works, it cannot handle high packet rates. Also note that the PCI-X bus tops out in the 7 to 8 gbps range and it's half-duplex. High-rate routers try to keep the packets in an SRAM queue and instead of looking up destinations in a DRAM-based radix tree, they use a special memory device called a TCAM. http://www.pagiamtzis.com/cam/camintro.html Regards. Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
I wonder how difficult it would be to integrate such a device on to an x86 board cheaply. Something like NetFPGA (http://netfpga.org/) would be an interesting place to start. The board has on board SRAM, a bit of DRAM, an FPGA, and 2 GigE interfaces. I know it definitely isn't normal for Network Operators to fund research like this, but it would still be fairly interesting if there was an Open Router Consortium (something for Vyatta to start?) with hardware acceleration to X86 routers. Possibly even making Quagga a mainstream control plane. Right now Quagga is controlled by a few engineers from Sun. This nearly produces a conflict on interest (Sun used to have their own routing platform). Anyways, to end my rambling... As network operators would you finance a low, medium end router with decent ROI. The question for developers (Vyatta primarily), could you do what Digium did for Asterisk--become business front, and provide platforms for Asterisk deployment in the enterprise--for Quagga, Linux, etc? William Herrin wrote:
On Wed, Mar 26, 2008 at 4:26 PM, Sargun Dhillon <sdhillon@decarta.com> wrote:
from a viewpoint of hardware, x86 is a fairly decent platform. I can stuff 40 (4x10GigE multiplex with a switch) 1 GigE ports in it. Though, the way that Linux works, it cannot handle high packet rates.
Correction: The way DRAM works, it cannot handle high packet rates. Also note that the PCI-X bus tops out in the 7 to 8 gbps range and it's half-duplex.
High-rate routers try to keep the packets in an SRAM queue and instead of looking up destinations in a DRAM-based radix tree, they use a special memory device called a TCAM.
http://www.pagiamtzis.com/cam/camintro.html
Regards. Bill Herrin
-- +1.925.202.9485 Sargun Dhillon deCarta sdhillon@decarta.com www.decarta.com
On Wed, Mar 26, 2008 at 6:54 PM, Sargun Dhillon <sdhillon@decarta.com> wrote:
I wonder how difficult it would be to integrate such a device on to an x86 board cheaply. Something like NetFPGA (http://netfpga.org/) would be an interesting place to start. The board has on board SRAM, a bit of DRAM, an FPGA, and 2 GigE interfaces.
Hi Sargun, SRAM != TCAM. With SRAM you can only access one word per cycle. The coolness of the TCAM is that the entire memory is queried in one cycle, spitting out the best match. Nevertheless, there is some interesting hardware out there. The Endace DAG card with the coprocessor has a TCAM on it, but it's not big enough to handle a full BGP table. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
High-rate routers try to keep the packets in an SRAM queue and instead of looking up destinations in a DRAM-based radix tree, they use a special memory device called a TCAM.
FPGAs can be used to do both SRAM and TCAMs. All that is needed is an FPGA board with 10G or a 10G card with an FPGA on it. Although NetFPGA and RiceNIC are both 1G devices, there is a certain commercial market for programmable high-speed network cards for things like Intrusion Detection and data-center/GRID type applications. Anyone seriously interested in this area should start hunting amongst the developers (and researchers) of embedded systems. You might end up working with a university student in the Czech Republic to put his TCAM/FPGA implementation onto a 10G card because the Internet breaks down the barriers that high-margin vendors have used to create lock-in. Bleeding edge networks may not be able to do this type of deal but then, they are only 1% or less of the network operators out there. --Michael Dillon
FPGAs can be used to do both SRAM and TCAMs. All that is needed is an FPGA board with 10G or a 10G card with an FPGA on it.
The Xilinx Virtex family can already do 10G, if you are into FPGA development (I seem to recall the first Xilinx FPGA that could do 10G was 4-5 years ago; forever in Moore's law). Other vendors have equivalent parts. And the Xilinx family has an available PowerPC core. I seem to recall a couple of vendors making available a (micro)Linux kernel for running on same. All the hardware you need for building your own 10G router. Just add FPGA development resources, some planar board design, and software.
William Herrin wrote:
On Wed, Mar 26, 2008 at 4:26 PM, Sargun Dhillon <sdhillon@decarta.com> wrote:
from a viewpoint of hardware, x86 is a fairly decent platform. I can stuff 40 (4x10GigE multiplex with a switch) 1 GigE ports in it. Though, the way that Linux works, it cannot handle high packet rates.
Correction: The way DRAM works, it cannot handle high packet rates. Also note that the PCI-X bus tops out in the 7 to 8 gbps range and it's half-duplex.
Indeed. PCI-X is already an EOL'ed interface, if only cheap PCI-X cards were available. Once you add extensive ACL's, there's loads more [central] processing to be done than just packet routing (100k choices versus 2 to 4 interfaces). System throughput gets slammed rather quickly. Linux IPtables grumbles painfully at 100k line ACLs :) Not to mention the options of what to do with a packet are very limited. The AMD chips with extra L1 cache perform better on *bsd platforms as the forwarding code is tight and likes to stay close to the CPU, and context switching kills packet processing performance (thus the small but notable increase in the multicore performance). The GP registers on the AMD platform are also easy to deal with (and in 64 bit mode, you get double the number for free) essentially working an end around a broken stack architecture from a few decades ago....anyone recall the simplicity of assembly language of the 6800 or the 6502? :-) getting the latency down low enough for HPC clusters is a major hassle, as the x86 PC design just doesn't have the bandwidth. Of course, Intel makes some slick NPU's for custom work (e.g. cloudshield.com). If you like starting at bit 0. (isn't that like slot zero or port zero, it technically doesn't exist since zero is only a placeholder in larger numbers if you mean anything greater than none? I could swear back in the days of a SLC96, ports were 1-96, not 0-95 :-) ) http://developer.intel.com/design/network/products/npfamily/index.htm?iid=nc... too bad they [Intel] don't make a hypertransport capable version, or you'd have one helluva multicore multiNPU system with no glue logic required. Fun to play around though. regards, andy
High-rate routers try to keep the packets in an SRAM queue and instead of looking up destinations in a DRAM-based radix tree, they use a special memory device called a TCAM.
http://www.pagiamtzis.com/cam/camintro.html
Regards. Bill Herrin
On Thu, Mar 27, 2008, Andrew C Burnette wrote:
Indeed. PCI-X is already an EOL'ed interface, if only cheap PCI-X cards were available. Once you add extensive ACL's, there's loads more [central] processing to be done than just packet routing (100k choices versus 2 to 4 interfaces). System throughput gets slammed rather quickly. Linux IPtables grumbles painfully at 100k line ACLs :) Not to mention the options of what to do with a packet are very limited.
I agree, and the rest of the discussion is interesting, but the iptables deployments I've seen which do massive ACLs like this almost certainly end up having ACLs you can collapse into a small number of set-lookup-and-act rules. Those set-lookup-and-act rules are much faster than the linear ACL lookups which ipfw/iptables/ipf/pf/etc do by default (and all of them support IP sets in some form or other); I did this trick recently to reduce the CPU overhead on an old revision 2.8ghz P4 from 99% to <10% when routing 100mbit of average-pps TCP. Adrian
On Tue, Mar 25, 2008 at 1:59 PM, Chris Grundemann <cgrundemann@gmail.com> wrote:
Greg has laid out a great bit of information and I would like to add just one possibility to the list of budget 10GE routers: Vyatta. According to a recent press release from that company (http://www.vyatta.com/about/pressreleases.php?id=51) they offer a product that is "2 to 3X higher performance at a cost savings of more than 75 percent" when compared to Cisco's 7200. Unfortunately I have not had the
when did the 7200 go 10ge?
At 12:36 PM 3/25/2008, Greg VILLAIN wrote: I'd strongly suggest Foundry, I'm a big fan of their kits, price-wise
and performance-wise, provided you do not need rocket-science features. MLX/XMR models will surely do the trick perfectly.
I agree too. They still have a bit of development to do on the IPv6 side, but they are getting there. We are using them with Cat 65XXs with SXF Sup720-3BXLs and XMRs. We run ISIS, BGP, and BFD. Everything they say works really does. We have been very pleased. Definitely put them on your short list. The price per port can't be beat and their support is stellar. If you want to reliably route IPv4 and IPv6 at wire speeds regardless of packet size or rate and optionally filter at wire speed too on all ports then they make a great box. -Robert Tellurian Networks - Global Hosting Solutions Since 1995 http://www.tellurian.com | 888-TELLURIAN | 973-300-9211 "Well done is better than well said." - Benjamin Franklin
On Mar 25, 2008, at 1:42 PM, Robert Boyle wrote:
At 12:36 PM 3/25/2008, Greg VILLAIN wrote: I'd strongly suggest Foundry, I'm a big fan of their kits, price-wise
and performance-wise, provided you do not need rocket-science features. MLX/XMR models will surely do the trick perfectly.
I agree too. They still have a bit of development to do on the IPv6 side, but they are getting there. We are using them with Cat 65XXs with SXF Sup720-3BXLs and XMRs. We run ISIS, BGP, and BFD. Everything they say works really does. We have been very pleased. Definitely put them on your short list. The price per port can't be beat and their support is stellar. If you want to reliably route IPv4 and IPv6 at wire speeds regardless of packet size or rate and optionally filter at wire speed too on all ports then they make a great box.
-Robert
Totally agree. Foundry support is top notch and the boxes do deliver the promised performance. The headroom is impressive when the CPU is at 99%. Somehow *cough* we (me) pegged the CPU on the Server Irons and still had a very very responsive console. Was able to find the self inflicted error and fix the problem quickly. Out testers on the outside say they did not notice a performance degradation. Foundry's performance and support make the price a clear value. I've only experienced two flavors, Cisco and Foundry. Eddy
participants (34)
-
Aaron Glenn -
Adrian Chadd -
Alex Rubenstein -
Andrew C Burnette -
Andy Dills -
ann kok -
Buhrmaster, Gary -
Chris Grundemann -
Chris Marlatt -
Christopher Morrow -
Eddy Martinez -
Fred Reimer -
Greg VILLAIN -
Henning Brauer -
Joel Jaeggli -
Joel Snyder -
Justin Shore -
Kevin Oberman -
Lamar Owen -
Lincoln Dale -
Mark Newton -
Mark Tinka -
michael.dillon＠bt.com -
Patrick Clochesy -
Patrick Giagnocavo -
Paul Vixie -
Peter Wohlers -
Ray Burkholder -
Robert Bays -
Robert Boyle -
Sargun Dhillon -
Tore Anderson -
user user -
William Herrin