Re: NPE-G2 vs. Sup720-3BXL
We need true full routes and more CPU horsepower for crunching BGP (+12 smaller peers + ISIS). OC3 interfaces are going to be mandatory, one each at two locations. Oh yeah, we're still a larger startup without endless pockets. Power, rack space, and SmartNet are not concerns at any location (on-site cold spares). We may need an upstream OC12 in the future but that's a ways out and not a concern here.
Our engineering team has settled on three $20k/node options: - Sup720-3BXLs with PS and fan upgrades - Sup2s as switches + ISIS + statics and no BGP, push BGP edge routing off to NPE-G2s across a 2-3Gbps port-channel - Sup2s as switches + ISIS + statics and no BGP, push BGP edge routing off to a 12008 with E3 engines across a 2-3Gbps port-channel.
Ideas and constructive opinions welcome, especially software and stability-related.
For about $6k all in, you could pickup a monster dual Xeon server with a few 10GE PCI line cards and run a subscription service of the Vyatta open source router. With high end machine specs, we've been able to run 5 full tables and a solid amount of peers with about 6.5Gbps sustained to the net without any stress. For access, we just trunk one of the PCI cards down to a 6506 or a 3750 and it runs nice and clean. The only downside to this setup is the lack of cisco proprietary software features which it sounds like you might need. If anything you might be able to keep your existing setup and uplink everything to one of these routers as an edge device. Adam _______________ Adam LaFountain
So I figure a summary is an order, with a whole array of choices pitched so far... - Sup720-3BXL works for light-duty premium ISP services, decent CPU for BGP and an Ethernet hardware throughput monster. Decent enough for our deployment scenario at least. No obvious solution for the FlexWAN/OC3 but could easily be re-integrated with a stronger MSFC CPU to back it up, assuming the IOS-of-the-week doesn't have issues. The pesky OC3 could be pawned off to a dedicated G1/G2 router too along with any oddball <=OC3 stuff our sales guys dream up. - RSP720-3CXL is the best of all worlds option, if we had double the budget to work with. Meh. - ASR1002 is a hardware-assisted overhaul to the 7200/G2. Telco interface options are much better than 7200s, good for OC12s and OC48s. Using GoogleFu product pricing... a ASR1002 router with a SPA OC3, 5Gbps ESP, and base software runs in the $28-30k range + SmartNet. Beware the modular licensing model in addition to IOS editions. Maybe a bit early yet as a core router as some of the software is still getting bugs ironed out. - Vyatta was proposed as an alternative system, probably best architected out of the mainstream traffic flows (no hardware forwarding), say a BGP route reflector or GBE edge router, similar argument to a 7200/G[1|2]. I can't say I'm familiar with the software, but the cost savings of premium x86/x64 hardware and 8x PCI-x serving a few 10GBE interfaces + built-in GBEs is intriguing, especially paired against our budget and relative Cisco costs. A spec'd out 1U Dell box with dual power, 8x cores, 4GB, RAID1 SATA, and 2x 10GBE XFP+2x GBE built-in came in under $7k with CPU headroom to burn. Vyatta doesn't support ISIS though, best I can tell, but may not have to... Maybe yet-another Linux router distro doomed to fail? Worth a lab test internally on some demo hardware. - Mixed thoughts about 7304 hardware. Hardware forwarding quality vs. software and interface selection. - Lots of fans for the 12000 series. Stick with the E3 (~2.5Gbps) and E5 (~10Gbps) line cards for compatibility with XR software and best line card performance. Our team liked the variety of SONET options available too for our central office deployments, even though the systems are power and space hungry. ...and if you can afford them (the 12008/GRP-B being the relative exception). - 7200/G2s are great for <1Gbps throughput. Premium services cut into the performance dramatically, being a fully software-based forwarding platform. Don't bond interfaces looking for more throughput, architecture limitations actually decrease throughput. - Juniper MX series? A budget wildcard but indeed a worthy platform engineering-wise. You could break this list into "routers" and "switches", which in itself spurs the philosophical/pragmatic architecture discussion that got us the impasse to start with. Many thanks to all who've responded with real-life successes, battle wounds, and horror stories. All very helpful. -Dave
On Fri, 15 May 2009 22:20:28 EDT, David Storandt said:
- Vyatta was proposed as an alternative system, probably best architected out of the mainstream traffic flows (no hardware forwarding), say a BGP route reflector or GBE edge router, similar argument to a 7200/G[1|2]. I can't say I'm familiar with the software, but the cost savings of premium x86/x64 hardware and 8x PCI-x serving a few 10GBE interfaces + built-in GBEs is intriguing, especially paired against our budget and relative Cisco costs. A spec'd out 1U Dell box with dual power, 8x cores, 4GB, RAID1 SATA, and 2x 10GBE XFP+2x GBE built-in came in under $7k with CPU headroom to burn.
Did you check PCI bus bandwidth? That's probably going to be the biggest constraint on "a few 10GBE interfaces" if they all get going full blast. Remember that each packet is going to burn bandwidth twice - once in and once out...
In a message written on Fri, May 15, 2009 at 10:25:12PM -0400, Valdis.Kletnieks@vt.edu wrote:
Did you check PCI bus bandwidth? That's probably going to be the biggest constraint on "a few 10GBE interfaces" if they all get going full blast. Remember that each packet is going to burn bandwidth twice - once in and once out...
PCIe, x8 or x16, which is serial point to point. http://www.csc.kth.se/~olofh/10G_OSR/10Gbps.pdf 25 Gb/sec across 4x10G ports on higher end but far from topped out hardware. -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/
New architectures might be helpful to achieve such throughput e.g. Myricom pci-e Gen2 10GE cards on new Intel Nehalem based servers. -Azher Leo Bicknell wrote:
In a message written on Fri, May 15, 2009 at 10:25:12PM -0400, Valdis.Kletnieks@vt.edu wrote:
Did you check PCI bus bandwidth? That's probably going to be the biggest constraint on "a few 10GBE interfaces" if they all get going full blast. Remember that each packet is going to burn bandwidth twice - once in and once out...
PCIe, x8 or x16, which is serial point to point.
http://www.csc.kth.se/~olofh/10G_OSR/10Gbps.pdf
25 Gb/sec across 4x10G ports on higher end but far from topped out hardware.
On Fri, 15 May 2009, Leo Bicknell wrote:
PCIe, x8 or x16, which is serial point to point.
http://www.csc.kth.se/~olofh/10G_OSR/10Gbps.pdf
25 Gb/sec across 4x10G ports on higher end but far from topped out hardware.
further illustrating the point - 10gige ~linerate load balancing on a single core2 e8200 using haproxy + myricom 10gige cards: http://haproxy.1wt.eu/10g.html -jba
ASR is embedded linux solution with Quantum Processor architect if I remember correctly. So it uses IOS-XE, which is a little bit different from standard IOS. If you have some room for budget, you can check Foundry MLX/XMR series router. It is more geared toward Ethernet Service Router. But if you need OC3/12/48, you can have those with additional license fee. Foundry router price is a lot lower than Juniper MX series router. Alex David Storandt wrote:
So I figure a summary is an order, with a whole array of choices pitched so far...
- Sup720-3BXL works for light-duty premium ISP services, decent CPU for BGP and an Ethernet hardware throughput monster. Decent enough for our deployment scenario at least. No obvious solution for the FlexWAN/OC3 but could easily be re-integrated with a stronger MSFC CPU to back it up, assuming the IOS-of-the-week doesn't have issues. The pesky OC3 could be pawned off to a dedicated G1/G2 router too along with any oddball <=OC3 stuff our sales guys dream up. - RSP720-3CXL is the best of all worlds option, if we had double the budget to work with. Meh. - ASR1002 is a hardware-assisted overhaul to the 7200/G2. Telco interface options are much better than 7200s, good for OC12s and OC48s. Using GoogleFu product pricing... a ASR1002 router with a SPA OC3, 5Gbps ESP, and base software runs in the $28-30k range + SmartNet. Beware the modular licensing model in addition to IOS editions. Maybe a bit early yet as a core router as some of the software is still getting bugs ironed out. - Vyatta was proposed as an alternative system, probably best architected out of the mainstream traffic flows (no hardware forwarding), say a BGP route reflector or GBE edge router, similar argument to a 7200/G[1|2]. I can't say I'm familiar with the software, but the cost savings of premium x86/x64 hardware and 8x PCI-x serving a few 10GBE interfaces + built-in GBEs is intriguing, especially paired against our budget and relative Cisco costs. A spec'd out 1U Dell box with dual power, 8x cores, 4GB, RAID1 SATA, and 2x 10GBE XFP+2x GBE built-in came in under $7k with CPU headroom to burn. Vyatta doesn't support ISIS though, best I can tell, but may not have to... Maybe yet-another Linux router distro doomed to fail? Worth a lab test internally on some demo hardware. - Mixed thoughts about 7304 hardware. Hardware forwarding quality vs. software and interface selection. - Lots of fans for the 12000 series. Stick with the E3 (~2.5Gbps) and E5 (~10Gbps) line cards for compatibility with XR software and best line card performance. Our team liked the variety of SONET options available too for our central office deployments, even though the systems are power and space hungry. ...and if you can afford them (the 12008/GRP-B being the relative exception). - 7200/G2s are great for <1Gbps throughput. Premium services cut into the performance dramatically, being a fully software-based forwarding platform. Don't bond interfaces looking for more throughput, architecture limitations actually decrease throughput. - Juniper MX series? A budget wildcard but indeed a worthy platform engineering-wise.
You could break this list into "routers" and "switches", which in itself spurs the philosophical/pragmatic architecture discussion that got us the impasse to start with. Many thanks to all who've responded with real-life successes, battle wounds, and horror stories. All very helpful.
-Dave
participants (7)
-
Adam LaFountain -
Alex H. Ryu -
Azher Mughal -
David Storandt -
jeffrey.arnold -
Leo Bicknell -
Valdis.Kletnieks＠vt.edu