Upgrade Path Options from 6500 SUP720-3BXL for Edge Routing
I’m curious what other providers have gone with when moving away from SUP720-3BXL 6500 platforms. I’m platform agnostic and just as comfortable with Juniper as with Cisco. It’s a conversation were having since the 3BXL’s are running into limits with the large number of prefixes, long eBGP convergence times, and 10G port density. Basically were looking to carry multiple full routing tables from several 4+ carriers plus internet exchange traffic so the ability to handle 1-2M IPV4 and 500K+ IPv6 and decent 10G port density and/or 40G options as well. Also should have decent CPU capabilities so it can crunch routes in a reasonable amount of time. Right now my thinking are MX480 or ASR9k platforms. Opinions on those are equally welcome as alternatives, but I’d love to hear from those with personal experiences today vs sales people trying to tell me it would route the world :) Thanks, Corey T
On Tue Jul 29, 2014 at 02:21:32AM +0000, Corey Touchet wrote:
Right now my thinking are MX480 or ASR9k platforms. Opinions on those are equally welcome as alternatives, but I?d love to hear from those with personal experiences today vs sales people trying to tell me it would route the world :)
Or, protect your existing investment in 6500 and replace the SUP720 with the SUP2T. You can then deploy the WS-X6904-40G-XL blades which give you 4 * 40G or 16 * 10G on a 80G backplane (i.e. 2:1 oversubscription). I'm in the process of going through this upgrade at the moment and I'm happy with what I'm seeing. A lot depends on the total traffic throughput you're looking to switch/route. You can then look to migrate onto the 6880 chassis which gives you a faster backplane, whilst retaining compatibility with existing linecards. Simon
We gave up and went to ASR9ks but that that was also a pretty big budget upgrade... -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Simon Lockhart Sent: Tuesday, July 29, 2014 3:57 PM To: Corey Touchet Cc: nanog@nanog.org Subject: Re: Upgrade Path Options from 6500 SUP720-3BXL for Edge Routing On Tue Jul 29, 2014 at 02:21:32AM +0000, Corey Touchet wrote:
Right now my thinking are MX480 or ASR9k platforms. Opinions on those are equally welcome as alternatives, but I?d love to hear from those with personal experiences today vs sales people trying to tell me it would route the world :)
Or, protect your existing investment in 6500 and replace the SUP720 with the SUP2T. You can then deploy the WS-X6904-40G-XL blades which give you 4 * 40G or 16 * 10G on a 80G backplane (i.e. 2:1 oversubscription). I'm in the process of going through this upgrade at the moment and I'm happy with what I'm seeing. A lot depends on the total traffic throughput you're looking to switch/route. You can then look to migrate onto the 6880 chassis which gives you a faster backplane, whilst retaining compatibility with existing linecards. Simon
On Tue, Jul 29, 2014 at 5:56 PM, Simon Lockhart <simon@slimey.org> wrote:
On Tue Jul 29, 2014 at 02:21:32AM +0000, Corey Touchet wrote:
Right now my thinking are MX480 or ASR9k platforms. Opinions on those are Or, protect your existing investment in 6500 and replace the SUP720 with the SUP2T. You can then deploy the WS-X6904-40G-XL blades which give you 4 * 40G
I would generally suggest you look at it as a long term decision, at least before jumping to the next incremental (modest increase) on the upgrade treadmill. It depends on whether the 6500 is still a perfect match for your network other than the prefix limit. Your vendor should think of your equipment as an "investment" to be protected, by exploiting your feelings of loss aversion, but the upgrade treadmill is a trap..... next thing you know, you will have to replace the chassis, then you will need new linecards...... Keep in mind most of the MX series makes the 6500 look like a 5 port linksys home router, when it comes to carrying around and managing large BGP tables; both in terms of prefix capacity, speed, the policy/filtering/configuration management functionality of the OS, and how they will take the route update "beating" during setup of new multiple BGP sessions... The SUP2T is about a 100% increase in TCAM size, but still pretty limited in terms of system resources. You can also "protect" your investment if appropriate by taking this late 1990s gear off your BGP edge, or otherwise recruiting it for a role which it is more suited for in this day and age, where it is not handling full tables and thus the feeble amount of FIB size, CPU, memory are no potential hinderance now or on the next 10 years. The ability to link up 40G ports did not seem terribly useful when it would all be unsafely oversubscribed.
You can then look to migrate onto the 6880 chassis which gives you a faster backplane, whilst retaining compatibility with existing linecards.
Simon
-- -JH
On Wednesday, July 30, 2014 03:06:55 PM Jimmy Hess wrote:
I would generally suggest you look at it as a long term decision, at least before jumping to the next incremental (modest increase) on the upgrade treadmill. It depends on whether the 6500 is still a perfect match for your network other than the prefix limit. Your vendor should think of your equipment as an "investment" to be protected, by exploiting your feelings of loss aversion, but the upgrade treadmill is a trap..... next thing you know, you will have to replace the chassis, then you will need new linecards......
Next up the road are the 6800's. Essentially SUP-2T's, so you get software parity Day One, but still the same supervisor module. We are running 6880's (which are the fixed SUP-2T's, but with modular line cards), but only a core switching (Layer 2 Ethernet) role. Great port density since the 10Gbps ports are now SFP+, but oversubscribed line cards 2:1, since each slot is 80Gbps, but the line card comes with 16x 10Gbps ports. You can disable oversubscription and go into performance mode, which disables half the ports on the line card - we do that. IP-/MPLS-wise, whatever you can do on the 6500 you can do on the 6800, but I can't say for sure as we're running them as switches. That said, if your goal is IP, just consider the ASR9000, MX, and whatever else other vendors can do in this space. Mark.
On (2014-07-30 08:06 -0500), Jimmy Hess wrote:
Keep in mind most of the MX series makes the 6500 look like a 5 port linksys home router, when it comes to carrying around and managing large BGP tables; both in terms of prefix capacity, speed, the policy/filtering/configuration management functionality of the OS, and how they will take the route update "beating" during setup of new multiple BGP sessions...
The SUP2T is about a 100% increase in TCAM size, but still pretty limited in terms of system resources.
You can also "protect" your investment if appropriate by taking this late 1990s gear off your BGP edge, or otherwise recruiting it for a role which it is more suited for in this day and age, where it is not handling full tables and thus the feeble amount of FIB size, CPU, memory are no potential hinderance now or on the next 10 years.
These seem cute anecdotes but I'm not sure how appropriate they are. CAT6880 is XEON control-plane, and if we compare MX80 and RSP720, where RSP720 has slightly lower performance CPU, RSP720 out-performs MX80 (and MX104) in BGP convergence and BGP scale. Certainly if you compare SUP720 to XEON MX960, your anecdote is accurate. JunOS is architecturally quite similar to IOS-XE, single fat process (iosd, rpd) doing all the relevant work, running on modern control-plane (linux, freebsd). One advantage to iosd is, that it's actually multithreaded unlike rpd. Obviously Sup2T/6880 2M FIB is limited, but what is JNPR MX scale? Trio has 256MB RLDRAM for everything, looking at my MX IPv4 FIB memory consumption divided by entry size, it pegs IPv4 entry to 77B (seems massive), which would translate to 3.5M IPv4 FIB upper bound, if nothing else is there. Realistically, I don't think JNPR promises anywhere near this. So the FIB scale may be pretty similar in both. So I don't think FIB, control-plane or software are selling-points here. Where MX shines, is deep services, with CAT you have relatively dumb ASIC, while MX is capable for very deep services with its NPU. If you can reuse existing LC and skill investment while living with limited forwarding-plane functionality offered, it seems entirely sensible solution, and in no way more '90s technology' than MX. If you need deep services, of course it's wrong box, then MX or ASR9k is what you should be looking at. -- ++ytti
On Tuesday, July 29, 2014 04:21:32 AM Corey Touchet wrote:
Right now my thinking are MX480 or ASR9k platforms. Opinions on those are equally welcome as alternatives, but I’d love to hear from those with personal experiences today vs sales people trying to tell me it would route the world :)
Yep, MX480/960 and ASR9006/9010 are the way to go if you're looking at decent (Intel-based) CPU's, good performance and good 10Gbps/100Gbps port density, incuding combinations thereof. 40Gbps might be a little tricky on these boxes; for that, looking at Ethernet switches (Nexus, C6880, Juniper EX) are better options. We don't mess around with 40Gbps - it's 10Gbps or 100Gbps :-). IOS XR on the CRS and ASR9000 is based on QNX, which suffers from being only a 32-bit kernel. So even if the hardware will ship with >4GB of RAM, the OS will only see 4GB (I have 12GB in my CRS's and 8GB on my ASR9001's). IOS XR on the NCS runs on Linux, which removes the memory limitation, but it's not clear whether that philosophy will make it down to earlier IOS XR platforms (CRS, ASR9000). Whatever the case, I've been following Blackberry for a while on this, and it doesn't seem like they have any plans to release a 64-bit version of QNX. AFAIK, their phones are all 32-bit, so... Junos has no issue seeing 32GB of RAM (their currently highest RAM on their RE's), as it's a properly 64-bit OS. That said, some of the applications that run within Junos (notably "rpd") are still playing catch-up in terms of how much memory it can "see", and how well it can use the multiple cores present on the RE's. A lot of work is going on in this area, and generally, the later the Junos code you run, the more enhancements to the software you will see (and the accompanying bugs, hehe). I've been testing Junos 14.1R1 in production on a couple of MX80's and MX480's for some weeks now. No issues to report (yet). Mark.
❦ 30 juillet 2014 09:53 +0200, Mark Tinka <mark.tinka@seacom.mu> :
IOS XR on the CRS and ASR9000 is based on QNX, which suffers from being only a 32-bit kernel. So even if the hardware will ship with >4GB of RAM, the OS will only see 4GB (I have 12GB in my CRS's and 8GB on my ASR9001's).
What's the point of shipping more memory then? Maybe the OS can only address 4GB per process but is able to use up to 64GB in total (PAE)? -- Use self-identifying input. Allow defaults. Echo both on output. - The Elements of Programming Style (Kernighan & Plauger)
On Wednesday, July 30, 2014 11:12:44 AM Vincent Bernat wrote:
What's the point of shipping more memory then? Maybe the OS can only address 4GB per process but is able to use up to 64GB in total (PAE)?
That was one argument from Cisco - that when the software catches up, they might be able to compartmentalize so that applications gain access to it individually. I didn't grill them too much on this, as we use IOS XR in the core mostly (CRS), and we don't need RAM too much since IPv4 is switched on MPLS labels, negating the need to hold a full IPv4 table on the routers. That said, I can see a use-case where the additional RAM on the CRS and ASR9000 can make sense if IOS XR is allowed to run separate VM's on the same control plane. I know that iso one of the ideas behind the NCS, but not sure whether it will be added to the CRS and ASR9000. Mark.
participants (7)
-
Corey Touchet -
Jimmy Hess -
John van Oppen -
Mark Tinka -
Saku Ytti -
Simon Lockhart -
Vincent Bernat