Nanog, I would like to poll the collective for experiences both positive and negative with the Nexus line. More specifically I am interested in hearing about FEX with N2K at the ToR and if this has indeed made any impact on Opex as well as non-obvious shortcomings to using the fabric extenders. Also if anyone is using any of the Nexus line for I/O convergence (FCoE) I would be interested in hearing your experience with this as well. Thank you in advance, -A
The biggest thing we ran into was no support of spanning-tree on the FEX's. The way we are setup, being STATE government, our agency controls the network up to the FEX port. Beyond that, the agency's were in control of what they plugged into our FEX ports. -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Herman, Anthony Sent: Monday, February 02, 2015 11:17 AM To: nanog@nanog.org Subject: Cisco Nexus Nanog, I would like to poll the collective for experiences both positive and negative with the Nexus line. More specifically I am interested in hearing about FEX with N2K at the ToR and if this has indeed made any impact on Opex as well as non-obvious shortcomings to using the fabric extenders. Also if anyone is using any of the Nexus line for I/O convergence (FCoE) I would be interested in hearing your experience with this as well. Thank you in advance, -A
The n2k ToR is not a great design for user or storage interfaces if most of your traffic is east/west. It is great as a low cost ilo/drac/choose your oob port, or if most of your traffic is north/south. Biggest thing to remember is that it is not a switch, and has limitations such as not connecting other switches to it. Like anything else you have to understand the product so that you don't engineer something that it wasn't designed to do. Lots of very large companies using Nexus gear... That being said I prefer Arista when I'm architecting DCs.
On Feb 2, 2015, at 12:17 PM, Herman, Anthony <Anthony.Herman@mattersight.com> wrote:
Nanog,
I would like to poll the collective for experiences both positive and negative with the Nexus line. More specifically I am interested in hearing about FEX with N2K at the ToR and if this has indeed made any impact on Opex as well as non-obvious shortcomings to using the fabric extenders. Also if anyone is using any of the Nexus line for I/O convergence (FCoE) I would be interested in hearing your experience with this as well.
Thank you in advance,
-A
On Mon, Feb 02, 2015 at 12:51:04PM -0600, David Bass wrote:
The n2k ToR is not a great design for user or storage interfaces if most of your traffic is east/west. It is great as a low cost ilo/drac/choose your oob port, or if most of your traffic is north/south. Biggest thing to remember is that it is not a switch, and has limitations such as not connecting other switches to it. Like anything else you have to understand the product so that you don't engineer something that it wasn't designed to do.
And remember -- The Nexus 2K performs absolutely ZERO local switching -- all frames received from client ports are just copied to the upstream device, so it can handle the frame/packet forwarding logic. -- Brandon Ewing (nicotine@warningg.com)
On Mon, 2 Feb 2015, Brandon Ewing wrote:
On Mon, Feb 02, 2015 at 12:51:04PM -0600, David Bass wrote:
The n2k ToR is not a great design for user or storage interfaces if most of your traffic is east/west. It is great as a low cost ilo/drac/choose your oob port, or if most of your traffic is north/south. Biggest thing to remember is that it is not a switch, and has limitations such as not connecting other switches to it. Like anything else you have to understand the product so that you don't engineer something that it wasn't designed to do.
And remember -- The Nexus 2K performs absolutely ZERO local switching -- all frames received from client ports are just copied to the upstream device, so it can handle the frame/packet forwarding logic.
Also remember that the Nexus (5K, at least) does cut-through switching. If you receive an errored frame on one port, the switch can and often will happily forward those errored frames once it figures out the destination MAC address(es). jms
Brandon Ewing <nicotine@warningg.com> wrote:
David Bass wrote: The n2k ToR is not a great design for user or storage interfaces if most of your traffic is east/west. It is great as a low cost ilo/drac/choose your oob port, or if most of your traffic is north/south. Biggest thing to remember is that it is not a switch, and has limitations such as not connecting other switches to it. Like anything else you have to understand the product so that you don't engineer something that it wasn't designed to do.
And remember -- The Nexus 2K performs absolutely ZERO local switching -- all frames received from client ports are just copied to the upstream device, so it can handle the frame/packet forwarding logic.
What this really does is force you to consider how much of your East-West is rack-local, versus off rack. Rack-local-heavy hurts as badly as off rack, with FEX. If you want to / can localize E/W tighter than that then you want real TOR switching. If the average E-W is cross rack then the FEX are performance equivalent. For random distributions this comes at a few racks. For intentional distributions it's probably better to TOR switch from day one. George William Herbert Sent from my iPhone
I think it depends on what the upstream product from the FEX is and what your requirements are. Last I checked, eVPC was not supported on the N7K, but it was supported as an option on the N5K platform. eVPC being the dual homed FEX to two a pair of N7K's running a VPC cluster. I know this is an old post, but here's a good one that explains precisely what I mean: If you look at the N5K verified scalability guide, you see this: Maximum FEXs dual homed to a vPC Cisco Nexus 5500 Series Switch Pair: 24 http://www.cisco.com/en/US/docs/switches/datacenter/nexus5500/sw/configurati... If you look at the N7K verified scalability guide, there is *no* mention of dual-homed fex architectures: http://www.cisco.com/en/US/docs/switches/datacenter/sw/verified_scalability/... If I'm wrong and someone is dual homing 2K FEXs to 7K VPC pairs, please correct me. If you're interested in latency numbers fex-to-fex, here are some numbers provided to me by our SE: http://jumboframe.net/jumboframe/2013/5/5/n2k-fex-to-fex-latency-and-a-reade... <http://static1.squarespace.com/static/513c6d36e4b0cc0702f94292/t/51866c97e4b0580e000cf5bd/1367764121313/fex-to-fex-latency.jpg?format=750w> As you can see, these numbers are decent, but you'd have to be very careful in choosing your FEX model if you're looking for better than average store and forward performance out of your FEX. On Mon, Feb 2, 2015 at 10:17 AM, Herman, Anthony < Anthony.Herman@mattersight.com> wrote:
Nanog,
I would like to poll the collective for experiences both positive and negative with the Nexus line. More specifically I am interested in hearing about FEX with N2K at the ToR and if this has indeed made any impact on Opex as well as non-obvious shortcomings to using the fabric extenders. Also if anyone is using any of the Nexus line for I/O convergence (FCoE) I would be interested in hearing your experience with this as well.
Thank you in advance,
-A
I wasn't the implementing engineer but I've been at two places that did that, a larger game company and a network gear manufacturer in their engineering support computational hubs. I was there during planning and rollout at the game company, very early in the Nexus lifespan. Both sites brought the FEXes back to 5500s; one used a 6-something for core, the other a pair of 7ks. Game company was more east-west, telco eqpt was very heavy east west. In both cases it's working fine. George William Herbert Sent from my iPhone
On Feb 2, 2015, at 10:17 AM, "Herman, Anthony" <Anthony.Herman@mattersight.com> wrote:
Nanog,
I would like to poll the collective for experiences both positive and negative with the Nexus line. More specifically I am interested in hearing about FEX with N2K at the ToR and if this has indeed made any impact on Opex as well as non-obvious shortcomings to using the fabric extenders. Also if anyone is using any of the Nexus line for I/O convergence (FCoE) I would be interested in hearing your experience with this as well.
Thank you in advance,
-A
There are some unfortunate limitations in classifying incoming traffic. It's been a while, but I think the rule is that Nexus 2000 devices can only classify based on incoming 802.1p cos values. It's a pretty strange and disappointing limitation for an edge device where you're less likely to have incoming dot1q tags and you're less likely to trust the other end of the link to mark its own traffic. /chris
I have a small setup, Nexus 2 x 5596UP + 12 x 2248TP FEX, 2 x B22DELL, 2 x B22HP, 1 x C2248PQ-10GE. Been using this setup since 2012, so it's getting a bit long in the tooth. It's in an Active-Active setup because there wasn't much guidance at the time on which way to go. There are some restrictions with an AA setup you probably want to avoid. We currently don't do any FCoE because we're mostly a NetApp and NFS environment. The performance and stability have been great. It works well for a traditional environment with a lot of wired ports to stand-alone servers. If you do a lot with virtualization it's not a great solution. You really want to avoid connecting VM host servers to FEX ports because of all the restrictions that come with it. One restriction that's a real PITA for me right now is that a FEX port can't be a promiscuous trunk port if you're using PVLAN. Using config-sync has been a lot of trouble. There are a lot of actions that will verify OK but then fail. The result is that things are partially configured and the whole system gets out of sync not letting you make any other changes; the fix is having to manually go in to each switch to try and get the configuration to match (which requires comparing the running-configuration to the running switch-profile configuration). On Mon, Feb 2, 2015 at 1:17 PM, Herman, Anthony <Anthony.Herman@mattersight.com> wrote:
Nanog,
I would like to poll the collective for experiences both positive and negative with the Nexus line. More specifically I am interested in hearing about FEX with N2K at the ToR and if this has indeed made any impact on Opex as well as non-obvious shortcomings to using the fabric extenders. Also if anyone is using any of the Nexus line for I/O convergence (FCoE) I would be interested in hearing your experience with this as well.
Thank you in advance,
-A
-- Ray Patrick Soucy Network Engineer University of Maine System T: 207-561-3526 F: 207-561-3531 MaineREN, Maine's Research and Education Network www.maineren.net
participants (9)
-
Brandon Ewing -
Chris Marget -
Daniel Rohan -
David Bass -
George Herbert -
Herman, Anthony -
Justin M. Streiner -
Mann, Jason -
Ray Soucy