Looking for recommendation on 10G Ethernet switch
Colleagues, I'm looking for a recommendation on a smallish 10G Ethernet switch for a small virtualization/SAN implementation (4-5 hosts, 2 SAN boxes) over iSCSI with some legacy boxes on GigE. Preferably - 8-16 10G ports - several GigE ports for legacy GigE hosts or cross connect to a legacy GigE switch - preferably not a large chassis based solution with blades The hosts aren't going to be driving full line rate, nor the SAN boxes providing full line rate, but their offered loads will definitely exceed 1Gbps. Assessing whether it is better to go 10G now vs. multi-pathing with quad GigE cards. Trying to find the best solution for > 1G on a trunk and < $50K per box. Any recommendations appreciated. Thanks EKG
On Fri, Nov 2, 2012 at 11:13 AM, Eric Germann <egermann@limanews.com> wrote:
Colleagues,
I'm looking for a recommendation on a smallish 10G Ethernet switch for a small virtualization/SAN implementation (4-5 hosts, 2 SAN boxes) over iSCSI with some legacy boxes on GigE.
Preferably
- 8-16 10G ports - several GigE ports for legacy GigE hosts or cross connect to a legacy GigE switch - preferably not a large chassis based solution with blades
The hosts aren't going to be driving full line rate, nor the SAN boxes providing full line rate, but their offered loads will definitely exceed 1Gbps. Assessing whether it is better to go 10G now vs. multi-pathing with quad GigE cards. Trying to find the best solution for > 1G on a trunk and < $50K per box.
Any recommendations appreciated.
Thanks
EKG
This topic has been covered completely and often on the Beowulf list. http://www.beowulf.org/pipermail/beowulf/ or do a site search via your friendly search engine. -- ~ Andrew "lathama" Latham lathama@gmail.com http://lathama.net ~
If you're looking at sub 50k, the Nexus 5k isn't a terrible option. It gives you 32 10Gig SFP slots for ~$25,000 or less if you don't mind used from ebay. On Fri, Nov 2, 2012 at 8:18 AM, Andrew Latham <lathama@gmail.com> wrote:
On Fri, Nov 2, 2012 at 11:13 AM, Eric Germann <egermann@limanews.com> wrote:
Colleagues,
I'm looking for a recommendation on a smallish 10G Ethernet switch for a small virtualization/SAN implementation (4-5 hosts, 2 SAN boxes) over iSCSI with some legacy boxes on GigE.
Preferably
- 8-16 10G ports - several GigE ports for legacy GigE hosts or cross connect to a legacy GigE switch - preferably not a large chassis based solution with blades
The hosts aren't going to be driving full line rate, nor the SAN boxes providing full line rate, but their offered loads will definitely exceed 1Gbps. Assessing whether it is better to go 10G now vs. multi-pathing with quad GigE cards. Trying to find the best solution for > 1G on a trunk and < $50K per box.
Any recommendations appreciated.
Thanks
EKG
This topic has been covered completely and often on the Beowulf list. http://www.beowulf.org/pipermail/beowulf/ or do a site search via your friendly search engine.
-- ~ Andrew "lathama" Latham lathama@gmail.com http://lathama.net ~
-- 09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0
On Fri, Nov 2, 2012 at 1:58 PM, Mike Hale <eyeronic.design@gmail.com> wrote:
If you're looking at sub 50k, the Nexus 5k isn't a terrible option. It gives you 32 10Gig SFP slots for ~$25,000 or less if you don't mind used from ebay.
On Fri, Nov 2, 2012 at 8:18 AM, Andrew Latham <lathama@gmail.com> wrote:
On Fri, Nov 2, 2012 at 11:13 AM, Eric Germann <egermann@limanews.com> wrote:
Colleagues,
I'm looking for a recommendation on a smallish 10G Ethernet switch for a small virtualization/SAN implementation (4-5 hosts, 2 SAN boxes) over iSCSI with some legacy boxes on GigE.
Preferably
- 8-16 10G ports - several GigE ports for legacy GigE hosts or cross connect to a legacy GigE switch - preferably not a large chassis based solution with blades
The hosts aren't going to be driving full line rate, nor the SAN boxes providing full line rate, but their offered loads will definitely exceed 1Gbps. Assessing whether it is better to go 10G now vs. multi-pathing with quad GigE cards. Trying to find the best solution for > 1G on a trunk and < $50K per box.
Any recommendations appreciated.
Thanks
EKG
This topic has been covered completely and often on the Beowulf list. http://www.beowulf.org/pipermail/beowulf/ or do a site search via your friendly search engine.
-- ~ Andrew "lathama" Latham lathama@gmail.com http://lathama.net ~
Also note that new devices hit the market everyday. For example Supermicro has an offering at http://www.supermicro.com/products/accessories/Networking/SSE-X24S.cfm which at the very least will be supported forever. -- ~ Andrew "lathama" Latham lathama@gmail.com http://lathama.net ~
You may want to take a look at the Brocade VDX 6720, it provides 16 10gb ports, with 8 ports on demand with addl license. They are very reasonable, esp. if you only need 16 ports. Maintenance costs are less than cisco. ----- Original Message -----
From: "Eric Germann" <egermann@limanews.com> To: nanog@nanog.org Sent: Friday, November 2, 2012 10:13:01 AM Subject: Looking for recommendation on 10G Ethernet switch Colleagues,
I'm looking for a recommendation on a smallish 10G Ethernet switch for a small virtualization/SAN implementation (4-5 hosts, 2 SAN boxes) over iSCSI with some legacy boxes on GigE.
Preferably
- 8-16 10G ports - several GigE ports for legacy GigE hosts or cross connect to a legacy GigE switch - preferably not a large chassis based solution with blades
The hosts aren't going to be driving full line rate, nor the SAN boxes providing full line rate, but their offered loads will definitely exceed 1Gbps. Assessing whether it is better to go 10G now vs. multi-pathing with quad GigE cards. Trying to find the best solution for > 1G on a trunk and < $50K per box.
Any recommendations appreciated.
Thanks
EKG
On Fri, Nov 2, 2012 at 11:13 AM, Eric Germann <egermann@limanews.com> wrote:
I'm looking for a recommendation on a smallish 10G Ethernet switch for a small virtualization/SAN implementation (4-5 hosts, 2 SAN boxes) over iSCSI with some legacy boxes on GigE.
1Gbps. Assessing whether it is better to go 10G now vs. multi-pathing with quad GigE cards. Trying to find the best solution for > 1G on a trunk and < $50K per box.
Certainly the days of doing NxGE to servers should be behind us. There are many good 10GE switch offerings. The Juniper QFX and Arista (insert one of three good product lines here) have been excellent for us. We like them for different reasons -- Arista is quite good if you want to integrate with a provisioning system; QFX is our choice when most provisioning is done manually. Both are way under $50k per box. The biggest difference between the TOR-style switches and chassis offerings, aside from the obvious, is buffers. All the TOR-type 10G switches have really small buffers and that can be a performance issue for iSCSI when utilization is high. Most of the chassis-type switches have very generous buffers so you do not run into problems with micro-bursting, etc. The vendors will all tell you about lossless ethernet, flow control, etc. and that crap sounds great on paper. Try making it actually work. You'll want those days of your life back. :) $0.02. -- Jeff S Wheeler <jsw@inconcepts.biz> Sr Network Operator / Innovative Network Concepts
On 02/11/2012 20:10, Jeff Wheeler wrote:
The biggest difference between the TOR-style switches and chassis offerings, aside from the obvious, is buffers. All the TOR-type 10G switches have really small buffers and that can be a performance issue for iSCSI when utilization is high
not particularly when utilisation is high, but in situations where congestion occurs, e.g. when you either have a high write load from multiple clients to a single server, or if you're down-shifting from a 10G server to 1G clients or something.
The vendors will all tell you about lossless ethernet, flow control, etc. and that crap sounds great on paper. Try making it actually work.
flow control on a switch port can lead to hol blocking, which is bad bad news - guaranteed to trash multi-access network performance. Some vendors actually push this as a feature. I don't completely understand why, but maybe it has something to do with customers mistakenly believing that it will make their lives better. People believe in all sorts of odd superstitions though: black cats, spilling salt, having full features on ethernet LAGs, vendor marketing blurb, fear of the number 13, etc. All very odd stuff. Nick
On Fri, Nov 2, 2012 at 4:10 PM, Jeff Wheeler <jsw@inconcepts.biz> wrote:
On Fri, Nov 2, 2012 at 11:13 AM, Eric Germann <egermann@limanews.com> wrote:
I'm looking for a recommendation on a smallish 10G Ethernet switch for a small virtualization/SAN implementation (4-5 hosts, 2 SAN boxes) over iSCSI with some legacy boxes on GigE.
1Gbps. Assessing whether it is better to go 10G now vs. multi-pathing with quad GigE cards. Trying to find the best solution for > 1G on a trunk and < $50K per box.
Certainly the days of doing NxGE to servers should be behind us. There are many good 10GE switch offerings. The Juniper QFX and Arista (insert one of three good product lines here) have been excellent for us. We like them for different reasons -- Arista is quite good if you want to integrate with a provisioning system; QFX is our choice when most provisioning is done manually. Both are way under $50k per box.
The biggest difference between the TOR-style switches and chassis offerings, aside from the obvious, is buffers. All the TOR-type 10G switches have really small buffers and that can be a performance issue for iSCSI when utilization is high. Most of the chassis-type switches have very generous buffers so you do not run into problems with micro-bursting, etc. The vendors will all tell you about lossless ethernet, flow control, etc. and that crap sounds great on paper. Try making it actually work. You'll want those days of your life back. :)
I'm curious if anyone here has experience with the Extreme Summit X650 or X670 switches? I'm also currently looking for 10GBASE-T switch for my first SAN, but one with 48 ports. The price for X670V-48T seems reasonable, but I have no experience with 10 GbE over copper, so it's difficult to figure out what criteria to use (other than price) when comparing Extreme, Arista, Dell, and others. Any tips? - Max
On Fri, Nov 2, 2012 at 5:13 PM, Eric Germann <egermann@limanews.com> wrote:
Colleagues,
I'm looking for a recommendation on a smallish 10G Ethernet switch for a small virtualization/SAN implementation (4-5 hosts, 2 SAN boxes) over iSCSI with some legacy boxes on GigE.
Preferably
- 8-16 10G ports - several GigE ports for legacy GigE hosts or cross connect to a legacy GigE switch - preferably not a large chassis based solution with blades
The hosts aren't going to be driving full line rate, nor the SAN boxes providing full line rate, but their offered loads will definitely exceed 1Gbps. Assessing whether it is better to go 10G now vs. multi-pathing with quad GigE cards. Trying to find the best solution for > 1G on a trunk and < $50K per box.
You can look ar Brocade TurboIron 24. It has 24 ports of 1/10G depending on the SFP you put in.
The Juniper ex4500/4550 could work, small chassis, can be made part of a virtual chassis. Works well in an enterprise setup but can cause configuration headaches if within a service provider environment where vlans need to be translated. Darius On Fri, Nov 2, 2012 at 4:13 PM, Eric Germann <egermann@limanews.com> wrote:
Colleagues,
I'm looking for a recommendation on a smallish 10G Ethernet switch for a small virtualization/SAN implementation (4-5 hosts, 2 SAN boxes) over iSCSI with some legacy boxes on GigE.
Preferably
- 8-16 10G ports - several GigE ports for legacy GigE hosts or cross connect to a legacy GigE switch - preferably not a large chassis based solution with blades
The hosts aren't going to be driving full line rate, nor the SAN boxes providing full line rate, but their offered loads will definitely exceed 1Gbps. Assessing whether it is better to go 10G now vs. multi-pathing with quad GigE cards. Trying to find the best solution for > 1G on a trunk and < $50K per box.
Any recommendations appreciated.
Thanks
EKG
Dell Force10 S4810 is a decent ToR switch: 48 dual-speed 1/10GbE (SFP+) ports and four 40GbE (QSFP+) uplinks Peter
On Fri, Nov 2, 2012 at 4:13 PM, Eric Germann <egermann@limanews.com> wrote:
Colleagues,
I'm looking for a recommendation on a smallish 10G Ethernet switch for a small virtualization/SAN implementation (4-5 hosts, 2 SAN boxes) over iSCSI with some legacy boxes on GigE.
Preferably
- 8-16 10G ports - several GigE ports for legacy GigE hosts or cross connect to a legacy GigE switch - preferably not a large chassis based solution with blades
The hosts aren't going to be driving full line rate, nor the SAN boxes providing full line rate, but their offered loads will definitely exceed 1Gbps. Assessing whether it is better to go 10G now vs. multi-pathing with quad GigE cards. Trying to find the best solution for > 1G on a trunk and < $50K per box.
Any recommendations appreciated.
Thanks
EKG
participants (10)
-
Andrew Latham -
Dan Olson -
Darius Seroka -
Eric Germann -
Eugeniu Patrascu -
Jeff Wheeler -
Maxim Khitrov -
Mike Hale -
Nick Hilliard -
Peter Nowak