Last year we installed four 1RU TRILL switches in SIX - see http://www.six.sk/images/trill_ring.png Our experience after 100 days of production is only the best - TRILL setup is pretty straightforward and thanks to IS-IS it provides shortest-path IP-like "routing" for L2 ethernet packets over any reasonable topology out of the box (without the burden and cost implications of VPLS). Trident ASICs perform deep packet inspection so ECMP loadbalancing based on L3 and L4 headers inside TRILL-encapsulated packets works for both IPv4 and IPv6. Port-security is supported on physical ports as well as on LAGs - and L4 access-lists could be applied at the same time. As most 1RU switches are based on Trident ASICs, you just need to pick a vendor which implements TRILL properly and of course thoroughly test before deployment. We selected Huawei Cloud Engine 6850 boxes. Regards, M.
Dear Nanog community
We are trying to build a new IXP in some US Metro areas where we have multiple POPs and I was wondering what do you recommend for L2 switches. I know that some IXPs use Nexus, Brocade, Force10 but I don't personally have experience with these switches. It would be great if you can share your experience and recommendations. There are so many options that I don't know if it makes sense to start with a modular switch (usually expensive because the backplane, dual dc, dual CPU, etc) or start with a 1RU high density switch that support new protocols like Trill and that supposedly allow you to create Ethernet Fabric/Clusters. The requirements are simple, 1G/10G ports for exchange participants, 40G/100G for uplinks between switches and flow support for statistics and traffic analysis.
Thank you and have a great day.
Regards
On (2015-01-17 12:02 +0100), Marian Ďurkovič wrote:
Our experience after 100 days of production is only the best - TRILL setup is pretty straightforward and thanks to IS-IS it provides shortest-path IP-like "routing" for L2 ethernet packets over any reasonable topology out of the box (without the burden and cost implications of VPLS).
I'm not sure what the burden refers to, but cost implications to me seem same, trident HW can do VPLS.
From complexity POV, I don't expect much different development time to write functioning control-plane to either.
I'm not against Trill, I think Trill, and especially SPB-M are great, now they just feel too little and 20 years too late. There was no particular reason why SPB-M couldn't have existed 20 years ago in HW. But perhaps it's good it didn't, it might have made ethernet 'good enough', that selling MPLS might have been much more difficult. -- ++ytti
On Sat, Jan 17, 2015 at 09:15:04PM +0200, Saku Ytti wrote:
On (2015-01-17 12:02 +0100), Marian Ďurkovič wrote:
Our experience after 100 days of production is only the best - TRILL setup is pretty straightforward and thanks to IS-IS it provides shortest-path IP-like "routing" for L2 ethernet packets over any reasonable topology out of the box (without the burden and cost implications of VPLS).
I'm not sure what the burden refers to, but cost implications to me seem same, trident HW can do VPLS.
Well, it can, but as usual the devil is in the detail. For example, loadbalancing on outgoing LAGs depends on *inbound* packet encapsulation as follows: - native ethernet, TRILL, L3 MPLS : hash based on L3 and L4 headers - L2 MPLS, MACinMAC : hash based on L2 headers only. Thus if you use VPLS or SPB-M on Trident HW, the egress PE doesn't support per-flow loadbalancing on IXP participants' LAGs. In any case, we preferred TRILL over SPB-M not just because of that, but mainly due to a fact that TRILL provides real routing using IS-IS as we know it from IP world, while SPB still builds on top of MST and just cleverly uses multiple trees. Yes, compatibility with existing ASICs was one of the main design goals of SPB, but that's irrelevant once you have Trident HW. Regards, M.
On 19/01/2015 10:12, Marian Ďurkovič wrote:
Thus if you use VPLS or SPB-M on Trident HW, the egress PE doesn't support per-flow loadbalancing on IXP participants' LAGs.
not completely true. Extreme XOS has an interesting hack to work around this. Nick
On 1/17/15, 7:15 PM, "Saku Ytti" <saku@ytti.fi> wrote:
On (2015-01-17 12:02 +0100), Marian Ďurkovič wrote:
Our experience after 100 days of production is only the best - TRILL setup is pretty straightforward and thanks to IS-IS it provides shortest-path IP-like "routing" for L2 ethernet packets over any reasonable topology out of the box (without the burden and cost implications of VPLS).
I'm not sure what the burden refers to, but cost implications to me seem same, trident HW can do VPLS. From complexity POV, I don't expect much different development time to write functioning control-plane to either.
I'm not against Trill, I think Trill, and especially SPB-M are great, now they just feel too little and 20 years too late. There was no particular reason why SPB-M couldn't have existed 20 years ago in HW. But perhaps it's good it didn't, it might have made ethernet 'good enough', that selling MPLS might have been much more difficult.
-- ++ytti
I think in fairly short order both TRILL and 802.1AQ will be depercated in place of VXLAN and using BGP EVPN as the control plane ala Juniper QFX5100/Nexus 9300. Phil
On Mon, Jan 19, 2015 at 09:37:35PM -0500, Phil Bedard wrote:
I think in fairly short order both TRILL and 802.1AQ will be depercated in place of VXLAN and using BGP EVPN as the control plane ala Juniper QFX5100/Nexus 9300.
We also evaluated VXLAN for IXP deployment, since Trident-2 introduced HW support for it. But VXLAN does *not* create a network for you, it relies on some existing underlying IP network, on top of which VXLAN creates stateless tunnels. By using TRILL, we could connect 4 switches into a ring (or any other reasonable topology) and have a fully functional network with shortest-path "routing" of L2 packets. With VXLAN, we'd need at least two additional IP routers with bunch of 40GE interfaces to perform the functions TRILL supports out of the box. Regards, M.
For many people eliminating L2 switching and building on top of a L3 network is a good thing, especially if you are using BGP as the control plane. I'm not sure I follow the two routers with 40GE interfaces if you are just building L2 domains to interconnect people. Phil On 1/20/15, 8:04 AM, "Marian Ďurkovič" <md@bts.sk> wrote:
On Mon, Jan 19, 2015 at 09:37:35PM -0500, Phil Bedard wrote:
I think in fairly short order both TRILL and 802.1AQ will be depercated in place of VXLAN and using BGP EVPN as the control plane ala Juniper QFX5100/Nexus 9300.
We also evaluated VXLAN for IXP deployment, since Trident-2 introduced HW support for it. But VXLAN does *not* create a network for you, it relies on some existing underlying IP network, on top of which VXLAN creates stateless tunnels.
By using TRILL, we could connect 4 switches into a ring (or any other reasonable topology) and have a fully functional network with shortest-path "routing" of L2 packets.
With VXLAN, we'd need at least two additional IP routers with bunch of 40GE interfaces to perform the functions TRILL supports out of the box.
Regards,
M.
participants (4)
-
Marian Ďurkovič -
Nick Hilliard -
Phil Bedard -
Saku Ytti