All, I'm currently going through a network design for an upgrade for one of the networks I run. Much of the wide-area traffic on the network is used purely to transport Ethernet tail circuits back from an edge PoP to a core PoP. Currently we're using Extreme X460 and X670 switches to achieve this, carrying the tail circuits within VPLS. Two things are making me look at a change of solution for this - firstly Extreme have stated that they're not interested in the service provider market any more (and reflected this in significant reductions in discounts), and secondly we need to look at higher bandwidth port options (40G + 100G, particularly for backhaul circuits). As we're primarily a Cisco house, I've been looking at suitable replacements, and the Nexus 9k range looks good - 92160YC and 9236C in particular. However, this would mean a shift from VPLS to VXLAN. We're also looking at Cisco-like products, such as the Arista range. We've been doing some testing in the lab, and so far, things look good - it's easy to configure, and appears to do the job of getting packets from A to B. We do have two concerns, though: 1) Cisco are strongly advising against using the Nexus switches in a WAN scenario - as they're designed for "datacentre" use. They've so far said they can't find anyone who can help validate designs using Nexus, and instead are pushing us towards the NCS-5000 series switches. Same chipset, but 2-3 times the price! NCS does, however, support VPLS, so would be an easier drop-in to our existing network. 2) Traffic engineering - we don't have a lot of requirement for this, but do have a small number of customers who buy A and B circuits, and require them to be routed across different paths on our network. This is easy with MPLS using explicit LSPs, but we've not yet worked out how to achieve the same thing in VXLAN. So, my question to the community is - have any of you used VXLAN as a wide-area layer 2 transport technology? Any pros or cons? Gotchas? Scare stories? Recommendations? Am I trying to shoot myself in the foot? Many thanks, Simon
However, this would mean a shift from VPLS to VXLAN. On this particular sentence - it is incorrect to compare VPLS and VXLAN. One is a set of control plane mechanisms for discovery and tunnel setup and data plane encapsulations and multipoint reachability model, while
2) Traffic engineering - we don't have a lot of requirement for this, but do have a small number of customers who buy A and B circuits, and require them to be routed across different paths on our network. This is easy with MPLS using explicit LSPs, but we've not yet worked out how to achieve the same thing in VXLAN. VXLAN is a tunnel over a routed IP network. The path to reach the tunnel endpoint will define how VXLAN encapsulated payload is traversing over your network. If tunnel endpoint is reachable via explicitly routed LSP,
Hi there, the other is just a data plane encapsulation. VXLAN is equivalent to a (point to point) pseudowire used as a data plane component in VPLS. the payload will follow that path. VXLAN by itself does nothing to influence the underlay path.
So, my question to the community is - have any of you used VXLAN as a wide-area layer 2 transport technology? Yes, there are such deployments. The number of such deployments is increasing.
Any pros or cons? Gotchas? Scare stories? Recommendations? Am I trying to shoot myself in the foot?
VXLAN is a tunnelling mechanism. You seem to be looking for a end to end solution, for which VXLAN is a (rather small) component. You need to start from the requirements. What is the underlay technology being used? How much of traffic engineering functionality will be required, and how the particular underlay technology can implement it? What about ECMP support and requirements for its use - flow mapping to VXLAN tunnel is the same problem as flow mapping to any other tunnel in underlay ECMP environment. To build all of this is one thing, but to operate is quite the other - what about OAM requirements? VXLAN has no OAM support, and will not get one. If your operations systems rely on MPLS based OAM for service layer (eg, an MPLS based PW) - that will not be available. How the tunnels will be set up - is there a need to do discovery, or will all of it be orchestrated? Two main limiting factors in VXLAN applicability are lack of payload type indicator - the tunnel can carry only a single type of payload (ethernet in the canonic definition, no possibility to use single tunnel for multiple user payload types without additional encapsulation), and no ability to indicate that the payload is not a user payload but some other special type like OAM (pseudowire control word would be an example of such indicator). This leaves VXLAN as a solution of limited applicability for anything else than original intended one - ethernet tunnel over IP underlay. For other uses VXLAN is of limited applicability, there may be (and in fact are) vendor specific extensions but there is no point in talking about interoperability then. This problem space is well understood and there are successors to VXLAN - Geneve is getting into good shape (there are vendor specific early implementations but it is too early to talk about interoperability at this time), plus different vendors have their own tunnel encapsulation mechanisms (VXLAN-GPE being a more visible one, however it is not going to be standardized soon). The other interesting part to solve is the control plane. BGP based one will likely be the most practical option here. Combined with ability to tunnel multiple payload types this gives you a single control plane for L3VPN and point to point and multipoint L2VPN built on the same underlay with the single tunnelling mechanism. Your operations team will thank you violently for doing this. Ignas
Been there, got the t-shirt ... VXLAN was not designed to be a direct replacement for L2VPN. It was built to scale L2 broadcast domains. With Cisco, L2 control protocols like STP are not supported. Can't speak for other vendors. So what someone would typically expect from EoMPLS and L2 protocol tunneling, they're not going to get with a VXLAN substitute. If all you're looking to do is create a virtual Ethernet cable between two L3 interfaces with absolutely no L2 protocol tunneling involved, it works like a champ. We use them for that purpose, building virtual cross-connects between routers, as well as in a few virtual environments to overlay L2 networks across a BGP CLOS. I would not recommend VXLAN to replace L2VPNs unless you plan to wait for / hope for L2 protocol tunneling support. You would not get a direct replacement for your current VPLS circuits. Segment routing is in the same boat - no L2VPN support until next year, and then it's supposed to be limited to -EX and -FX Nexus chassis. I can't speak on the subject of using it across a WAN; we dump our VXLAN tunnels to ASR9Ks at the edge where traditional MPLS takes over. Generally, it's not recommended to try and extend a LAN-based protocol across a WAN. While it's L3 traffic once the VXLAN overlay takes over, I would look into the control plane requirements, overhead, etc. before even considering it. NCS is what Cisco currently recommends for a scalable MPLS fabric. There's also the ASR9000V, which acts as a satellite / remote line card of larger ASR9K routers, but you'd still want some kind of aggregation in there to scale or your ASR9K port cost is going to start to hurt. Hit me up off-list if you want to know a little more in detail. -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Simon Lockhart Sent: Thursday, July 20, 2017 2:12 AM To: nanog@nanog.org Subject: VXLAN for WAN Pseudowires? All, I'm currently going through a network design for an upgrade for one of the networks I run. Much of the wide-area traffic on the network is used purely to transport Ethernet tail circuits back from an edge PoP to a core PoP. Currently we're using Extreme X460 and X670 switches to achieve this, carrying the tail circuits within VPLS. Two things are making me look at a change of solution for this - firstly Extreme have stated that they're not interested in the service provider market any more (and reflected this in significant reductions in discounts), and secondly we need to look at higher bandwidth port options (40G + 100G, particularly for backhaul circuits). As we're primarily a Cisco house, I've been looking at suitable replacements, and the Nexus 9k range looks good - 92160YC and 9236C in particular. However, this would mean a shift from VPLS to VXLAN. We're also looking at Cisco-like products, such as the Arista range. We've been doing some testing in the lab, and so far, things look good - it's easy to configure, and appears to do the job of getting packets from A to B. We do have two concerns, though: 1) Cisco are strongly advising against using the Nexus switches in a WAN scenario - as they're designed for "datacentre" use. They've so far said they can't find anyone who can help validate designs using Nexus, and instead are pushing us towards the NCS-5000 series switches. Same chipset, but 2-3 times the price! NCS does, however, support VPLS, so would be an easier drop-in to our existing network. 2) Traffic engineering - we don't have a lot of requirement for this, but do have a small number of customers who buy A and B circuits, and require them to be routed across different paths on our network. This is easy with MPLS using explicit LSPs, but we've not yet worked out how to achieve the same thing in VXLAN. So, my question to the community is - have any of you used VXLAN as a wide-area layer 2 transport technology? Any pros or cons? Gotchas? Scare stories? Recommendations? Am I trying to shoot myself in the foot? Many thanks, Simon
Hi Simon, In the previous job, we used it in a similar scenario and from that experience × × What works fine across end points: Routing protocols (OSPF, BGP), VLAN, QinQ, Multicast What doesn't' work across end points: LLDP, LACP, CoS preservation (you can remark), 802.1x So, test your requirements in the lab (as you are already doing it), its not a VPLS replacement in many ways but it worked like a charm in our requirement. We used Open-network boxes (Dell, HP, etc) along with CumulusLinux (Dell OS9 also has VXLAN support). Arista (Trident II/II+) also works fine with EOS. All these switches were and still are for DC but they are extremely cheaper than NCS5000 series and works fine. On Thu, 20 Jul 2017 at 19:14 Simon Lockhart <simon@slimey.org> wrote:
All,
I'm currently going through a network design for an upgrade for one of the networks I run. Much of the wide-area traffic on the network is used purely to transport Ethernet tail circuits back from an edge PoP to a core PoP. Currently we're using Extreme X460 and X670 switches to achieve this, carrying the tail circuits within VPLS.
Two things are making me look at a change of solution for this - firstly Extreme have stated that they're not interested in the service provider market any more (and reflected this in significant reductions in discounts), and secondly we need to look at higher bandwidth port options (40G + 100G, particularly for backhaul circuits).
As we're primarily a Cisco house, I've been looking at suitable replacements, and the Nexus 9k range looks good - 92160YC and 9236C in particular. However, this would mean a shift from VPLS to VXLAN. We're also looking at Cisco-like products, such as the Arista range.
We've been doing some testing in the lab, and so far, things look good - it's easy to configure, and appears to do the job of getting packets from A to B.
We do have two concerns, though:
1) Cisco are strongly advising against using the Nexus switches in a WAN scenario - as they're designed for "datacentre" use. They've so far said they can't find anyone who can help validate designs using Nexus, and instead are pushing us towards the NCS-5000 series switches. Same chipset, but 2-3 times the price! NCS does, however, support VPLS, so would be an easier drop-in to our existing network.
2) Traffic engineering - we don't have a lot of requirement for this, but do have a small number of customers who buy A and B circuits, and require them to be routed across different paths on our network. This is easy with MPLS using explicit LSPs, but we've not yet worked out how to achieve the same thing in VXLAN.
So, my question to the community is - have any of you used VXLAN as a wide-area layer 2 transport technology? Any pros or cons? Gotchas? Scare stories? Recommendations? Am I trying to shoot myself in the foot?
Many thanks,
Simon
-- Best Wishes, Aftab A. Siddiqui
From: "Simon Lockhart" <simon@slimey.org>
Hi,
2) Traffic engineering - we don't have a lot of requirement for this, but do have a small number of customers who buy A and B circuits, and require them to be routed across different paths on our network. This is easy with MPLS using explicit LSPs, but we've not yet worked out how to achieve the same thing in VXLAN.
You may be able to achieve this by using a second loopback for the B circuit VTEP IP address, and use either PBR or *cough* a static *cough* route *cough* towards a different path. Not pretty, but will probably work. Thanks, Sabri
On Fri, Jul 21, 2017 at 12:38 PM, Sabri Berisha <sabri@cluecentral.net> wrote:
From: "Simon Lockhart" <simon@slimey.org>
Hi,
2) Traffic engineering - we don't have a lot of requirement for this, but do have a small number of customers who buy A and B circuits, and require them to be routed across different paths on our network. This is easy with MPLS using explicit LSPs, but we've not yet worked out how to achieve the same thing in VXLAN.
You may be able to achieve this by using a second loopback for the B circuit VTEP IP address, and use either PBR or *cough* a static *cough* route *cough* towards a different path. Not pretty, but will probably work.
Thanks,
Sabri
Or just use RSVP-TE... It's just IP.. T2 can do a MPLS lookup after encapsulating in VXLAN.. -- Tim
participants (6)
-
Aftab Siddiqui
-
Ignas Bagdonas
-
Sabri Berisha
-
Sean Pedersen
-
Simon Lockhart
-
Tim Jackson