Re: MPLS in the campus Network?
Compared to MPLS, a L2 solution with 100 Gb/s interfaces between core switches and a 10G connection for each buildings looks so much cheaper. But we worry about future trouble using Trill, SPB, or other technologies, not only the "open" ones, but specifically the proprietary ones based on central controller and lots of magic (some colleagues feel the debug nightmare are garanteed).
If you had to make such a choice recently, did you choose an MPLS design even at lower speed ?
A year ago we built NREN backbone using TRILL instead of MPLS. 40 POPs, no central controller, RFC standardized TRILL protocol i.e. L2 routing using IS-IS, no STP. See my recent presentation at http://md.bts.sk/sanet-100g-2.pdf for more details. Much easier to setup, operate & maintain than MPLS and obviously much lower cost. Based on 6-months production experience, my recommendation would be to stay away from MPLS in the campus. With kind regards, M.
On 21/Oct/16 16:19, Marian Ďurkovič wrote:
Much easier to setup, operate & maintain than MPLS and obviously much lower cost. Based on 6-months production experience, my recommendation would be to stay away from MPLS in the campus.
I'd be curious to hear what MPLS-specific issues you faced in the 6 months you had to operate such a network. Been running IP/MPLS Core, Edge and Access networks for over 15 years, and apart from bugs which affect any protocol or feature implementation, I can't say it has been a nightmare to operate to the point of not recommending it. I have far fewer words to say about STP, although - I'll admit - I've never run TRILL. Mark.
Our campus started off with L2 vlans spanning through the core, but we migrated to routing in the core and moved our many spanning tree/broadcast domains to the edge of buildings fronted by redundant routing with ecmp to a redundant core utilizing ospf. In a campus network the challenge becomes extending subnets across your core. You may have a college that started in one building with their own /24, but now have offices and labs in other buildings. They want to stay on the same network, but that's not feasible with the routed core setup without some other technology overlay. We end up not being able to extend the L2 like we did in the past and today we modify router ACL's to allow communications. If you already have hundreds of vlans spanned across the network, it's hard to get a campus to migrate to the routed core. I think this may be one of Marks challenge, correct me if I'm wrong please. With that said, what are the best options to be able to cost effectively scale without using vlans and maintaining a routed core? What technology would someone suggest (mpls, vxlan,etc) to be the best possible solution? Thank you to the participants in the discussion. I always enjoy reading comments posted. -Javier On Oct 21, 2016 11:46 AM, "Mark Tinka" <mark.tinka@seacom.mu> wrote:
On 21/Oct/16 16:19, Marian Ďurkovič wrote:
Much easier to setup, operate & maintain than MPLS and obviously much lower cost. Based on 6-months production experience, my recommendation would be to stay away from MPLS in the campus.
I'd be curious to hear what MPLS-specific issues you faced in the 6 months you had to operate such a network.
Been running IP/MPLS Core, Edge and Access networks for over 15 years, and apart from bugs which affect any protocol or feature implementation, I can't say it has been a nightmare to operate to the point of not recommending it.
I have far fewer words to say about STP, although - I'll admit - I've never run TRILL.
Mark.
In a message written on Fri, Oct 21, 2016 at 12:02:24PM -0500, Javier Solis wrote:
In a campus network the challenge becomes extending subnets across your core. You may have a college that started in one building with their own /24, but now have offices and labs in other buildings. They want to stay on the same network, but that's not feasible with the routed core setup without some other technology overlay. We end up not being able to extend the L2 like we did in the past and today we modify router ACL's to allow communications. If you already have hundreds of vlans spanned across the network, it's hard to get a campus to migrate to the routed core. I think this may be one of Marks challenge, correct me if I'm wrong please.
FWIW, if I had to solve the "college across buildings with common access control" problem I would create MPLS L3 VPN's, one subnet per building (where it is a VLAN inside of a building), with a "firewall in the cloud" somewhere to get between VLAN's with all of the policy in one place. No risk of the L2 across buildings mess, including broadcast and multicast issues at L2. All tidy L3 routing. Can use a real firewall between L3 VPN instances to get real policy tools (AV, URL Filtering, Malware detection, etc) rather than router ACL's. Scales to huge sizes because it's all L3 based. Combine with 802.1x port authentication and NAC, and in theory every L3 VPN could be in every building, with each port dynamically assigning the VLAN based on the user's login! Imagine never manually configuring them again. Write a script that makes all the colleges (20? 40? 60?) appear in every building all attached to their own MPLS VPN's, and then the NAC handles port assignment. -- Leo Bicknell - bicknell@ufp.org PGP keys at http://www.ufp.org/~bicknell/
FWIW, if I had to solve the "college across buildings with common access control" problem I would create MPLS L3 VPN's, one subnet per building (where it is a VLAN inside of a building), with a "firewall in the cloud" somewhere to get between VLAN's with all of the policy in one place.
No risk of the L2 across buildings mess, including broadcast and multicast issues at L2. All tidy L3 routing. Can use a real firewall between L3 VPN instances to get real policy tools (AV, URL Filtering, Malware detection, etc) rather than router ACL's. Scales to huge sizes because it's all L3 based.
Until people start complaining they can no more auto discover their Time Capsule left in the other building whereas their colleagues in the other building can etc etc. All fancy discover protocols breaks without L2 continuity ! Welcome to the campus network nightmare :) For now, there is no perfect solution ! either you cope with L2 hell or users inconvenience (and yes people tend to think that the campus network is expected to work as their home network) I've also stumbled upon some "Building Automation and Control Networks" (BACnet/IP for instance) where each building has some automats that all needs to be in the same network segment.
Combine with 802.1x port authentication and NAC, and in theory every L3 VPN could be in every building, with each port dynamically assigning the VLAN based on the user's login! Imagine never manually configuring them again. Write a script that makes all the colleges (20? 40? 60?) appear in every building all attached to their own MPLS VPN's, and then the NAC handles port assignment.
Here again, it's perfect until you start coping with old stuff, all fancy new ethernet capable "things" or scientific/industrial equipments. The "802.1x what ? it's plug'n play man !" attitude. (my experience is with research institutes/academy kind of campuses) Youssef Ghorbal
On Oct 21, 2016, at 4:18 PM, Youssef Ghorbal <youssef.ghorbal@gmail.com> wrote, in part:
Until people start complaining they can no more auto discover their Time Capsule left in the other building whereas their colleagues in the other building can etc etc. All fancy discover protocols breaks without L2 continuity !
Minor Correction: Correctly configured*, an Airport Extreme basestation (Time Capsule or not) does not require L2 connectivity to discover. In fact, Wide Area is used for discovery of many services not necessarily reachable by L2 connectivity. Apple’s Back to My Mac service is one example. *Apple’s "Back to My Mac” Wide Area Bonjour is enabled on an Airport basestation by entering appropriate Apple ID and password data in the Base Station tab as accessed by Airport Utility. James R. Cutler James.cutler@consultant.com PGP keys at http://pgp.mit.edu
This is exactly what we are recommending and building for our customers in that space. Most of the time the university network acts as a provider, so to me it only makes sense to use that type of tech. The biggest problem then is support, which could be something they are unwilling or unable to overcome.
On Oct 21, 2016, at 1:45 PM, Leo Bicknell <bicknell@ufp.org> wrote:
In a message written on Fri, Oct 21, 2016 at 12:02:24PM -0500, Javier Solis wrote:
In a campus network the challenge becomes extending subnets across your core. You may have a college that started in one building with their own /24, but now have offices and labs in other buildings. They want to stay on the same network, but that's not feasible with the routed core setup without some other technology overlay. We end up not being able to extend the L2 like we did in the past and today we modify router ACL's to allow communications. If you already have hundreds of vlans spanned across the network, it's hard to get a campus to migrate to the routed core. I think this may be one of Marks challenge, correct me if I'm wrong please.
FWIW, if I had to solve the "college across buildings with common access control" problem I would create MPLS L3 VPN's, one subnet per building (where it is a VLAN inside of a building), with a "firewall in the cloud" somewhere to get between VLAN's with all of the policy in one place.
No risk of the L2 across buildings mess, including broadcast and multicast issues at L2. All tidy L3 routing. Can use a real firewall between L3 VPN instances to get real policy tools (AV, URL Filtering, Malware detection, etc) rather than router ACL's. Scales to huge sizes because it's all L3 based.
Combine with 802.1x port authentication and NAC, and in theory every L3 VPN could be in every building, with each port dynamically assigning the VLAN based on the user's login! Imagine never manually configuring them again. Write a script that makes all the colleges (20? 40? 60?) appear in every building all attached to their own MPLS VPN's, and then the NAC handles port assignment.
-- Leo Bicknell - bicknell@ufp.org PGP keys at http://www.ufp.org/~bicknell/
On 21/Oct/16 19:02, Javier Solis wrote:
With that said, what are the best options to be able to cost effectively scale without using vlans and maintaining a routed core? What technology would someone suggest (mpls, vxlan,etc) to be the best possible solution?
IME, MPLS is a good use-case here. If you are going to use the same /24 (or whatever prefix applies to you) across multiple locations, you will need some kind of overlay. Be it IP-in-IP, GRE, MPLS (l2vpn's or l3vpn's) or plain old Ethernet, you will need something. MPLS makes a lot of sense to me. It's native in hardware, upper-layer agnostic, mature, and reasonably affordable even at low scale. While I would not say MPLS is simple from a "complexity in your network" standpoint, it does provide a notable amount of simplicity when you're looking for an overlay that is transparent to all manner of Layer 2 and Layer 3 applications that a packet-based network needs to transport. Mark.
On Sat, 22 Oct 2016 21:29:22 +0200, Mark Tinka wrote
On 21/Oct/16 19:02, Javier Solis wrote:
With that said, what are the best options to be able to cost effectively scale without using vlans and maintaining a routed core? What technology would someone suggest (mpls, vxlan,etc) to be the best possible solution?
IME, MPLS is a good use-case here. If you are going to use the same /24 (or whatever prefix applies to you) across multiple locations, you will need some kind of overlay. Be it IP-in-IP, GRE, MPLS (l2vpn's or l3vpn's) or plain old > Ethernet, you will need something.
MPLS makes a lot of sense to me. It's native in hardware, upper-layer agnostic, mature, and reasonably affordable even at low scale.
The question here is, whether MPLS is the *optimal* solution for campus needs. The same functionality could be obviously achived by multiple technologies, and while MPLS is well supported on high-end SP routers, various limitations appear when people try to use it on commodity ASICs which typically empower today's ethernet switches - one of them being e.g. limited ability to effectively load-balance traffic over multiple parallel links. Yes, in theory we could build all campus LANs using high-end SP routers, but when 100GE backbone is desired (which is often the case in EDU/NREN sector), the costs of such solution jump to unacceptable heights. Thus we looked for another technology, which doesn't have the usual L2 problems and is able to provide services we need (including L2 extensions to remote campuses) at reasonable costs and with enough simplicity. To avoid typical L2 problems, you clearly need a solution based on L3 routing. And TRILL is exactly that - although it maintains L2 interface to the outside world, internally it performs dynamic L3 routing by IS-IS protocol with all safety belts like TTL check, RPF check etc. IMHO, TRILL is much better fit for campus needs, since it was specifically designed for this networking space - and our 6-months production fully confirms that view (of course, YMMV). With kind regards, M.
On 22/Oct/16 23:59, Marian Ďurkovič wrote:
The question here is, whether MPLS is the *optimal* solution for campus needs.
The same functionality could be obviously achived by multiple technologies, and while MPLS is well supported on high-end SP routers, various limitations appear when people try to use it on commodity ASICs which typically empower today's ethernet switches - one of them being e.g. limited ability to effectively load-balance traffic over multiple parallel links.
Yes, in theory we could build all campus LANs using high-end SP routers, but when 100GE backbone is desired (which is often the case in EDU/NREN sector), the costs of such solution jump to unacceptable heights.
Thus we looked for another technology, which doesn't have the usual L2 problems and is able to provide services we need (including L2 extensions to remote campuses) at reasonable costs and with enough simplicity.
To avoid typical L2 problems, you clearly need a solution based on L3 routing. And TRILL is exactly that - although it maintains L2 interface to the outside world, internally it performs dynamic L3 routing by IS-IS protocol with all safety belts like TTL check, RPF check etc.
IMHO, TRILL is much better fit for campus needs, since it was specifically designed for this networking space - and our 6-months production fully confirms that view (of course, YMMV).
I don't consider the ASR920 or CES2000 to be particularly high-end routers, but YMMV. True, merchant silicon presents a number of data plane challenges that may, at first, seem non-trivial or completely go unnoticed. That is why we stay away from the ACX5000, for example. I expect improvements to come with newer-generation ASIC's/NP's, but that tests one's patience. But, like I said, I have not run TRILL myself, so I'm not going to tell you that it's not an ideal technology for this use-case. All I'll say is that MPLS is not limited to high-end platforms, even when custom silicon is involved. Mark.
participants (7)
-
David Bass -
James R Cutler -
Javier Solis -
Leo Bicknell -
Marian Ďurkovič -
Mark Tinka -
Youssef Ghorbal