MPLS and Carrier Ethernet, Oh My!
A few years back, every Tom, Dick, and Harry was touting MPLS or Carrier Ethernet NNIs with 10G ports everywhere. They still are. However, I rarely have seen that graduate to 100G ports and I don't think I've seen anyone talk about 400G ports. Is the hardware not there, or is it more a case of the technology hasn't been deployed widely, so it'll only be seen at a limited number of locations? I'm assuming it's more of the latter as the bigger hardware I've been looking at touts those features and port sizes, but maybe there's some other unadvertised limitation. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP
On 12/27/24 05:15, Jason Bothe via NANOG wrote:
I’m thinking there is not a a wide market for them, not to mention last mile support. Folks that need 100 or 400 ports typically want point to point and at that juncture, wavelength or IPoDWDM makes sense as more than likely these are going between data centers and not to your average enterprise premises in a multi tenant building. Just MHO.
This is correct. It is a lot of capacity to put in the hands of a router that is sharing its hardware and capacity with other services that have nothing to do with plain transport. Also, it is a cheaper way to deliver transport capacity for both operator and customer. Mark.
They’re out there. I’ve helped stand up carriers recently that sell them and have also bought such service from other carriers. At least for 100G. I haven’t seen 400G Ethernet point to point services but also have not been asked to engineer to sell them either so I suspect that demand is either soft or the majority that want 400G point to point today are as Aaron says electing to serve that need via alternate solutions . Ed ________________________________ From: NANOG <nanog-bounces+edwin.mallette=gmail.com@nanog.org> on behalf of Aaron1 <aaron1@gvtc.com> Sent: Friday, December 27, 2024 10:17:31 AM To: Mike Hammett <nanog@ics-il.net> Cc: NANOG <nanog@nanog.org> Subject: Re: MPLS and Carrier Ethernet, Oh My! Since 2023, we’ve (GVTC) been migrating our CBH MTSO handoffs and partner ENNI’s from 1g/10g to 100g. …moving off older ME3600’s, ASR9k’s and ACX5048’s, to newer MX series (204,304,240,480,960) Aaron On Dec 26, 2024, at 8:35 PM, Mike Hammett <nanog@ics-il.net> wrote: A few years back, every Tom, Dick, and Harry was touting MPLS or Carrier Ethernet NNIs with 10G ports everywhere. They still are. However, I rarely have seen that graduate to 100G ports and I don't think I've seen anyone talk about 400G ports. Is the hardware not there, or is it more a case of the technology hasn't been deployed widely, so it'll only be seen at a limited number of locations? I'm assuming it's more of the latter as the bigger hardware I've been looking at touts those features and port sizes, but maybe there's some other unadvertised limitation. ----- Mike Hammett Intelligent Computing Solutions<http://www.ics-il.com/> [http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/ICSIL>[http://www.ics-il.com/images/googleicon.png]<https://plus.google.com/+IntelligentComputingSolutionsDeKalb>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/intelligent-computing-solutions>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/ICSIL> Midwest Internet Exchange<http://www.midwest-ix.com/> [http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/mdwestix>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/midwest-internet-exchange>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/mdwestix> The Brothers WISP<http://www.thebrotherswisp.com/> [http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/thebrotherswisp>[http://www.ics-il.com/images/youtubeicon.png]<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
On 12/27/24 05:27, Edwin Mallette wrote:
They’re out there. I’ve helped stand up carriers recently that sell them and have also bought such service from other carriers. At least for 100G.
I haven’t seen 400G Ethernet point to point services but also have not been asked to engineer to sell them either so I suspect that demand is either soft or the majority that want 400G point to point today are as Aaron says electing to serve that need via alternate solutions .
The bulk of 400G services are being ordered by the cloud and content folk. In the telco space, 400G services are seen mostly on the line-side to run the optical backbone. Most telco's are still ordering 100G or N x 100G DWDM services, especially on the subsea side where the distance limits the channel bandwidth. For short metro links, we are seeing some telco's deploy 400G in IPoDWDM scenarios, where coherent QSFP-DD 400G plugs are cheap enough because they only need 1 - 8 400G channels. Some of these may even be amplified links if they are longer than 60km. But this is still quite rare. Exchange points are also the other use-cases that will run 400G links within their short metro network. I think there will be some 400G interest from telco's by Q4'25 for client-side services, and I suspect we shall start to see some content and cloud demand for 800G services by the start of Q2'26, if not slightly earlier. Mark.
On 12/27/24 04:34, Mike Hammett wrote:
A few years back, every Tom, Dick, and Harry was touting MPLS or Carrier Ethernet NNIs with 10G ports everywhere. They still are. However, I rarely have seen that graduate to 100G ports and I don't think I've seen anyone talk about 400G ports.
Is the hardware not there, or is it more a case of the technology hasn't been deployed widely, so it'll only be seen at a limited number of locations? I'm assuming it's more of the latter as the bigger hardware I've been looking at touts those features and port sizes, but maybe there's some other unadvertised limitation.
There was enough of a gap between 10G trunks and <1G customer services, together with the relatively low cost of 10G hardware, that made it sensible to deploy 10G NNI's everywhere. But because most core links are 10G or N x 10G, selling a 10G, 20G, 30G or 40G EoMPLS service means you will struggle with a lot of ECMP chaos in the core, if going to 100G core links is nowhere near your budget cycle. Moreover, if you want to deliver 10G or N x 10G EoMPLS services, 100G hardware is not as cheap now as 10G was when EoMPLS was all the rage. In such cases, a provider will find it easier to deliver your 10G or N x 10G as an EoDWDM service. There isn't much of a gap between 100G and 400G (only 4X vs. 10X for 1G-to-10G and 10G-to-100G). So trying to deliver a, say, 100G or N x 100G EoMPLS service means you will need to build multiple 400G links (either as N x 100G or N x 400G, depending on your budget), or even start to consider 800G hardware. When you look at the cost of 100G, 400G and 800G routers/switches vs. DWDM gear, you'd be mad to try to deliver EoMPLS services of that magnitude on your IP/MPLS network. My suspicion is that as operators start to get closer to building 100G-and-beyond core links on a regular basis, they will slowly migrate away from EoMPLS to EoDWDM for their point-to-point circuits. Mark.
It seems like the bulk of the responses are assuming a point-to-point service. What about aggregation of smaller services? I don't necessarily mean aggregation of hundreds of sub 1 gig circuits, but when you need more than 10 gig, your next step is needing a hundred, even if it's far away. You can only fit so many SLAed multi-gig services on a 10 gig port. This becomes a big deal when cross connects are as expensive as they are -----Mike HammettIntelligent Computing SolutionsMidwest Internet ExchangeThe Brothers WISP ----- Original Message ----- From: Mike Hammett <nanog@ics-il.net> To: NANOG <nanog@nanog.org> Sent: Thu, 26 Dec 2024 20:34:50 -0600 (CST) Subject: MPLS and Carrier Ethernet, Oh My! A few years back, every Tom, Dick, and Harry was touting MPLS or Carrier Ethernet NNIs with 10G ports everywhere. They still are. However, I rarely have seen that graduate to 100G ports and I don't think I've seen anyone talk about 400G ports. Is the hardware not there, or is it more a case of the technology hasn't been deployed widely, so it'll only be seen at a limited number of locations? I'm assuming it's more of the latter as the bigger hardware I've been looking at touts those features and port sizes, but maybe there's some other unadvertised limitation. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP
On 12/27/24 09:21, Mike Hammett wrote:
It seems like the bulk of the responses are assuming a point-to-point service. What about aggregation of smaller services? I don't necessarily mean aggregation of hundreds of sub 1 gig circuits, but when you need more than 10 gig, your next step is needing a hundred, even if it's far away. You can only fit so many SLAed multi-gig services on a 10 gig port. This becomes a big deal when cross connects are as expensive as they are
That is logical, yes. My experience has been the backhaul provider either does not have a 100G NNI product, or will require your kidney for them to deploy one. This is, of course, before they look at what their network will require to support that kind of transport, if you grow. I have also found that internal product and business development teams are happy to have a 100G NNI with a backhaul provider, but they assume the cost of a 100G port is the same as that of a 10G port. When they realize it's not, their business case falls apart. So not unheard of, sure... but a few more bits that need to be ironed out, I've found, both internally and with your backhaul provider. Mark.
The average subscriber consumes 1.5Mbps in a busy hour (even for the GE access). 100GE is 60k subscribers. Many access regions just do not have enough subscribers to justify 100GE. Some regions have. Ed/ From: NANOG <nanog-bounces+vasilenko.eduard=huawei.com@nanog.org> On Behalf Of Mike Hammett Sent: Friday, December 27, 2024 05:35 To: NANOG <nanog@nanog.org> Subject: MPLS and Carrier Ethernet, Oh My! A few years back, every Tom, Dick, and Harry was touting MPLS or Carrier Ethernet NNIs with 10G ports everywhere. They still are. However, I rarely have seen that graduate to 100G ports and I don't think I've seen anyone talk about 400G ports. Is the hardware not there, or is it more a case of the technology hasn't been deployed widely, so it'll only be seen at a limited number of locations? I'm assuming it's more of the latter as the bigger hardware I've been looking at touts those features and port sizes, but maybe there's some other unadvertised limitation. ----- Mike Hammett Intelligent Computing Solutions<http://www.ics-il.com/> [http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/ICSIL>[http://www.ics-il.com/images/googleicon.png]<https://plus.google.com/+IntelligentComputingSolutionsDeKalb>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/intelligent-computing-solutions>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/ICSIL> Midwest Internet Exchange<http://www.midwest-ix.com/> [http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/mdwestix>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/midwest-internet-exchange>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/mdwestix> The Brothers WISP<http://www.thebrotherswisp.com/> [http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/thebrotherswisp>[http://www.ics-il.com/images/youtubeicon.png]<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
These days, it's a LOT easier to get dedicated ethernet wave service between A and Z than it used to be. The pseudowire options were developed to fill that gap that customers wanted. Still certainly use cases for it, but generally the dedicated waves are much easier to get and probably cheaper. The providers would certainly prefer NOT to sell you that service either in most cases, because they don't want the overhead of running their parts either, and would like to see it die on the vine. On Thu, Dec 26, 2024 at 9:37 PM Mike Hammett <nanog@ics-il.net> wrote:
A few years back, every Tom, Dick, and Harry was touting MPLS or Carrier Ethernet NNIs with 10G ports everywhere. They still are. However, I rarely have seen that graduate to 100G ports and I don't think I've seen anyone talk about 400G ports.
Is the hardware not there, or is it more a case of the technology hasn't been deployed widely, so it'll only be seen at a limited number of locations? I'm assuming it's more of the latter as the bigger hardware I've been looking at touts those features and port sizes, but maybe there's some other unadvertised limitation.
----- Mike Hammett Intelligent Computing Solutions <http://www.ics-il.com/> <https://www.facebook.com/ICSIL> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb> <https://www.linkedin.com/company/intelligent-computing-solutions> <https://twitter.com/ICSIL> Midwest Internet Exchange <http://www.midwest-ix.com/> <https://www.facebook.com/mdwestix> <https://www.linkedin.com/company/midwest-internet-exchange> <https://twitter.com/mdwestix> The Brothers WISP <http://www.thebrotherswisp.com/> <https://www.facebook.com/thebrotherswisp> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
On 12/27/24 18:28, Tom Beecher wrote:
These days, it's a LOT easier to get dedicated ethernet wave service between A and Z than it used to be. The pseudowire options were developed to fill that gap that customers wanted.
Still certainly use cases for it, but generally the dedicated waves are much easier to get and probably cheaper. The providers would certainly prefer NOT to sell you that service either in most cases, because they don't want the overhead of running their parts either, and would like to see it die on the vine.
You begin to run into commercial and technical constraints (let's not call them limits) when your EoMPLS orders start to enter the 15G - 20G range and higher. Technically, if a customer wants DWDM-type behaviour where the path is fixed, you have to build an RSVP-TE tunnel, which is admin. It's possible this can be done more easily with SR-MPLS, but it's still a bit of admin. Then you have to make considerations for failure scenarios in your core, and how you guarantee your pw customer remains happy. Commercially, if customers do not demand DWDM-type behavious driven by ERO's, you have a conundrum about whether you charge for automatic protection or not, because without an ERO, the customer will failover when the backbone fails over due to a core outage of some sort. Mark.
*nods* for PtP, I agree. As a buyer (and a seller) waves, waves, waves. As a seller, it's less stuff for me to manage. As a buyer, I don't have to trust you on oversubscribing a wave because you can't. You can oversubscribe the hell out of an Ethernet circuit, though. Waves are much harder in a PtMP environment, especially with cross connect costs. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Tom Beecher" <beecher@beecher.cc> To: "Mike Hammett" <nanog@ics-il.net> Cc: "NANOG" <nanog@nanog.org> Sent: Friday, December 27, 2024 10:28:02 AM Subject: Re: MPLS and Carrier Ethernet, Oh My! These days, it's a LOT easier to get dedicated ethernet wave service between A and Z than it used to be. The pseudowire options were developed to fill that gap that customers wanted. Still certainly use cases for it, but generally the dedicated waves are much easier to get and probably cheaper. The providers would certainly prefer NOT to sell you that service either in most cases, because they don't want the overhead of running their parts either, and would like to see it die on the vine. On Thu, Dec 26, 2024 at 9:37 PM Mike Hammett < nanog@ics-il.net > wrote: A few years back, every Tom, Dick, and Harry was touting MPLS or Carrier Ethernet NNIs with 10G ports everywhere. They still are. However, I rarely have seen that graduate to 100G ports and I don't think I've seen anyone talk about 400G ports. Is the hardware not there, or is it more a case of the technology hasn't been deployed widely, so it'll only be seen at a limited number of locations? I'm assuming it's more of the latter as the bigger hardware I've been looking at touts those features and port sizes, but maybe there's some other unadvertised limitation. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP
On 12/28/24 00:31, Mike Hammett wrote:
*nods* for PtP, I agree. As a buyer (and a seller) waves, waves, waves. As a seller, it's less stuff for me to manage. As a buyer, I don't have to trust you on oversubscribing a wave because you can't. You can oversubscribe the hell out of an Ethernet circuit, though.
Waves are much harder in a PtMP environment, especially with cross connect costs.
That is a problem OpenXR has been designed to fix: https://openxropticsforum.org Infinera have already started shipping product: https://www.infinera.com/innovation/xr-optics/ Mark.
participants (7)
-
Aaron1 -
Edwin Mallette -
Jason Bothe -
Mark Tinka -
Mike Hammett -
Tom Beecher -
Vasilenko Eduard